Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization
Abstract
Optimizing expensive-to-evaluate black-box functions of discrete (and potentially continuous) design parameters is a ubiquitous problem in scientiﬁc and engineering applications.
Bayesian optimization (BO) is a popular, sample-efﬁcient method that leverages a probabilistic surrogate model and an acquisition function (AF) to select promising designs to evaluate.
However, maximizing the AF over mixed or high-cardinality discrete search spaces is challenging standard gradient-based methods cannot be used directly or evaluating the AF at every point in the search space would be computationally prohibitive.
To address this issue, we propose using probabilistic reparameterization (PR).
Instead of directly optimizing the AF over the search space containing discrete parameters, we instead maximize the expectation of the AF over a probability distribution deﬁned by continuous parameters.
We prove that under suitable reparameterizations, the BO policy that maximizes the probabilistic objective is the same as that which maximizes the AF, and therefore, PR enjoys the same regret bounds as the original BO policy using the underlying AF.
Moreover, our approach provably converges to a stationary point of the probabilistic objective under gradient ascent using scalable, unbiased estimators of both the probabilistic objective and its gradient. Therefore, as the number of starting points and gradient steps increase, our approach will recover of a maximizer of the AF (an often-neglected requisite for commonly used BO regret bounds).
We validate our approach empirically and demonstrate state-of-theart optimization performance on a wide range of real-world applications.
PR is complementary to (and beneﬁts) recent work and naturally generalizes to settings with multiple objectives and black-box constraints.
Introduction
Many scientiﬁc and engineering problems involve tuning discrete and/or continuous parameters to optimize an objective function.
Often, the objective function is “black-box”, meaning it has no known closed-form expression.
For example, optimizing the design of an electrospun oil sorbent—a material that can be used to absorb oil in the case of a marine oil spill to mitigate ecological harm—to maximize properties such as the oil absorption capacity and mechanical strength [59] can involve tuning both discrete ordinal experimental conditions and continuous parameters controlling the composition of the material.
For another example, optimizing the structural design of a welded beam can involve tuning the type of metal (categorical), the welding type (binary), and the dimensions of the different components of the beam (discrete ordinals)–resulting in a search space with over 370 million possible designs [54].
We consider the scenario where querying the objective function is expensive and sample-efﬁciency is crucial.
In the case of designing the oil sorbent, evaluating the objective function requires manufacturing the material and measuring its properties in a laboratory, requiring signiﬁcant time and resources.
Bayesian optimization (BO) is a popular technique for sample-efﬁcient black-box optimization, due to its proven performance guarantees in many settings [5, 52] and its strong empirical performance [23, 55].
BO leverages a probabilistic surrogate model of the unknown objective(s) and an acquisition function (AF) that provides utility values for evaluating a new design to balance exploration and exploitation.
Typically, the maximizer of the AF is selected as the next design to evaluate.
However, maximizing the AF over mixed search spaces (i.e., those consisting of discrete and continuous parameters) or large discrete search spaces is challenging 1 and continuous (or gradient-based) optimization routines cannot be directly applied.
Theoretical performance guarantees of BO policies require that the maximizer of the AF is found and selected as the next design to evaluate on the black-box objective function [52].
When the maximizer is not found, regret properties are not guaranteed, and the performance of the BO policy may degrade.
To tackle these challenges, we propose a technique for improving AF optimization using a probabilistic reparameterization (PR) of the discrete parameters.
Our main contributions are:
We propose a technique, probabilistic reparameterization (PR), for maximizing AFs over discrete and mixed spaces by instead optimizing a probabilistic objective (PO): the expectation of the AF over a probability distribution of discrete random variables corresponding to the discrete parameters.
We prove that there is an equivalence between the maximizers of the acquisition function and the the maximizers of the PO and hence, the policy that chooses designs that are best with respect to the PO enjoys the same performance guarantees as the standard BO policy.
We derive scalable, unbiased Monte Carlo estimators of the PO and its gradient with respect to the parameters of the introduced probability distribution.
We show that stochastic gradient ascent using our gradient estimator is guaranteed to converge to a stationary point on the PO surface and will recover a global maximum of the underlying AF as the number of starting points and gradient steps increase.
This is important because many BO regret bounds require maximizing the AF [52].
Although the AF is often non-convex and maximization is hard, empirically, with a modest number of starting points, PR leads to better AF optimization than alternative methods.
We show that PR yields state-of-the-art optimization performance on a wide variety of real-world design problems with discrete and mixed search spaces.
Importantly, PR is complementary to many existing approaches such as popular multi-objective, constrained, and trust region-based approaches; in particular, PR is agnostic to the underlying probabilistic model over discrete parameters—which is not the case for many alternative methods.
Preliminaries
Bayesian Optimization We consider the problem of optimizing a black-box function f : X ×Z → R over a compact search space X × Z , where X = X
(1) × · · · × X ( d ) is the domain of the d ≥continuous parameters ( x ( i )
∈ X ( i ) for i
= 1 , ..., d ) and Z = Z (1) × · · · × Z ( d z ) is the domain of the d z ≥ 1 discrete parameters ( z ( i
) ∈
Z ( i ) for i
= 1 , ..., d z ).
BO leverages (i) a probabilistic surrogate model—typically a Gaussian process (GP) [46]—ﬁt to a data set D n = { x i , z i , y i } ni =1 of designs and corresponding (potentially noisy) observations
y i
=
f
( x i , z i ) + (cid:15) i , (cid:15) i ∼ N (0 , σ 2 ) , and (ii) an acquisition function α ( x , z ) that uses the surrogate model’s posterior distribution to quantify the value of evaluating a new design.
Common AFs include expected improvement (EI) [32] and upper conﬁdence bound (UCB) [52]—the latter of which enjoys no-regret guarantees in certain settings [52].
The next design to evaluate is chosen by maximizing evaluated at every discrete element, then acquisition optimization can be solved trivially.
Throughout this paper, we use a mixed search space
X ×
Z in our derivations, theorems, and proofs, without loss of generality with respect to the case of a purely discrete search space.
If d = 0 , then the objective function f : Z → R is deﬁned over the discrete space Z and the continuous parameters in this exposition can simply be ignored. the AF α ( x , z ) over X × Z .
Although the black-box objective f is expensive-to-evaluate, the AF is relatively cheap-to-query, and therefore, it can be optimized numerically.
Gradient-based optimization routines are often used to maximize the AF over continuous domains [25].
Discrete Parameters In its basic form, BO assumes that the inputs are continuous.
However, discrete parameters such as binary, discrete ordinal, and non-ordered categorical parameters are ubiquitous in many applications.
In the presence of such parameters, optimizing the AF is more difﬁcult, as standard gradient-based approaches cannot be directly applied.
Recent works have proposed various approaches including multi-armed bandits [40, 48] and local search [41] for discrete domains and interleaved discrete/continuous optimization procedures for mixed domains [15, 57].
A simple and widely-used approach across many popular BO packages [1, 53] is to one-hot encode the categorical parameters, apply a continuous relaxation when solving the optimization, and discretize (round) the resulting continuous candidates.
Examples of continuous relaxations and discretization functions are listed in Table 1.
[Table]
Although using a continuous relaxation allows for efﬁcient optimization using standard optimization routines in an alternate continuous domain Z (cid:48)
⊂ R m , the AF value for an infeasible continuous value (i.e., z (cid:48) / ∈ Z ) does not account for the discretization that must occur before the black-box function is evaluated.
Moreover, the acquisition value for an infeasible continuous value can be larger than the AF value after discretization.
For an illustration of this, see Fig.
1 (middle/right).
In the worst case, BO will repeatedly select the same infeasible continuous design due to its high
AF value, but discretization will result in a design that has already been evaluated and has zero AF value.
To mitigate this degenerate behavior and avoid the over-estimation issue, Garrido-Merchán and Hernández-Lobato [26] propose discretizing z (cid:48) before evaluating the AF, but the AF is then non-differentiable with respect to the z (cid:48) .
While this improves performance on small search spaces, the response surface has large ﬂat regions after discretizing z (cid:48) , which makes it difﬁcult to optimize the
AF.
The authors of [26] propose to approximate the gradients using ﬁnite differences, but, empirically, we ﬁnd that this approach to be leads to sub-optimal AF optimization relative to PR.
Probabilistic Reparameterization
Algorithm 1 BO with PR 1: Input: black-box objective f : X × Z
→ R 2: Initialize D 0 ← ∅ , GP 0 ← GP ( 0 , k ) 3: for n = 1 to N iterations do 4: ( x n , θ n ) ← arg max ( x , θ ) ∈X× Θ E Z ∼ p ( Z | θ ) [ α ( x , Z )] 5:
Sample z n ∼ p
( Z | θ n ) 6:
Evaluate f ( x n , z n ) 7:
D n ← D n − 1 ∪ { ( x n , z n , f ( x n , z n )) } 8:
Update posterior GP n given D n 9: end for We propose an alternative approach based on probabilistic reparameterization, a relaxation of the original optimization problem involving discrete parameters. Rather than directly optimizing the AF via a continuous relaxation z (cid:48) of the design z , we instead reparameterize the optimization problem by introducing a discrete probability distribution p ( Z | θ ) over a random variable Z with support exclusively over Z .
This distribution is parameterized by a vector of continuous parameters θ .
We use z to denote the vector ( z (1) , ..., z ( d z ) ) , where each element is a different (possibly vector-valued) discrete parameter.
Given this reparameterization, we deﬁne the probabilistic objective (PO):
[Equation]
Algorithm 1 outlines BO with probabilistic reparameterization.
PR allows us to optimize θ and x over a continuous space to maximize the PO instead of optimizing x and z to maximize α directly over the mixed search space
X × Z .
As we will show later, maximizing
[Figure]
Relax.
AF Value
[Figure] PO Value [Figure] the PO allows us to recover a maximizer of α over the space X × Z .
Choosing p ( Z | θ ) to be a discrete distribution over Z means the realizations of Z are feasible values in Z .
Hence, the AF is only evaluated for feasible discrete designs.
Since p ( Z | θ ) is a discrete probability distribution, we can express E Z ∼ p
( Z | θ )
[ α ( x , Z )] as a linear combination where each discrete design is weighted by its probability mass:
[Equation]
Example distributions for binary, ordinal, and categorical parameters are provided in Table 2. [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
(cid:98)
[Equation]
(cid:99) )
θ
[Equation]
C
[Equation]
[Table]
[Equation] [Table] [Equation] C
[Equation] C
[Equation]
Although ordinal parameters could use the same categorical distributions as the non-ordered categorical parameters, we opt for the provided proposal distribution since it uses a scalar θ (rather than a C -element vector) and it naturally encodes the ordering of the values. Using an independent random variable Z ( i ) ∼ p ( Z ( i ) | θ ( i ) ) for each parameter z ( i ) for i = 1 , ..., d z means that the probabilistic objective can be expressed as
[Equation]
3.1 Analytic Gradients
One important beneﬁt of PR is that the PO in (1) is differentiable with respect to θ (and x , if the gradient of α with respect to x exists), whereas α ( x , z ) is not differentiable with respect to z .
The gradients of the PO with respect to θ and x can be obtained by differentiating Equation 2:
[Equation]
This enables optimizing the PO (line 4 of Algorithm 1) efﬁciently and effectively using gradient-based methods.
3.2 Theoretical Properties
In this section, we derive theoretical properties of PR.
Proofs are provided in Appendix B.
Our ﬁrst result is that there is an equivalence between the maximizers of the PO and the maximizers of the AF over X × Z .
Theorem 1 (Consistent Maximizers) .
Suppose that α is continuous in x for every z ∈ Z .
Let H ∗ be the maximizers of α ( x , z ) : H ∗ = { ( x , z ) ∈ arg max ( x , z )
∈X×Z α ( x , z ) } .
Let J ∗ ⊆
X × Θ be the maximizers of E Z ∼ p
( Z | θ )
[ α ( x , Z )] : J ∗ = { ( x , θ ) ∈ arg max ( x , θ ) ∈X×
Θ E Z ∼ p
( Z | θ )
[ α ( x , Z )] } , where Θ is the domain of θ .
Let ˆ H ∗ ⊆
X × Z be deﬁned as: ˆ H ∗ =
{ ( x , ˜ z ) : ( x , θ ) ∈ J ∗ , ˜ z ∼ p ( Z | θ ) } .
Then, ˆ H ∗ =
H ∗ .
Algorithm 1 outlines BO with probabilistic reparameterization.
Importantly, Theorem 1 states that sampling from the distribution parameterized by a maximizer of the PO yields a maximizer of α , and therefore, Algorithm 1 enjoys the performance guarantees of α ( · ) .
Corollary 1 (Regret Bounds) .
Let α ( x , z ) be an acquisition function over a search space X ×Z such that when α is applied as part of a BO strategy that strategy has bounded regret .
If the conditions for the regret bounds of that BO strategy using α are satisﬁed, then Algorithm 1 using α enjoys the same regret bound.
Examples of BO policies with bounded regret include those based on AFs such as upper conﬁdence bound (UCB) [52] or Thompson sampling (TS) [49] for single objective optimization, and UCB or
TS with Chebyshev [43] or hypervolume [66] scalarizations in the multi-objective setting.
Although the BO policy selects a maximizer of α is equivalent to the BO policy in Algorithm 1, maximizing the AF over mixed or high-dimensional discrete search spaces is challenging because commonly used gradient-based methods cannot directly be applied.
The key advantage of our approach is that maximizers of the AF can be identiﬁed efﬁciently and effectively by optimizing the PO using gradient information instead of directly optimizing the AF.
We ﬁnd that optimizing PR yields better results than directly optimizing α or other common relaxations as shown in Figure 1(Left), where we compare AF optimization methods on the mixed Rosenbrock test problem (see Appendix C for details).
Practical Monte Carlo Estimators 4.1 Unbiased estimators of the Probabilistic Reparameterization and its Gradient
As the number of discrete conﬁgurations ( |Z| ) increases, the PO and its gradient may become computationally expensive to evaluate analytically because both require a summation of |Z| terms.
Therefore, we propose to estimate the PO and its gradient using Monte Carlo (MC) sampling.
The
MC estimator of the PO is given by
[Equation]
where ˜ z 1 , ...,
˜ z N are samples from p ( Z | θ ) .
This estimator is unbiased and can be computed for a large number of samples by evaluating the AF independently (or in chunks) for each input ( x , ˜ z n ) .
MC can also be used to estimate the gradient of the PO with respect to θ .
We opt for using a score function gradient estimator [35] (also known as REINFORCE [62] and the likelihood ratio estimator [27]) because it is simple, scalable, and can be computed using the acquisition values { α ( x ,
˜ z i ) } Ni = that are used in the MC estimator of the PO.
Many alternative lower variance estimators (e.g. Yin et al.
[64], Yin et al.
[65]) would require many additional AF evaluations (see Mohamed et al.
[39] for a review of MC gradient estimation).
The score function is the gradient of the log probability with respect to the parameters of the distribution: ∇ θ log p ( Z | θ ) = ∇ θ p ( Z | θ ) p
( Z | θ ) .
Using this score function, we can express the analytic gradient as
[Equation]
The unbiased MC estimator of the gradient of the PO with respect to θ is given by
[Equation]
Since the score function gradient is only deﬁned when p ( z | θ ) > 0 , we reparameterize θ to ensure p ( z | θ ) > 0 for all z and θ by using the softmax transformations provided in Table 3, which are commonly used for computational convenience and stability in probablistic reparameterization [64, 65], and the solution converges as τ → 0 .
P ARAMETER T YPE
T RANSFORMATION ( θ = g ( φ ) )
B INARY θ = σ (( φ − 12 ) /τ )
O RDINAL θ = (cid:98) φ (cid:99) + σ (( φ − (cid:98) φ (cid:99) − 12 ) /τ )
C ATEGORICAL θ ( c ) =
SOFTMAX (( φ − 0 . 5) /τ ) ( c )
Moreover, even though p ( z | θ ) > 0 , when p ( z | θ ) is small, a small number N of MC samples are unlikely to produce any samples where ˜ z
= z .
Instead of optimizing θ directly, we instead optimize φ .
Since the transformations g ( · ) are differentiable with respect to φ , the gradient (and MC gradient estimator) of the PO with respect to φ are easily obtained using the gradient of the PO with respect to θ and a simple application the chain rule (multiplying by ∇ φ θ ).
4.2 Variance Reduction in Monte Carlo Gradient Estimation
Although the MC gradient estimator in (7) is unbiased, score function gradient estimators can suffer from high variance [39].
Therefore, we adopt a popular technique for variance reduction where the score function itself is used as a control variate, since its expectation is zero under p ( Z | θ ) [39].
Score function estimators with this control variate have been shown to be among the best performing gradient estimators [39].
Moreover, this technique is simple and merely amounts to subtracting a value β from the acquisition value in the score function estimator in Equation (7):
[Equation]
The β is commonly known as a baseline and is often taken to be a moving average of the (acquisition) values [39].
See Appendix C for details on β .
4.3 Convergence Guarantee using Stochastic Gradient Ascent
Since the score function gradient estimator is unbiased, we can leverage previous work on convergence in probability under stochastic gradient ascent [47] to arrive at our main convergence result for acquisition optimization.
Theorem 2 (Convergence Guarantee) .
Let α : X × Z
→ R be differentiable in x for every z ∈ Z .
Let (ˆ x t,m , ˆ θ t,m ) be the best solution after running stochastic gradient ascent for t time steps on the probabilistic objective E
Z ∼ p
( Z | θ )
[ α ( x , Z )] from m starting points with its unbiased MC estimators proposed above.
Let { a t } ∞ t =1 be a sequence of positive step sizes such that 0 < (cid:80) ∞ t =1 a 2 t =
A < ∞ and (cid:80) ∞ t =1 a t = ∞ , where a t is the step size used in stochastic gradient ascent at time step t .
Let ˆ z t,m ∼ p
( Z | ˆ θ t,m ) .
Then as t → ∞ , m → ∞ , and τ → 0 , (ˆ x t,m , ˆ z t,m ) → ( x ∗ , z ∗ ) ∈ arg max ( x , z ) ∈X×Z α ( x , z ) in probability.
The signiﬁcance of Theorem 2 is that optimizing the PO is guaranteed to converge in probability to a global maximizer of the AF, meaning that optimizing the PO guarantees that resulting candidate design has maximal AF value.
The implication is that the intended BO policy is followed and the underlying regret bounds of the AF are recovered (provided that the other conditions of the regret bound are met).
Although global convergence is only guaranteed as m → ∞ , we observe in Figure 1(left) that PR yields strong, stable acquisition optimization with only m = 20 starting points, 200 steps, and τ = 110 (see Appendix G for further discussion) and outperforms alternative optimization approaches.
Related Work
Many methods for BO over discrete and mixed search spaces have been proposed.
Previous work has largely focused on (i) improving the surrogate models or (ii) improving AF optimization.
Improving models : Historically, methods leveraging tree-based surrogate models, e.g., SMAC [30] and TPE [4], have been popular for optimizing discrete or mixed search spaces.
Many recent works have considered alternative surrogate models.
BOCS encodes categorical parameters as binary variables and uses Bayesian linear regression with pairwise interactions [2].
COMBO uses a diffusion kernel on the graph deﬁned by the Cartesian product of discrete parameters [41].
M ER CBO similarly exploits the combinatorial graph, but with Mercer features and Thompson sampling [16].
H Y BO extends the diffusion kernels to mixed continuous-discrete spaces [15].
However, these methods scale poorly with respect to the number of data points and parameters.
Moreover, of the methods listed above, only H Y BO supports continuous parameters without restricting them to a discrete set.
H Y BO enjoys a universal function approximation property, but relies on summing over all possible orders of interactions between base kernels for each parameter which results in exponential complexity with respect to the number of parameters and limits its applicability to low-dimensional problems.
Moreover, the computational issues of such approaches make it difﬁcult to apply them to multi-objective and constrained optimization.
G RYFFIN [29] uses kernel density estimation, but is limited to categorical search spaces.
M I V A BO uses a linear combination of basis functions (e.g. pseudo Boolean features [7] for discrete parameters) with interaction terms[13].
MVRSM [6] uses ReLU-based surrogates for computational efﬁciency, but is limited by the expressiveness of these models.
Optimizing acquisition functions : As discussed previously, Garrido-Merchán and HernándezLobato [26] propose using continuous relaxation and discretize the inputs before evaluating the
AF. However, the resulting AF after discretization is piece-wise-ﬂat along slices of the continuous relaxation of the discrete parameters and therefore is difﬁcult to optimize.
C O C
A BO [48] samples discrete parameters using a multi-armed bandit and optimizes the continuous parameters conditional upon the sampled discrete parameters.
However, C O C A BO’s performance degrades as number of discrete conﬁgurations increases.
C ASMOPOLITAN [57] uses local trust regions combined with an interleaved AF optimization strategy that alternates between local search steps on the discrete parameters and gradient ascent for the continuous parameters.
Furthermore, both C O C A BO and C ASMOPOLITAN do not inherently exploit ordinal structure.
Probabilistic reparameterization : PR has been considered for optimizing discrete parameters in other domains such as reinforcement learning [62] and sparse regression [65].
However, PR has not been leveraged for BO. Although the reparameterization trick used by Wilson et al. [63] is in a similar vein to PR, Wilson et al.
[63] reparameterize an existing multivariate normal random variable in terms of standard normal random variables and then use sample-path gradient estimators.
In contrast, our approach introduces a new probabilistic formulation using discrete probability distributions and uses likelihood-ratio-based gradient estimators since sample-path gradients cannot be computed through discrete sampling.
Alternative methods for propagating gradients : Alternative methods for propagating gradients through discrete structures have been considered in the deep learning community (among others).
One approach is to use approximate discrete Concrete distributions [31, 37], which admit samplepath gradients.
However, samples from Concrete distributions are not discrete and approximation error can result in pathologies similar to evaluating the AF using continuous relaxation.
Moreover, approximately discrete samples prohibit using surrogate models that require discrete inputs (without discretizing the samples)—e.g., GPs with Hamming distance kernels [48].
Another approach for gradient propagation in the deep learning community is to use straight-through gradient estimators (STE) [3], where the gradient of the discretization function with respect to its input is estimated using, for example, an identity function.
This approach works well empirically in some cases, these estimators are not well-grounded theoretically.
Nevertheless, we discuss and evaluate using STE for AF optimization in Appendix H.
Experiments
In this section, we provide an empirical evaluation of PR on a suite of synthetic problems and real world applications.
For PR, we use stochastic mini-batches of N = 128 MC samples in our experiments and demonstrate that PR is robust with respect to the number of MC samples (and compare against analytic PR, where computationally feasible) in Appendix F. We optimize PR using
Adam [34] with an initial learning rate of 140 . We show that PR Adam is generally robust to the choice of learning rate (more so than vanilla stochastic gradient ascent) in the sensitivity analysis in
Figure 21 in Appendix M. We compare PR against two alternative acquisition optimization strategies: using a continuous relaxation (C ONT .
R ELAX .)
and using exact discretization with approximate gradients (E XACT R OUND )
These approaches optimize the acquisition function with L-BFGS-B with exact and approximate gradients, respectively.
In addition, we compare against two state-ofthe-art methods for discrete/mixed BO: a modiﬁed version of C ASMOPOLITAN [57] that additionally supports ordinal variables introduced in Wan et al.
[58] and H Y BO [15], both of which are shown to outperform the other related works discussed in Section 5.
In addition, we showcase how PR is complementary to existing methods such as trust region methods [22].
We demonstrate this by using PR with a trust region for the continuous and discrete ordinal parameters and optimize PR within this trust region.
In Appendix H, we provide comparison of TR methods with alterative optimizers and ﬁnd that PR is the best optimizer when using TRs on 6 of the 7 benchmark problems.
See
Appendix C for additional discussion of PR + TR.
For PR, E XACT R OUND , and PR + TR we use the sum of a product kernel and a sum kernel of a categorical kernel [48] for the categorical parameters and Matérn-5/2 kernel for all other parameters.
3 Alternative kernels over different representations of categorical parameters such as one-hot encoded vectors, latent embeddings [67], and known embeddings (e.g. using ﬁngerprint-based reaction encodings for categorical parameters in chemical reaction optimization [51]) are evaluated in Appendix J.
C ONT .
R ELAX .
, E XACT R OUND , PR, and PR + TR use expected improvement [24, 32] for single objective (constrained problems) and expected hypervolume improvement [20] for the multi-objective oil sorbent problem (where exact gradients with respect to continuous parameters are computed using auto-differentiation [8]).
We report the mean for each method ± 2 standard errors acrossreplications.
Performance is evaluated in terms of regret (feasible regret for constrained problems and hypervolume regret for multi-objective problems).
C ASMOPOLITAN and H Y BO are not run on
Welded Beam and Oil Sorbent as they do not support constrained and multi-objective optimization.
We also leave the multi-objective extension of PR+TR to future work because it would add additional complexity [11].
For H Y BO, we only run 60 BO iterations on SVM due to the large wall time (see
Figure 3) and only report partial results on Cellular Network due to a singular covariance matrix error.
See Appendix C for details on the experiment setup, regret metrics, benchmark problems, and methodological details.
We leverage existing open source implementations of C ASMOPOLITAN and
H Y BO (see Appendix C for links), and the implementations of all of other methods are available at https://github.com/facebookresearch/bo_pr .
6.1 Synthetic Problems
We evaluate all methods on 3 synthetic problems. Ackley is a 13-dimensional function withbinary and 3 continuous parameters (a modiﬁed version of the problem in Bliek et al.
Mixed
Int F1 is a 16-dimensional variant of the F1 function from Tušar et al.
[56] with 2 binary, 6 discrete ordinal parameters, and 8 continuous parameters.
The discrete ordinal parameters have following cardinalities: 2 parameters with 3 values, 2 with 5 values, and 2 with 7 values.
Rosenbrock is a 10-dimensional Rosenbrock function with 6 discrete ordinal parameters with 4 values each andcontinuous parameters.
6.2 Real World Problems
We consider 5 real world applications including a problem with 5 black-box outcome constraints and a 3-objective problem (see Appendix D for details on constrained and multi-objective BO).
Welded Beam Optimizing the design of a welded steel beam is a classical engineering optimization.
In this problem, the goal is to minimize manufacturing cost subject to 5 black-box constraints on structural properties of the beam (including shear stress, bending stress, and buckling load) by tuning 6 parameters: the welding conﬁguration (binary), the metal material type (categorical with 4 options), and 4 ordinal parameters controlling the dimensions of the beam [54].
SVM Feature Selection
This problem involves jointly performing feature selection and hyperparameter optimization for a Support Vector Machine (SVM) trained on the CTSlice UCI data set [18, 36].
The design space for this problem involves 50 binary parameters controlling whether a particular feature is included or not, and 3 continuous hyperparameters of the SVM.
Cellular Network Optimization
In this 30-dimensional problem, the goal is to tune the tilt (ordinal with 6 values) and transmission power (continuous) for a set of 15 antennas [50] to maximize a coverage quality metric that is a function of signal power and interference [38] over a geographic region of interest.
We use the simulator from Dreifuerst et al.
Direct Arylation Chemical Synthesis Palladium-catalysed direct arylation has generated signiﬁcant interest in the pharamceutical development sector[12].
In this problem, the goal is maximize yield for a direct arylation chemical reaction by tuning 3 categorical parameters corresponding to the choice of solvent, base, and ligand, as well 2 continuous parameters controlling the temperature and concentration.
We ﬁt a surrogate model to the direct arylation dataset from Shields et al.
[51] in order to facilitate continuous optimization of temperature and concentration.
In Appendix J, we demonstrate that PR can leverage a kernel over ﬁngerprint-based reaction encodings computed via density functional theory (DFT) for the categorical parameters [51]
[Figure]
Electrospun Oil Sorbent Marine oil spills can cause ecological catastrophe.
One avenue for mitigating environmental harm is to design and deploy absorbent materials to capture the spilled oil.
In this problem, we tune 5 ordinal parameters (3 parameters with 5 values and 2 with 4 values) andcontinuous parameters controlling the composition and manufacturing conditions for an electrospun oil sorbent material to maximize 3 competing objectives: the oil absorbing capacity, the mechanical strength [Figure] and the water contact angle [59] [Figure]
[Figure]
F
[Figure]
Rosenbrock
[Figure]
Ackley
[Figure]
Welded
[Figure]
(5 constraints)
[Figure]
6.3 Results
We ﬁnd PR consistently delivers strong empirical performance as shown in Figure 2.
On all benchmark problems, PR (or PR + TR ) outperforms all baseline methods (except for Mixed Int
F1, where H Y BO performs comparably) .
Figure 3 shows the wall time for candidate generation over the number of BO iterations.
Although PR is computationally intensive, the computation is embarrassingly parallel and therefore exploiting GPU acceleration yields competitive wall times.
Importantly, PR’s wall time scales well with the number of observations and design parameters, unlike H Y BO which scales poorly with both.
However, the complexity of PR scales additively in the number of GPs being used (e.g. outcomes being modeled), assuming they are evaluated sequentially.
Hence, in multi-objective or constrained settings, PR incurs a high cost in terms of wall time.
However, empirically PR achieves better optimization performance on constrained and multiobjective problems relative to C ONT .
R ELAX .
and E XACT R OUND .
We note that C ASMOPOLITAN does not support multi-objective BO or constrained BO, and although H Y BO could be used in those settings, it would be impractically slow because 1) its wall time would scale linearly with the number of modeled outcomes (using independent GPs) and 2) its diffusion kernel is non-differentiable, which would make optimizing hypervolume-based AFs slow [8, 9].
Discussion
The performance and regret properties of BO depend critically on properly maximizing the AF.
For problems with discrete features, exhaustively trying all possible combinations of discrete values quickly becomes infeasible as the number of combinations grows.
Alternatives such as trying a subset of the possible combinations or resorting to continuous relaxations often leads to a failure to effectively optimize the AF which may result in sub-optimal BO performance.
As an alternative, we propose using PR to better optimize the AF, and we demonstrate that PR achieves strong performance on a large number of real-world problems.
Our approach is complementary to many other BO extensions, and combines seamlessly with, for example, trust region-based BO and specialized kernels for discrete parameters.
One limitation of PR is that it requires computationally-demanding MC integration.
However, given that the computation in PR is embarrassingly parallel, it motivates for future research on optimizing AFs on distributed hardware. Acknowledgements
We thank Ben Letham, James Wilson, and Michael Cohen, as well as the members of the Oxford
Machine Learning Research Group, for providing insightful feedback.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Appendix C. (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix to:
Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization
A Potential Societal Impacts
Our work advances Bayesian optimization, a generic class of methods for optimization of expensive, difﬁcult-to-optimize black-box problems.
With this paper in particular, we improve the performance of Bayesian optimization on problems with mixed types of inputs.
Given the ubiquity of such problems in many practical applications, we believe that our method could lead to positive broader impacts by solving these problems better and more efﬁciently while reducing the costs incurred for solving them.
Concrete and high-stake examples where our method could be potentially applied (some of which have been already demonstrated by the benchmark problems considered in the paper) include but are not limited to applications in communications, chemical synthesis, drug discovery, engineering optimization, tuning of recommender systems, and automation of machine learning systems.
On the ﬂip side, while the method proposed is ethically neutral, there is potential of misuse given that the exact objective of optimization is ultimately decided by the end users; we believe that practitioners and researchers should be aware of such possibility and aim to mitigate any potential negative impacts to the furthest extent.
B Theoretical Results and Proofs B.1 Results
Let P ( i ) Z :=
P ( Z ( i ) ) denote the set of probability measures on Z ( i ) for each i
= 1 , ..., d z , and let
P Z := (cid:81)
d
z i =1 P ( i ) Z .
For any α : X × Z → R , deﬁne ˜
α : X × P → R as [Equation] z
[Equation]
( x , [Equation] p ( { z [Equation]
Let Θ be a compact metric space, and consider the set of functionals Φ = { ϕ s.t.
ϕ : Θ → P Z } .
Let ˆ α ( x , θ ) : = ˜
[Equation] ( x , ϕ ( θ )) = (cid:90) Z
[Equation]
z ∈Z [Equation] [Equation] Since Z is ﬁnite, each element of ϕ ∈ Φ can be expressed as a mapping from Θ to R |Z| .
Namely, each ϕ ( θ ) corresponds to a vector with |Z| elements containing the probability mass for each element of Z under p ϕ ( θ ) .
Thus ( P Z , (cid:107) · (cid:107) ) is a metric space under any norm (cid:107) · (cid:107) on R | Z | .
Let α ∗ := max ( x , z ) ∈ ( X×Z )
α ( x , z ) and let H ∗ := arg max ( x , z ) ∈ ( X×Z ) α
( x , z ) denote the set of maximizers of α .
Lemma 1.
Suppose α is continuous in x for every z ∈ Z and that ϕ : Θ (cid:55)→ ( P Z , (cid:107)·(cid:107) ) is continuous with ϕ (Θ)
= P Z .
Let J ∗ :
= arg max ( x , θ ) ∈X×
Θ ˆ α ( x , θ ) .
Then for any ( x ∗ , θ ∗ ) ∈
J ∗ , it holds that ( x ∗ , z ) ∈
H ∗ for all z ∈ supp p ϕ
( θ ∗ ) .
Proof.
First, note that ˆ α : X × Θ → R is continuous (using that ϕ is continuous and α is bounded).
Since both X and Θ are compact ˆ α attains its maximum, i.e., J ∗ exists.
Let ( x ∗ , θ ∗ ) ∈
J ∗ .
Clearly, there exists z ∗ ∈ arg max z ∈Z α ( x ∗ , z ) such that α ( x ∗ , z ∗ ) =
α ∗ .
Suppose there exists z (cid:48) ∈ supp p ϕ θ ∗ ) such that ( x ∗ , z (cid:48) ) / ∈ H ∗ .
Then α ( x ∗ , z (cid:48) ) <
α ∗ and, since Z is ﬁnite, p ϕ ( θ ∗ ) ( { z (cid:48) } ) > 0 .
Consider the probability measure p (cid:48) given by p
[Equation] [Equation]
if z
[Equation]
p
[Equation]
otherwise
Then
[Equation]
Now p (cid:48) ∈
P Z , and so p (cid:48) =
ϕ ( θ (cid:48) ) for some θ (cid:48) ∈ Θ .
But then ˆ α ( x ∗ , θ (cid:48) )
>
ˆ α ( x ∗ , θ ∗ ) .
This is a contradiction.
Corollary 2.
Suppose the optimizer of g is unique, i.e., that H ∗ = { ( x ∗ , z ∗ ) } is a singleton.
Then the optimizer of ˆ α is also unique and J ∗ =
{ ( x ∗ , θ ∗ ) } , with p ϕ ( θ ∗ ) ( { z ∗ } ) = 1 .
Corollary 3.
Consider the following mappings:
• Binary: ϕ : [0 , 1] → P { 0 , 1 } with p ϕ ( θ ) ( { 1 } )
= θ and p
ϕ ( θ ) ( { 0 } ) =
1 − θ .
• Ordinal: ϕ : [0 , C − 1]
→ P { 0 , 1 ,...,C } with p ϕ ( θ ) ( { i } )
= (1 − | i − θ | ) 1 {| i − θ | ≤ 1 } for i = 1 , . . .
, C . •
Categorical: ϕ : [0 , 1] C → P { 0 , 1 ,...,C } with p ϕ ( θ ) ( { i } ) =
[Equation]
These mappings satisfy the conditions for Lemma 1.
In the setting with multiple discrete parameters where the above mappings are applied in component-wise fashion for each discrete parameter, the component-wise mappings also satisfy the conditions for Lemma 1.
Clearly, the mappings given in Corollary 3 are continuous functions of θ .
In the setting with multiple discrete parameters , the component-wise function is also continuous with respect to the distribution parameters for each discrete parameter.
Hence, the mappings satisfy the conditions for Lemma 1.
Lemma 2.
If ( x ∗ , z ∗ ) ∈ H ∗ = arg max ( x , z ) ∈X×Z
α ( x , z ) , then
[Equation]
Proof.
For any z ∗ , let θ ∗ be the parameters such that p ( z ∗ | θ ∗ )
= 1 (i.e. a point mass on z ∗ ).
From
Equation (2),
[Equation]
Claim:
[Equation]
Suppose there exists θ (cid:48)
such that E Z ∼ p
( Z | θ (cid:48) )
[ α ( x ∗ , Z )] > E Z ∼ p
( Z | θ ∗ )
[ α ( x ∗ , Z )] .
Since ( x ∗ , z ∗ ) ∈
H ∗ , α ( x ∗ , z ∗ ) = max ( x , z )
∈X×Z
α ( x , z ) .
Hence, there is no convex combination of values of α that is greater than α ( x ∗ , z ∗ ) .
This is a contradiction.
Theorem 1 (Consistent Maximizers) .
Suppose that α is continuous in x for every z ∈ Z .
Let H ∗ be the maximizers of α ( x , z ) : H ∗ = { ( x , z ) ∈ arg max ( x , z )
∈X×Z α ( x , z ) } .
Let J ∗ ⊆
X × Θ be the maximizers of E Z ∼ p
( Z | θ )
[ α ( x , Z )] : J ∗ = { ( x , θ ) ∈ arg max ( x , θ ) ∈X×
Θ E Z ∼ p
( Z | θ )
[ α ( x , Z )] } , where Θ is the domain of θ .
Let ˆ H ∗ ⊆
X × Z be deﬁned as: ˆ H ∗ =
{ ( x , ˜ z ) : ( x , θ ) ∈ J ∗ , ˜ z ∼ p ( Z | θ ) } .
Then, ˆ H ∗ =
H ∗ .
Proof.
From Lemma 1, we have that for any ( x ∗ , θ ∗ ) ∈
J ∗ , it holds that ( x ∗ , z ) ∈
H ∗ for all z ∈ supp p ϕ
( θ ∗ ) .
Hence, ˆ H ∗ ⊆
H ∗ .
Now, let ( x ∗ , z ∗ ) ∈
H ∗ .
Let θ ∗ ∈
Θ such that p ( z ∗ | θ ∗ ) = 1 .
From the proof of
Lemma 2, we have that E
Z ∼ p
( Z | θ ∗ )
[ α ( x ∗ , Z )]
= α ( x ∗ , z ∗ ) .
As in the proof of Lemma 2, there is no convex combination of values of α greater than α ( x ∗ , z ∗ ) .
So E Z ∼ p
( Z | θ ∗ )
[ α ( x ∗ , Z )]
= max ( x , θ ) ∈X×
Θ E Z ∼ p
( Z | θ )
[ α ( x , Z )] , and therefore, x ∗ , θ ∗ ∈ J ∗ .
Hence ( x ∗ , z ∗ ) ∈
ˆ H ∗ .
So
H ∗ ⊆
ˆ H ∗ , and hence, ˆ H ∗ =
H ∗ .
Lemma 3.
Suppose that α : ( x , z ) (cid:55)→ R is differentiable with respect to x for all z ∈ Z , and that the mapping ϕ : θ (cid:55)→ P Z is such that p ϕ ( θ ) ( { z } ) is differentiable with respect to θ for all z ∈ Z .
Then the probabilistic objective E Z ∼ p
( Z | θ )
[ α ( x , Z )] is differentiable with respect to ( x , θ ) .
Proof.
For any z ∈ Z , the function p ( z , θ ) α
( x , z ) =
p ϕ ( θ ) ( { z } ) α
( x , z ) is the product of two differentiable functions, hence differentiable.
Therefore the probabilistic objective is a (ﬁnite) linear combination of differentiable functions, hence differentiable.
Theorem 2 (Convergence Guarantee) .
Let α : X × Z
→ R be differentiable in x for every z ∈ Z .
Let (ˆ x t,m , ˆ θ t,m ) be the best solution after running stochastic gradient ascent for t time steps on the probabilistic objective E
Z ∼ p
( Z | θ )
[ α ( x , Z )] from m starting points with its unbiased MC estimators proposed above.
Let { a t } ∞ t =1 be a sequence of positive step sizes such that 0 < (cid:80) ∞ t =1 a 2 t =
A < ∞ and (cid:80) ∞ t =1 a t = ∞ , where a t is the step size used in stochastic gradient ascent at time step t .
Let ˆ z t,m ∼ p
( Z | ˆ θ t,m ) .
Then as t → ∞ , m → ∞ , and τ → 0 , (ˆ x t,m , ˆ z t,m ) → ( x ∗ , z ∗ ) ∈ arg max ( x , z ) ∈X×Z α ( x , z ) in probability.
Proof.
The binary and categorical mappings in Corollary 3 are differentiable in θ (the ordinal mapping is differentiable almost everywhere 4 ).
Since the acquisition function α : X × Z → R is differentiable in x for every z ∈ Z , this means that the PO is differentiable.
Using the prescribed sequence of step sizes, optimizing the PO using stochastic gradient ascent will converge almost surely to a local maximum after a sufﬁcient number of steps [47].
As we increase the number of randomly distributed starting points, the probability of not ﬁnding the global maximum of the PO will converge to zero [60].
From Theorem 1, the PO and the AF have the same set of maximizers.
Hence, convergence in probability to a global maximizer of the PO means convergence in probability to a global maximizer of the AF.
C Experiment Details
For each BO optimization replicate, we use N init =
min(20 , 2 ∗ d eff ) points from a scrambled Sobol sequence, where d eff is the “effective dimensionality” after one-hot encoding categorical parameters.
Unless otherwise noted, all experiments use 20 replications and conﬁdence intervals representstandard errors of the mean.
The same initial points are used for all methods for that replicate and different initial points are used for each replicate.
For each method we report the log 10 regret.
Since the optimal value is unknown for many problems, we set the optimal value to be f ∗ + 0 .
1 where f ∗ is the best observed value across all methods and all replications.
For constrained optimization f ∗ is the best feasible observed value and for multi-objective optimization f ∗ is the maximum hypervolume across all methods and replications.
In total, the experiments in the main text (excluding H Y BO and C ASMOPOLITAN ) ran for an equivalent of 2,009.82 hours on a single Tesla V100-SXM2-16GB GPU.
The baseline experiments (H Y BO and C ASMOPOLITAN ) ran for an equivalent of 745.10 hours on a single Intel Xeon Gold 6252N CPU.
C.1 Additional Problem Details
In this section, we describe the details of each synthetic problem considered in the experiments (the details of the remaining real-world problems are already described in Section 6.2).
Ackley.
We use an adapted version of the 13-dimensional Ackley function modiﬁed from Bliek et al.
The function is given by:
[Equation]
where in this case a = 20 , n = 0 . 2 , c = 2 π and d = 13 and x ∈ [ − 1 , 1] 13 .
We discretize the ﬁrstdimensions to be binary with the choice {− 1 , 1 } , and the ﬁnal 3 dimensions are unmodiﬁed with the original bounds.
Mixed Int F1.
Mixed Int F1 is a partially discretized version of the 16-dimensional Sphere optimization problem [28], given by:
[Equation]
where f opt is sampled from a Cauchy distribution with median = 0 and roughly 50% of the values between − 100 and 100 .
The sampled f opt is then clamped to be between [ − 1000 , 1000] and rounded to the nearest integer.
x opt is sampled uniformly in [ − 4 , 4] d , and in this case d = 16 .
We discretize the ﬁrst 8 dimensions as follows: the ﬁrst 2 dimensions are binary with 2 choices {− 5 , 5 } ; the nextdimensions are ordinal with 3 choices {− 5 , 0 , 5 } ; the next 2 dimensions are ordinal with 5 choices {− 5 , − 2 . 5 , 0 , 2 . 5 , 5 } ; the ﬁnal 2 dimensions are ordinal with 7 choices {− 5 , − 103 , − 53 , 0 , 53 , 103 , 5 } .
The remaining 8 dimensions are continuous with bounds [ − 5 , 5] 8 .
Rosenbrock.
We use an adapted version of the Rosenbrock function, given by:
[Equation]
where in this case d = 10 .
The ﬁrst 6 dimensions are discretized to be ordinal variables, withpossible values each x i ∈ {− 5 , 0 , 5 , 10 }∀ i ∈ [1 , 6] .
The ﬁnal 4 dimensions are continuous with bounds [ − 5 , 10] 4 .
Chemical Reaction (Direct Arylation Chemical Synthesis).
For this problem, we ﬁt a GP surrogate (with the same kernel used by the BO methods) to the dataset from Shields et al.
[51] (available at https://github.com/b-shields/edbo/tree/master/experiments/data/direct_ arylation under the MIT license) in order to facilitate continuous optimization of temperature and concentration.
The surrogate is included with our source code.
Oil Sorbent.
We set the reference point for this problem to be [ − 125 . 3865 , − 57 .
8292 , 43 . 2665] , which we choose using a commonly used heuristic to scale the nadir point (component-wise worst objective values across the Pareto frontier) [61].
C.2 Method details
PR, C ONT .
R ELAX .
, E XACT R OUND , PR + TR, and E XACT R OUND + STE .
We implemented all of these methods using BoTorch [1], which is available under the MIT license at https:// github.com/pytorch/botorch .
PR and PR + TR use stochastic minibatches of 128 samples and the probabilistic objectives are optimized via Adam using a learning rate of 140 .
The AFs of
C ONT .
R ELAX . , E XACT R OUND , E XACT R OUND + STE are deterministic and are optimized via L-BFGS-B—E XACT R OUND approximates gradients via ﬁnite differences [26]. All methods userandom restarts and are run for a maximum of 200 iterations.
We follow the default initialization heuristic in BoTorch [1], which initializes the optimizer by evaluating the acquisition function at a large number of starting points (here, 1024, chosen from a scrambled Sobol sequence), and selecting (20) points using Boltzmann sampling [19] of the 1024 initial points, according to their acquisition function utilities.
Combining PR with trust regions : When combining PR with the trust regions used in T U RBO we only use a trust region over the continuous parameters and discrete ordinals with at leastvalues.
While methods like C ASMOPOLITAN uses a Hamming distance for the trust regions over the categorical parameters, we choose not to do so as there is no natural way of efﬁciently optimizing PR using gradient-based methods.
Finally, we do not use a trust region over the Boolean parameters as the trust region will quickly shrink to only include one possible value.
We use the same hyperparameters as T U RBO [22] for unconstrained problems and SCBO [21] in the presence of outcome constraints, including default trust region update settings.
Casmopolitan : We use the implementation of C ASMOPOLITAN —which is available at https: //github.com/xingchenwan/Casmopolitan under the MIT licence—but modify it where appropriate to additionally handle the ordinal variables.
Speciﬁcally, the ordinal variables are treated as continuous when computing the kernel.
However, during interleaved search, ordinal variables are searched via local search similar to the categorical variables.
We use a set of C ASMOPOLITAN hyperparameters (i.e. success/failure sensitivity, initial trust region sizes and expansion factor) recommended by the authors.
We use the same implementation of interleaved search for the acquisition optimization comparisons.
HyBO :
We use the ofﬁcial implementation of H Y BO at https://github.com/aryandeshwal/
HyBO , which is licensed by the University of Amsterdam.
We use the default hyperparameters recommended by the authors in all the experiments, and we use the full H Y BO method with marginalization treatment of the hyperparameters as it has been shown to perform stronger empirically [14].
C.3 Gaussian process regression
When there are no categorical variables we use k ordinal which is a product of an isotropic Matern5/2 kernel for the binary parameters and a Matern-5/2 kernel with ARD for the remaining ordinal parameters.
In the presence of categorical parameters, this kernel is combined with a categorical kernel [48] k cat as k cat × k ordinal + k cat + k ordinal .
We use a constant mean function.
The GP hyperparameters are ﬁtted using L-BFGS-B by optimizing the log-marginal likelihood.
The ranges for the ordinal parameters are rescaled to [0 , 1] and the outcomes are standardized before ﬁtting the
GP.
C.4 Variance Reduction via Control Variates
As discussed in Section 4.2, we use moving average baseline for variance reduction. Speciﬁcally, the baseline is an exponential moving average with a multiplier of 0.7, where each element is the mean acquisition value across the N MC samples obtained while evaluating the probabilistic objective. C.5 Deterministic Optimization via Sample Average Approximation
Although multi-start stochastic ascent is provably convergent, an alternative optimization approach is to use common random numbers (i.e. a ﬁxed set of base samples) to reduce variance when comparing a stochastic function at different inputs by using the same random numbers.
The method of common random numbers leads to biased deterministic estimators that are lower-variance than their stochastic counterparts where random numbers are re-sampled at each step.
Such techniques have been used in the context of BO in settings such as efﬁciently optimizing MC acquisition functions [1] and for optimizing risk measures of acquisition functions under random inputs [10].
Sampling a ﬁxed set of points ˜
z 1 , ...,
˜ z N ∼ p
( Z | θ ) would be a poor choice because p ( Z | θ ) can vary widely during AF optimization as θ changes.
Therefore, instead sample from p ( Z | θ ) using reparameterizations provided in Table 4.
Speciﬁcally, we reparameterize Z as a deterministic function h ( · , · ) that operates component-wise on θ and the random variable U
=
( u (1) , ..., u ( d z ) ) , u ( i ) ∼
Uniform (0 , 1) :
Z = h ( θ , U ) .
That is, each random variable Z ( j ) , where j = 1 , ..., d z has a corresponding independent base random variable U ( j ) such that Z ( j ) =
h ( θ ( j ) , U ( j ) ) .
Using a ﬁxed a set of base samples { ˜ u i } Ni =1 , the samples of Z can be be computed as z i
= h ( θ , ˜ u i ) .
We note that even with ﬁxed base samples, the samples { z i } Ni =1 depends on θ , and hence, by using common base uniform samples, we obtain a deterministic estimator where the values of the samples ˜
z 1 , ...,
˜ z N can still vary with θ .
Under this reparameterization, our probabilistic objective can be written as
[Equation]
where under the reparameterizations in Table 4, U is a uniform random variable across the d zdimensional unit cube— P ( U )
=
Uniform (0 , 1) dz .
Under this reparameterization we can deﬁne our sample average approximation estimator of the probabilistic objective as
[Equation]
Our sample average approximation estimator of the gradient of the probabilistic objective with respect to θ is given by
[Equation]
Sample average approximation estimators are deterministic and biased conditional on the selection of base samples.
However, the reparameterizations in Table 4 create discontinuities in the PO, and the number of discontinuities increases with the number of MC samples.
Nevertheless, we ﬁnd that optimizing the PO using L-BFGS-B delivers strong performance on the benchmark problems and we compare against stochastic optimization in Figures 21 and 20.
As in the stochastic case, we reduce the variance further by leveraging quasi-MC sampling [42] instead of i.i.d.
sampling.
function h ( · , · ) .
[Table]
R ANDOM
[Table]
R EPARAMETERIZATION
( Z = h ( θ, U ))
[Table]
Z ∼
[Table]
ERNOULLI ( θ ) h ( θ, U ) =
1 ( U < θ )
[Table]
Z = (cid:98) θ (cid:99) + B , h ( θ, U ) = (cid:98) θ
(cid:99)
{++ 
+}
( U < θ − (cid:98) θ (cid:99) )
B ∼
[Table]
ERNOULLI ( θ − (cid:98) θ (cid:99) )
[Table]
ATEGORICAL
[Table]
[Table]
[Table]
(arg [Table] D Constrained and Multi-Objective Bayesian Optimization In many practical problems, the black-box objective must be maximized subject to V > 0 black-box outcome constraints f ( v ) c
( x , z ) ≥ 0 for v = 1 , ..., V .
See Gardner et al.
[24] for a more in depth review of black-box optimization with black-box constraints and BO techniques for this class of problems.
In the multi-objective setting, the goal is to maximize (without loss of generality) a set of M objectives f (1) , ..., f ( M ) .
Typically there is no single best solution, and hence the goal is to learn the Pareto frontier (i.e. the set of optimal trade-offs between objectives).
In the multi-objective setting, the hypervolume indicator is a common metric for evaluating the quality of a Pareto frontier.
See [20] for a review of multi-objective optimization.
E Comparison with Enumeration
When computationally feasible, the gold standard for acquisition optimization over discrete and mixed search spaces is to enumerate the discrete options and optimize any continuous parameters for each discrete conﬁguration (or simply evaluated each discrete conﬁguration for fully discrete spaces).
In Figures 4 and 5 we compare PR (optimized with Adam using stochastic mini-batches of 128 MC samples) and analytic PR (optimized with L-BFGS-B) against enumeration and show that
PR achieves log regret performance that is comparable to the gold standard of enumeration and does so in less wall time.
[Figure]
[Figure]
F Analysis of MC sampling in Probabilistic Reparameterization
The main text considers 1024 MC for PR.
We consider 128, 256, and 512 samples, in addition to the default of 1024.
For problems with discrete spaces that are enumerable, we also consider analytic PR.
We do not ﬁnd statistically signiﬁcant differences between the ﬁnal regret of any of these conﬁgurations (Figure 6).
Run time is linear with respect to MC samples, and so substantial compute savings are possible when fewer MC samples are used (Figure 7).
We observe comparable performance between PR with 1,024 MC samples and as few as 128 MC samples.
With 64 or fewer
MC samples, we observe performance degradation with respect to log regret in Figure 8, although wall time is considerably faster for fewer 64 or less MC samples as shown in Figure 9.
[Figure]
r
[Figure] [Figure]
We observe that wall time scales linearly with the number of MC samples, which is expected since we compute PR in N 32 chunks to avoid overﬂowing GPU memory.
[Figure]
[Figure]
[Figure]
[Figure] [Figure]
We examine the MC approximation error relative to analytic PR on the chemical reaction and ackley problems. The results in Figure 10 show the mean absolute percentage error (MAPE)
[Figure]
[Equation] evaluated over a random set of | X discrete | = | Θ discrete | 10 , 000 points from X × × (the sampled sets are denoted X discrete , Θ discrete ).
We observe a rapid reduction in MAPE as we increase the number of samples.
With 1024 samples, MAPE is 0.055% (+/- 0.0002 %) over 20 replications (different MC samples in PR) on the chemical reaction problem and MAPE is 0.018% (+/- 0.0003 %) on the ackley problem. With 128 samples, MAPE is 0.282% (+/- 0.0029 %) over 20 replications (different MC samples in PR) on the chemical reaction problem and MAPE is 0.052% (+/- 0.0021 %) on the ackley problem.
[Figure]
u
[Figure]
P e r c e n t
E rr o r
[Figure]
G Effect of τ in Transformation
Throughout the main text, we use τ = 0 . 1 , which we selected based on the observation that it provides a reasonable balance between retaining non-zero gradients of g ( φ ) with respect to φ and allowing θ to become close to 0 or 1 as shown in Figure 11.
[Figure]
As τ → 0 , the θ can take more extreme values, but the gradient of the transformation with respect to φ also moves closer to zero.
For larger values of τ , the gradient of the transformation with respect to φ is larger, but θ has a more limited domain with less extreme values.
We ﬁnd that τ = 0 .
1 is a robust setting across all experiments.
H Alternative methods H.1 Straight-through gradient estimators
An alternative approach to using approximating the gradients under exact rounding using ﬁnite differences is to approximate the gradients using straight-through gradient estimation (STE) [3].
The idea of STE is to approximate the gradient of a function with the identity function.
In our setting, the gradient of the discretization function with respect to its input is estimated using an identity function.
Using this estimator enables gradient-based AF optimization, even though the true gradient of the discretization function is zero everywhere that it is deﬁned.
Although STEs have been shown to work well empirically [Figure] these estimators are not well-grounded theoretically.
Their robustness and potential pitfalls in the context of AF optimization have not been well studied.
Below, we evaluate the aforementioned E XACT R OUND + STE approach and show that it offers competitive optimization performance (Figure 12) with fast wall times (Figure 13), but does not quite match the optimization performance of PR on several benchmark problems [Figure]
and report the log regret of the best feasible objective for Welded beam.
[Figure]
Welded Beam (5 constraints)
[Figure]
H. TR methods with alternative optimizers
In this section [Figure] we consider alternative methods to PR for optimizing AFs using within trust regions [Figure]
The results in Figure 14 show that PR is a consistent best optimizer using TRs [Figure] but that STEs work quite well with TRs in many scenarios. [Figure]
[Figure]
Welded Beam (5 constraints)
[Figure]
I Acquisition Function Optimization at a Given Wall Time Budget
In Figure
[Figure]
we provide additional starting points (64 points, rather than 20) to other non-PR methods in order to provide them with additional wall-time budget.
We ﬁnd that using PR with
MC samples, PR provides rapid convergence compared to other baselines and therefore is a good optimization routine for any wall time budget
[Figure]
J Alternative categorical kernels
In this section
[Figure]
we demonstrate that PR can be used with arbitrary kernels over the categorical parameters including those that require discrete inputs (which C ONT .
R ELAX . is incompatible with).
Speciﬁcally for the categorical parameters, we compare using (a) a Categorical kernel (default) versus with a Matérn-5/2 kernel with either (b) one-hot encoded categoricals, (c) a latent embedding kernel [67], or known embeddings based on density functional theory (DFT) [51].
For the latent embedding kernel, we follow Pelamatti et al.
[44] and use a 1-d latent embedding for categorical parameters where the cardinality is less than or equal to 3 and a 2-d embedding for categorical parameters where the cardinality is greater than 3.
For each latent embedding, we use an isotropic Matérn-5/ kernel and use product kernel across the kernels for the categorical, binary, ordinal, and continuous parameters.
For the kernel over DFT embeddings, we use the DFT embeddings for the direct arylation dataset from Shields et al.
[51], which are available at https://github.com/b-shields/edbo .
It is worth noting that in the Chemical Reaction problem, the black-box objective is a GP surrogate model with a Categorical kernel that is ﬁt to the direct arylation dataset. The purpose of this section is demonstrate that PR is agnostic to the choice of kernel over discrete parameters. Because the Chemical Reaction problem is based on a GP surrogate, we do not draw conclusions about which choice of kernel is best suited for modeling the true, unknown underlying Chemical Reaction yield function.
[Figure]
L og R e g r e t
[Figure]
K Alternative Acquisition Functions
In this section, we compare PR with expected improvement (EI) against PR with upper conﬁdence bound (UCB).
For UCB, we set the hyperparameter β in each iteration using the method in Kandasamy et al.
Although UCB comes enjoys bounded regret [52], we ﬁnd empirically that EI works better on most problems. [Figure] r e t Mixed Int F
[Figure]
Rosenbrock
[Figure]
Ackley
[Figure]
L Additional Results on Optimizing Acquisition Functions
In this section [Figure] we provide additional results on various approaches for optimizing acquisition functions using the same evaluation procedure as in the main text.
We use 50 replications
[Figure]
M Stochastic vs Deterministic Optimization
We compare optimizing PR with stochastic and deterministic optimization methods.
For stochastic optimizers, we compare stochastic gradient ascent (SGA) and Adam with various initial learning rates. For SGA, the learning rate is decayed each time step t by multiplying the initial learning rate by t − 0 . 7 and for Adam a ﬁxed learning rate is used.
For stochastic optimizers, the MC estimators of PR and its gradient stochastic mini-batches of N = 128 MC samples are used.
For deterministic optimization, base samples are kept ﬁxed.
All routines are run for a maximum of 200 iterations.
In Figure 20, we observe that Adam is more robust to the choice of learning rate than SGA and generally is the best performing method.
Furthermore, Adam consistently performs better than deterministic optimization.
We compare Adam with a learning rate of 140 against L-BFGS-B in Figure
[Figure]
and deterministic optimization methods.
The initial learning rate for stochastic gradient ascent is given in parentheses.
[Figure]
N Comparison with an Evolutionary Algorithm
In Figures 22 and 22, we compare against the evolutionary algorithm PortfolioDiscreteOnePlusOne, which is the recommended algorithm for discrete and mixed search spaces in the Nevergrad package [45].
We ﬁnd that PR signiﬁcantly outperforms this baseline by a large margin with respect to log regret, but is slower than the evolutionary algorithm with respect to wall time.
[Figure]
[Figure]