M ODEL - BASED C AUSAL B AYESIAN O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
How should we intervene on an unknown structural causal model to maximize a downstream variable of interest?
This optimization of the output of a system of interconnected variables, also known as causal Bayesian optimization (CBO), has important applications in medicine, ecology, and manufacturing.
Standard
Bayesian optimization algorithms fail to effectively leverage the underlying causal structure.
Existing CBO approaches assume noiseless measurements and do not come with guarantees.
We propose the model-based causal Bayesian optimization ( MCBO ) , an algorithm that learns a full system model instead of only modeling intervention-reward pairs.
MCBO propagates epistemic uncertainty about the causal mechanisms through the graph and trades off exploration and exploitation via the optimism principle.
We bound its cumulative regret, and obtain the first non-asymptotic bounds for CBO.
Unlike in standard Bayesian optimization, our acquisition function cannot be evaluated in closed form, so we show how the reparameterization trick can be used to apply gradient-based optimizers.
Empirically we find that MCBO compares favorably with existing state-of-the-art approaches.
I NTRODUCTION
Many applications, such as drug and material discovery, robotics, agriculture, and automated machine learning, require optimizing an unknown function that is expensive to evaluate.
Bayesian optimization (BO) is an efficient framework for sequential optimization of such objectives (Mo ˇ ckus, 1975).
The key idea in BO is to quantify uncertainty in the unknown function via a probabilistic model, and then use this to navigate a trade-off between selecting inputs where the function output is favourable (exploitation) and selecting inputs to learn more about the function in areas of uncertainty (exploration).
While most standard BO methods focus on a black-box setup (Figure 1 a), in practice, we often have more structure on the unknown function that can be used to improve sample efficiency.
In this paper, we exploit structural knowledge in the form of a causal graph specified by a directed acyclic graph (DAG).
In particular, we assume that actions can be modeled as interventions on a structural causal model (SCM) (Pearl, 2009) that contains the reward (function output) as a variable (Figure 1 b).
While we assume the graph structure to be known, we consider the functional relations in the SCM as unknown.
All variables in the SCM are observed along with the reward after each action.
This Causal BO setting has important potential applications, such as optimizing medical and ecological interventions (Aglietti et al., 2020b).
For illustrative purposes, consider the example of an agronomist trying to find the optimal Nitrogen fertilizer schedule for maximizing crop yield, described in Figure 1.
There, the concentration of Nitrogen in the soil causes its concentration in the soil at the later timesteps.
To exploit the causal graph structure for optimization, we propose model-based causal Bayesian optimization (MCBO).
MCBO explicitly models the full SCM and the accompanying uncertainty of all SCM components.
This allows our algorithm to select interventions based on an optimistic strategy similar to that used by the upper confidence bound algorithm (Srinivas et al., 2010).
We show that this strategy leads to the first CBO algorithm with a cumulative regret guarantee.
For a practical algorithm, maximizing the upper confidence bound in our setting is computationally more difficult, because uncertainty in all system components must be propagated through the entire estimated SCM to the reward variable.
We show that an application of the reparameterization trick allows MCBO to be practically implemented with common gradient-based optimizers.
Empirically,
MCBO achieves competitive performance on existing CBO benchmarks and a related setting called function network BO (Astudillo & Frazier, 2021b).
[Figure]
Causal Bayesian Optimization
[Figure]
a 1 . . .
a 3 are possible interventions involving adding Nitrogen fertilizer to the soil.
Contributions
• We introduce MCBO, a model-based algorithm for causal Bayesian optimization than can be applied with very generic classes of interventions.
• Using MCBO we prove the first sublinear cumulative regret bound for CBO.
We show how the bound scales depending on the graph structure.
We demonstrate that CBO can lead to a potentially exponential improvement in cumulative regret, with respect to the number of actions, compared to standard BO.
• By an application of the reparameterization technique, we show how our algorithm can be efficiently implemented with popular gradient-based optimizers.
• We evaluate MCBO on existing CBO benchmarks and the related setting of function network BO.
Our results show that MCBO performs favorably compared to methods designed specifically for these tasks.
B ACKGROUND AND P ROBLEM S TATEMENT
We consider the problem of an agent interacting with an SCM for T rounds in order to maximize the value of a particular target variable.
We first introduce SCMs, before describing the kinds of interventions an agent can perform on an SCM.
Structural Causal Models An SCM is described by a tuple ⟨G , Y, X , F , Ω ⟩ .
G is a known DAG,
Y is the reward variable to be maximized; X = { X i } m − 1 i =0 is a set of observed random variables; the set F = { f i } mi =0 defines the functional relations between these variables; and Ω = { Ω i } mi =0 is a set of independent noise random variables with zero-mean and known distribution.
Each is assumed to be b i subgaussian.
If Ω i is multivariate, we assume each component is mutually independent.
We use Y and X m interchangeably.
For notational convenience we also use pa i ⊂ { 0 , . . .
, m } to denote the set of indices of the parents of the i th node in G , and Z i = {
X j } j ∈ pa i to denote the set of parent variables of the i th variable according to
G .
We overload notation and sometimes use X i to refer to the i th node, as well as the i th random variable.
The elements of X are ordered according to a valid topological ordering (so X 0 is a root and X m is a leaf).
We use lowercase is used to denote a realization of the corresponding random variable.
Each observation x i ∈
X i ⊂ R d is defined over a compact set for all i ∈ { 0 , . . .
, m − 1 } .
We define the space of all non-reward variables with X = (cid:81) m − 1 i =
X i .
The reward is a scalar x m ∈ R .
We denote the set of all possible values for the parents of X i as Z i =
(cid:81) j ∈ pa i X j .
Each X
i is generated according to the function
f
i : Z i → X i , taking the parent nodes Z i of X i as input: x i =
f i ( z i ) + ω i .
Interventions At each round of interaction, the agent can interact with the SCM by performing an intervention.
In this work, we consider two types of intervention model.
We consider a soft intervention model (Eberhardt & Scheines, 2007) where interventions are parameterized by a set of controllable action variables.
Let A i ⊂ R q denote the compact space of each action a i and A be the space of all actions a = { a i } mi =0 .
We represent actions as additional nodes in
G : a i is a parent of X i (and hence an additional input to f i ) if and only if i = j .
Since f i is unknown, in our soft intervention model the agent does not know apriori the functional effect of a i on X i .
A simple example of a soft intervention is a shift intervention, which results in x i
=
f
i
( z i , a i ) + ω
i =
g i ( z i ) + a i + ω
i for some function g i .
A shift intervention might occur in our example of adding Nitrogen fertilizer to soil and then measuring the total soil Nitrogen concentration.
While our theoretical results will focus on data obtained via these unknown soft interventions, our experiments additionally consider two other types of data.
We also consider a hard intervention model.
Hard interventions (often referred to as do -interventions) modify the targeted variable to a specific distribution, independently of the variable’s parents.
An example might be a doctor setting the dosage of a patient’s medication, which fixes the dosage to a specific value (Aglietti et al., 2020b).
We will introduce the rest of the problem setup under the soft intervention model, before adapting it for the hard intervention model.
A special case in both the above models is the collection of observational data, which is when no intervention is performed on the system.
In the soft intervention model, not intervening on node i is equivalent to setting a i = 0 .
An example would be not applying any Nitrogen fertilizer to the soil.
In practice, the agent may have access to some previous observational data before it’s first round of interaction with the system.
Constraints on interventions In many applications, there exist several variables
X i that cannot be intervened upon at all.
For example, the soil starting conditions in Figure 1 (b).
In that case, we fix the corresponding a i to 0 .
Among intervenable nodes, we often cannot intervene on all simultaneously.
For example, a farmer may only have capacity to apply fertilizer at 2 out of 3 of the possible time windows.
The example of a cardinality constraint can be written as:
[Equation]
Problem statement We consider the problem of an agent sequentially interacting with an SCM, with known G and a fixed but unknown set of functions F with f i : Z i × A i
→ X i .
We add an additional subscript to our notation to show the round of interaction.
At round t we select actions a : ,t = { a i,t } mi =0 and obtain observations x : ,t = { x i,t } mi =0 such that
[Equation]
If i corresponds to a root node, z i,t will denote an empty vector and the output of f i only depends on a i,t .
Since we cannot intervene on the target variable X m , we fix a m = 0 .
This results in the reward evaluation:
[Equation]
that implicitly depends on the whole intervention a : ,t .
The best the agent can do in any given interaction is to select the action that maximizes the expected reward:
[Equation]
where, unless otherwise stated, expectations are taken over noise ω .
Performance metric Our agent’s goal is to design a sequence of interventions { a : ,t } Tt =0 that achieve high average expected reward.
We study this convergence via the cumulative regret (Lattimore & Szepesv´ari, 2020) over a time horizon T :
[Equation]
A sublinear growth rate of R T with T implies vanishing expected regret R T /T
→ 0 as T → ∞ .
As an alternative to cumulative regret, one can also study convergence through other notions such as simple regret E [ y | a ∗ ] − E
[ y | a T ] .
The most appropriate metric depends on the application.
In the
Nitrogen fertilizer example, cumulative regret might be preferable because we care about obtaining high crop yields across all years, not just in only one year.
Index notation Let x i,t
=
[ x i,t, 1 , . . .
, x i,t,d ] T denote a vector where x i,t,l indicates indexing the component
l ∈
[ d ] of the t th timepoint of the observations at the node i ∈ [ m ] .
For functions with vector output, e.g., f i : Z i → X i , we sometimes consider notation with additional input to f i that indicates the output dimension.
We write f i ( z , a ) =
[ f i ( z , a , 1) , . . .
, f i ( z , a , d )] T .
Regularity assumptions We consider standard smoothness assumptions for the unknown functions f i : S → X i defined over the compact domain S (Srinivas et al., 2010).
In particular, for each node i ∈ [ m ] , we assume that f i ( · ) belongs to a reproducing kernel Hilbert space (RKHS)
H k i , a space of smooth functions defined on
S =
Z i × A i .
This means that f i ∈
H k
i is induced by a kernel function k
i : ˜ S × ˜ S
→ R where ˜ S = S ×
[ d ] 1 .
We also assume that k i ( s, s ′ ) ≤ 1 for every s, s ′ ∈
˜
S 2 .
Moreover, the RKHS norm of f i ( · ) is assumed to be bounded ∥ f i ∥
k i ≤ B i for some fixed constant
B i > 0 .
Finally, to ensure the compactness of the domains Z i , we assume that the noise ω is bounded, i.e., ω i ∈
[ − 1 , 1] d .
Problem statement under hard interventions Under a hard intervention model, instead of selecting a the agent must select a set of intervention targets I ∈ I ⊂ P ([ m − 1]) and their values a
I ∈
A I .
Eq.
(2) then takes the form of
[Equation]
Here f i is still unknown and modeled using the same regularity assumptions.
Constraints can be placed on either I or a I .
Observational data corresponds to I = ∅
2. R ELATED W ORK
Optimal decision-making in SCMs has been the subject of several recent works e.g., in the bandit setting (Lattimore et al., 2016; Bareinboim et al., 2015).
Aglietti et al.
(2020b) introduce causal
BO (CBO) focusing on hard interventions.
CBO considers unobserved confounding and uses the do-calculus to estimate Y given I, a I using both observational data and interventional data with the same intervention targets I .
Aglietti et al.
(2020a) extend CBO to make use of data obtained from hard interventions with different intervention targets.
While both methods use do-calculus to estimate causal effects, they do not learn the full system model.
Branchini et al.
and Alabed & Yoneki (2022) extend these works to explore the CBO setting in the case of unknown DAG.
Function network BO (FNBO) (Astudillo & Frazier, 2021b) is similar to CBO setup where each function takes as input the output of its parent nodes and EI-based acquisition is used.
MCBO further generalizes the FNBO setup to a richer class of problems since the causal model formalism of MCBO allows for hard interventions while FNBO does not.
FNBO also assumes that the system is noiseless which might be a restrictive assumption in practice.
Kusakawa et al.
(2021) study stochastic function networks in the special case of a chain graph.
Similar to MCBO, they develop a UCB-based acquisition function but use a different approach for computing confidence bounds.
Since a UCB acquisition function is not the main focus of Kusakawa et al.
(2021), there is no empirical study of this method.
Both FNBO and CBO are part of a wider research direction on using intermediate observations from the computation of the unknown function to improve the sample efficiency of BO algorithms (Astudillo & Frazier, 2021a).
The function class studied in this paper is similar to that of deep Gaussian processes (GPs) (Damianou & Lawrence, 2013) in that MCBO models Y as a composition of GPs given a .
Deep GPs, however, do not make use of intermediate system variables and do not compose GPs according to a causal graph structure.
Our use of the reparameterization trick to practically implement an upper confidence bound acquisition function (Srinivas et al., 2010) in MCBO is inspired by Curi et al.
(2020), who apply ideas from BO for sample efficient reinforcement learning.
A LGORITHM
In this section, we propose MCBO, its statistical model and the acquisition strategy.
We first introduce MCBO under the soft intervention setup and then describe how to adapt it to hard interventions.
Statistical model We use GPs for learning the RKHS functions f 0 , . . .
, f m from observations.
Our regularity assumptions permit the construction of confidence bounds using these GP models with priors associated with the RKHS kernels.
We refer to Rasmussen (2003) for more background on the relation between GPs and RKHS functions.
For all i ∈ [ m ] , let µ i, 0 and σ 2 i, 0 denote the prior mean and variance functions for each f i () , respectively.
The corresponding posterior GP mean and variance, denoted by µ i,t ( · ) and σ 2 i,t ( · ) respectively, are computed based on the previous evaluations D t
= { z : , 1: t , a : , 1: t , x : , 1: t } .
In particular, for each function f
i ( · , l ) defined by the given kernel k i ( · , · ) and output component l :
[Equation]
where vec ( x i, 1: t ) =
[ x i, 1 , 1 , x i, 1 , 2 , . . .
, x i,t,d ] ⊤ and for ( t 1 , l ) , ( t 2 , l ′ ) ∈
, ( t, d )] : [ K t ]
[Equation]
[Equation]
The identity matrix is denoted by I , and b i is the variance proxy for ω i .
We write µ i,t =
[ µ i,t ( · , 1) , . . . , µ i,t ( · , d )] T and similarly for σ .
We give more background on the posterior updates of vector-valued GPs in Appendix A.1.
At time t , the known set M t of statistically plausible functions ˜
F = { ˜ f i } mi =0 that lie inside the confidence interval is defined by the posterior of the GP:
[Equation]
Here, β i,t is a parameter that ensures the validity of the confidence bounds.
Some examples of such concentration inequalities under similar regularity assumptions as well as explicit forms for β i,t can be found in Chowdhury & Gopalan (2019) and Srinivas et al.
In the following, we set β i,t = β t for all i such that the confidence bounds are still valid.
Acquisition function At each round t , interventions are selected based on maximizing an acquisition function.
We use an upper confidence bound acquisition function, which is defined as:
[Equation] [Equation]
Require: Parameters { β t } t ≥ 1 , Ω , prior means µ i, 0 = 0 , kernel functions k i, 0 ∀ i ∈ [0 , . . .
, m ] 1: for t = 1 , 2 , . . .
do 2:
Construct confidence bounds as in Eq.
Select a t ∈ arg max a ∈A max η ( · ) E
[ y | { ˜ f } , a ] as in Eq.
Observe samples { z i,t , x i,t } mi = 5:
Use D t to update posterior { µ i,t ( · ) , σ 2 i,t ( · ) } mi =0 as in Eqs.
(7) and (8) 6: end for
Require: Parameters { β t } t ≥ 1 , Ω , prior means µ i, 0 = 0 , kernel functions k i, 0 ∀ i ∈ [0 , . . .
, m ] 1: for t = 1 , 2 , . . .
do 2:
Construct confidence bounds as in Eq.
Select I, a I ∈ arg max I, a I max η E [ y | { ˜ f } , do ( X I = a I )] 4:
Observe samples { z i,t , x i,t } mi = 5:
Use D t to update posterior { µ i,t ( · ) , σ 2 i,t ( · ) } mi =0 as in Eqs.
(7) and (8) 6: end for
Thus, we optimistically pick interventions that yield the highest expected return among all system models that are still plausible given past observations.
Eq.
(10) is not amenable to commonly used optimization techniques, due to the maximization over a set of functions with bounded RKHS norm.
Therefore, following Curi et al.
(2020), we use the reparameterization trick to write any ˜ f i ∈ ˜ F ∈ M t using some function
η i : Z i ×
A i
→
[ − 1 , 1] d i as
[Equation]
where observations passed through one of the plausible models, and not necessarily the true model f , are denoted as ˜ x i = ˜
f i (˜ z
i , ˜ a i ) + ˜ ω i .
◦ denotes elementwise multiplication of vectors.
This allows for rewriting our acquisition function in terms of η : Z × A
→
[ − 1 , 1] |X| :
[Equation]
Intuitively, the variables η allow for choosing optimistic but plausible models given the confidence bounds.
The η can parameterized by, for example, a neural network and then standard optimization techniques applied.
For the theory, we assume access to an oracle that can find the global optimum for Eq.
In practice, such an oracle may be computationally infeasible due to the non-convexity of Eq.
We discuss the heuristics that we use for approximating this oracle in the appendix.
Algorithm 1 summarizes our Model-based Causal BO approach.
We note that, for the special case of the SCM following the DAG of Fig. 1(a), our algorithm and the associated guarantees reduce to standard BO (cf.
Srinivas et al., 2010).
Hard interventions MCBO also naturally generalizes to hard interventions, cf.
Algorithm 2.
In our experiments, we perform the combinatorial optimisation over I by enumeration, because |I| is not large for the instances we consider.
To perform this enumeration more efficiently we ignore intervention sets not contained in the minimal intervention set as defined by (Lee & Bareinboim, 2019).
We apply the notion of a minimal intervention set from Lee & Bareinboim (2019) to prune sets of intervention targets that contain redundant interventions, resulting in a smaller set to optimize over.
T HEORETICAL A NALYSIS
This section describes the convergence guarantees for MCBO under a soft intervention model, showing the first sublinear cumulative regret bounds for causal BO.
We start by introducing the assumptions required for this analysis.
Assumption 1 ( Functional relations ) .
All f i ∈ F are L f -Lipschitz continuous and, for all i ≥ 0 , the elements of the noise vector ω
i are i.i.d.
b i -sub-Gaussian.
Assumption 2 ( Continuity ) .
For all i , the functions µ i and σ i are L µ and L σ Lipschitz continuous.
Assumption 3 ( Calibrated model ) .
All statistical models are calibrated w.r.t.
F , so that with σ i,t ( · ) there exists a sequence β t ∈ R > 0 such that, with probability at least (1 − δ ) , it holds jointly for all i , and t ≥ 0 and z i , a
i ∈ Z i ×A i that |
f
i
( z i , a i ) − µ i,t − 1 ( z i , a i ) | ≤ β t σ i,t − 1 ( z i , a i ) , element-wise.
Assumption 1 follows directly from assuming bounded noise and the regularity assumptions.
Assumption 2 holds if the RKHS of each f i has a Lipschitz continuous kernel (see Curi et al.
Appendix G).
This means that Assumption 2 restricts the convergence guarantees to soft interventions that affect their target variable in a smooth way, meaning that our analysis does not apply to the case of hard interventions.
We nevertheless experimentally demonstrate the effectiveness of MCBO in non-smooth settings, such as CBO with hard interventions.
Assumption 3 holds when we assume that the i th GP prior uses the same kernel as the RKHS of f i and that β t is sufficiently large to ensure the confidence bounds in Eq.
(9) hold.
In the DAG G over nodes { X i } mi =0 , let N denote the maximum distance from a root node to X m : N = max i dist( X i , X m ) where dist( · , · ) is measured as the number of edges in the longest path from a node X i to the reward node X m .
Let K denote the maximum number of parents of any variable in G : K = max i | pa ( i ) | .
The following theorem bounds the performance of MCBO in terms of cumulative regret.
Theorem 1.
Consider the optimization problem in Eq.
(4) with SCM satisfying Assumptions 1– 3 where G is known but F is unknown.
Let { a t ∈ A , z t ∈ Z} t> 0 denote the sequence of actions chosen by MCBO and the parent variable realizations, respectively.
Suppose additionally that
A i is compact ∀ i .
Then for all T ≥ 1 , with probability at least 1 − δ , the cumulative regret of Algorithmis bounded by
[Equation]
Here, γ T = max i γ i,T where the node-specific γ i,T denotes the maximum information gain at a time T commonly used in the standard regret guarantees for Bayesian optimization (Srinivas et al., 2010).
This maximum information gain is known to be sublinear in T for many common kernels, such as linear and squared exponential kernels, resulting in an overall sublinear regret for MCBO.
We refer to Appendix A.2.3 for the proof.
Theorem 1 demonstrates that the use of the graph structure in MCBO results in a potentially exponential improvement in how the cumulative regret scales with the number of actions m .
Standard
Bayesian optimization as in Fig.
1 (a), that makes no use of the graph structure, results in cumulative regret exponential in m (Srinivas et al., 2010), when using a squared exponential kernel.
When all
X i in MCBO are modelled with squared exponential kernels that model output components independently, we have a greatest maximum information gain of γ T
= O
( d ( Kd + q )(log T )
Kd + q +1 ) , resulting in a cumulative regret that is exponential in K and exponential in N .
However, note that m ≥ K + N .
For several common graphs, the exponential scaling in N and K could be significantly more favorable than exponential scaling in m .
Consider the case of G having the binary-tree-like structure in appendix Fig.
3 (binary tree), where N = log( m ) and K = 2 .
In such settings, the cumulative regret of MCBO will have only polynomial dependence on m .
We give further discussion of the bound in Theorem 1 for specific kernels in Appendix A.2.3 and discuss the dependence of β T on T in Appendix A.2.4.
[Figure]
E XPERIMENTS
In this section
[Figure]
we empirically validate MCBO on six problems taken from previous CBO or function network papers (Aglietti et al., 2020b; Astudillo & Frazier, 2021b).
The DAGs corresponding to each task are given in Fig.
3 of the appendix.
We provide an open-source implementation of
MCBO 4 .
Baselines We compare our approach with three baselines (i) Expected Improvement for Function
Networks (EIFN) (Astudillo & Frazier, 2021b); (ii) Causal Bayesian Optimization (EICBO) (Aglietti et al., 2020b); and (iii) standard upper confidence bound (UCB) (Srinivas et al., 2010) which models the objective given interventions with a single GP (see Fig.
1 a).
To enable a fair comparison, we only apply EICBO to the hard intervention setting, since it was designed specifically for a hard intervention model.
Experimental setup We report the average reward as a function of the number of system interventions performed.
The average reward at time T is defined by (cid:80)
Tt =
E ω
[ Y | a t ] and is directly inversely related to cumulative regret in that high average expected reward is equivalent to low cumulative regret.
This matches the performance metric studied in our analysis.
Average expected reward and cumulative regret are natural metrics for many real applications, like crop management, in which we want consistently high yield, and the healthcare-inspired setting we study in the experiments, in which we seek good treatment for more than a single patient.
In the appendix, we show experiments measuring the best expected reward of any action previously chosen, which is more similar to an inverse of the simple regret.
We report mean performance over 20 random seeds, with error bars showing ± σ/ √ 20 where σ is the standard deviation across the repeats.
All figures that are referenced but not in the main paper can be found in the appendix.
For the guarantees in Theorem 1 to hold, { β t } Tt =0 must be chosen so that the model is calibrated at all time-steps as in Eq.
In practice, we select a single β such that β t = β , ∀ t .
Choosing β too pessimistically will result in high regret,
Toy Experiment First
ToyGraph is a hard intervention CBO task where I = {∅ , { 0 } , { 1 }} .
All methods start with 10 observational samples and 2 samples from different random interventions on each I ∈
I .
When EICBO obtains interventional data, it obtains noiseless samples because it is not designed for the noisy setting.
By noiseless samples, we mean that for action a EICBO observes
E ω
[ Y | a ] .
Other methods obtain single samples from the distribution Y, X | a .
In ToyGraph, I = { 1 } is the optimal target, but to efficiently learn the optimal a { 1 } the agent must generalise from both observational data and interventional data on other targets I = { 0 } .
EICBO models Y given a I with a separate GP for every I and is consequently only able to make use of observational data and interventional data with the same target.
Since MCBO learns a full system model, it incorporates all observations into the learned model, even when interventions do not match.
Figure 2 (a) shows that the average reward of MCBO (Algorithm 2) is favourable compared to the baselines.
Both baselines are built on expected improvement, which will continue to explore even after high reward solutions are found.
This explains the non-monotonic average reward of EICBO.
Healthcare Experiment PSAGraph is inspired by the DAG from a real healthcare setting (Ferro et al., 2015) and also benchmarked by Aglietti et al.
(2020b).
The agent intervenes by prescribing statins and/or aspirin, while specifying the dosage, to control prostate-specific antigen (PSA) levels.
Here I = {∅ , { 2 } , { 3 } , { 2 , 3 }} and all interventions are hard intervention.
Initial sample sizes are the same as for ToyGraph.
Figure 2 (b) again shows EICBO having nonmonotonic average reward and strong comparable performance of MCBO.
Noiseless Function Networks
In addition to the hard intervention setting, we evaluate MCBO and the baselines on the four tasks from Astudillo & Frazier (2021b).
All systems have up to six nodes and varying graph structures.
In function networks, actions can affect multiple system variables and system variables can be children of multiple actions.
Function networks are deterministic, so ω
= 0 .
MCBO (Algorithm 1) can be applied directly in this setting and the guarantees are also easily transferable.
Like in Astudillo & Frazier (2021b), there are no constraints (besides bounded domain) on actions, and the agent is initialized with 2 A + 1 samples from random actions, where
A is the number of action nodes.
Figure 2 (c) and Figure 4 (a,b,c) show that MCBO achieves competitive average reward on all tasks.
EIFN is better on Dropwave.
Meanwhile, MCBO is substantially better than EIFN on the Ackley and Rosenbrock tasks.
Overall, there are not sufficiently many tasks established in the literature to conclusively say which properties might make a task favor EIFN over MCBO.
However, this would be interesting to understand in future work and likely relates to the wider conversation in
BO comparing expected improvement and UCB algorithms (Merrill et al., 2021).
We find that the naive UCB approach, which does not use the graph structure, generally performs poorly, especially on problems with larger graphs like Alpine2.
On Ackley, EIFN does not achieve monotonically improving average reward.
Since it is also based on EI and has strong performance in terms of best expected reward, we suspect this can be explained by similar reasons as for EICBO.
Noisy Function Networks We modify three of the function networks settings to include an additive zero-mean Gaussian noise at every system variable, making ω non-zero.
EIFN is designed for deterministic function networks and has no convergence guarantee in this setting.
Results in terms of average (Figure 2 d,e,f) and best reward (Figure 6 g, h, i) are comparable to the noiseless case, with MCBO and EIFN both performing well compared to UCB.
C ONCLUSION
This paper introduces MCBO, a principled model-based approach to solving Bayesian optimization problems over structural causal models.
Our approach explicitly models all variables in the system and propagates epistemic uncertainty through the model to select interventions based on the optimism principle.
This allows MCBO to solve global optimization tasks in systems with known causal structure significantly more efficiently than prior work.
We prove the first theoretical convergence guarantees for an algorithm solving the CBO problem and demonstrate that its theoretical advantages are reflected in strong empirical performance.
Future work might consider how to apply the method to large graphs, where the sets of all possible discrete intervention targets cannot be efficiently enumerated.
A A PPENDIX M ODEL - BASED C AUSAL B AYESIAN OPTIMIZATION A. B ACKGROUND ON GP S
Here we give more background on the vector-valued GP models we use.
We will drop the i -node index and consider the modelling of a single function with scalar output f : A × Z → R in RKHS
H k with kernel k : S × S → R where S =
( Z × A ) .
We assume measurement noise with variance proxy b .
Assuming prior mean and variance µ 0 , σ 0 and dataset
D
[Equation] [Equation]
t
[Equation] [Equation]
} where x t ∈ R , we get a posterior GP mean and variance given by updates
[Equation]
14) where ( K
[Equation]
and I is the identity matrix.
This follows the standard GP update given in Rasmussen (2003).
In this work, we model functions with vector-valued outputs f : A×Z → X ∈ R d (we will continue to drop the i index).
For this we follow Chowdhury & Gopalan (2019) and use a scalar-output GP that takes the component of the output vector under consideration as an input.
That is, we model f ( · , l ) (the l th output component of the function under study) where f = [ f ( · , 1) , . . .
, f ( · , d )] .
We again assume that f is from RKHS H k but with kernel k : ˜ S × ˜ S
→ R where ˜ S
=
( Z × A ×
[ d ]) .
Assuming prior mean and variance µ 0 , σ 0 and dataset
D t = { z 1: t , a 1: t , x 1: t } where x t ∈ R d , we get a posterior GP mean and variance given by updates
[Equation]
where vec ( x 1: t ) =
[ x 1 , 1 , x 1 , 2 , . . .
, x t,d ] ⊤ and for ( t 1 , l ) , ( t 2 , l ′ ) ∈
, ( t, d )] : [
[Equation]
This follows the posterior update of Chowdhury & Gopalan (2019).
The key idea is to use a single scalar-output GP with kernel k modeling all output components, but introduce the component index as part of the input space.
This requires introducing the notation vec ( x 1: t ) and ordering all observations, of all time points and all components, in a single index, since the GP update for component l considers observations of all other components.
Under this GP model the components of f need not be independent if kernel k is designed to model dependency between the components.
In our work, we use one of these vector-valued GP models for each individual random variable in our causal model, leading to the reintroduction of the additional i index in Eqs.
(7) and (8).
A. P ROOFS FOR THE T HEORETICAL ANALYSIS
Our analysis closely follows Curi et al.
(2020), particularly the proofs in their Appendix D, where they prove similar guarantees for a model-based reinforcement learning problem.
In contrast to Curi et al.
(2020), which models RL transition dynamics with a single GP for all timesteps
, MCBO uses independent GPs for modelling the functional relation { f 1 , · · · , f m } and uses the causal graph G to determine the input and output variables of these functions.
This section is organized as follows.
In Appendix A.2.1, we discuss the notion of model complexity Γ T introduced in for RL setting (Curi et al., 2020) and applicable beyond the GP setting discussed in the main paper.
We then bound the cumulative regret in terms of Γ T in Appendix A.2.2.
Finally,
Appendix A.2.3, we prove the main result of the paper Theorem 1 that makes use of the previous analysis.
The norm notation ∥·∥ refers to ℓ 2 -norm if no additional notation is given.
A.2. M ODEL COMPLEXITY
The number of samples needed to learn a low-regret action is related to the number of samples needed to learn all GP models in our SCM.
This is demonstrated in the classic BO setting in Srinivas et al. (2010).
We quantify the model complexity of our entire model class M T as
[Equation]
This measure of model complexity closely relates to the maximum information gain γ T used for proving regret guarantees in BO (Srinivas et al., 2010).
For a single GP model, γ T is the maximum information gain about the unknown f that can be obtained from noisy evaluations of the f at fixed inputs (see Eq.
Later we show that Γ T can be bounded by a sum of the information gains for all m GPs.
It is worth noting that Equation 17 may be a loose notion of model complexity because it assumes we can independently choose every z i , but for many graphs, there could be overlap in the z i , z j for i ̸
= j (two nodes could have a shared parent).
A.2. A NALYSIS IN TERMS OF GENERAL MODEL COMPLEXITY Γ T
In this section we will prove a theorem similar to Theorem 1 but in terms of the model complexity we define in Eq.
Note that this version of the theorem does not require that µ and σ come from a GP model with independent outputs, but any model such that Assumptions 1–3 are satisfied.
In later sections, when using a GP model we bound Γ T in terms of the maximum information gain of the m GPs used to get Theorem 1.
We will use the function Σ i,t ( · ) to represent a matrix of all zeros except σ i,t ( · ) giving the values of the diagonal.
Theorem 2.
Consider the optimization problem in Eq.
(4) with SCM satisfying Assumptions 1– where G is known but F is unknown.
Let { a i,t ∈
A i , z i,t ∈
Z i } i,t> 0 denote the set of actions chosen by MCBO and parents realizations of node i , respectively.
Then, for all T ≥ 1 , with probability at least− δ , the regret of Algorithm 1 is bounded by
[Equation]
For the proof, we first show that with, high probability, there exists some set of functions η that allows the reparameterized plausible SCM model in Eq.
(11) to match the true SCM (Lemma 1).
Recall the mechanism of the ground-truth SCM in Eq.
[Equation]
and the mechanism of the optimistic SCM model using the reparameterization of Eq.
[Equation] [Equation]
In Lemma 2, Lemma 3 and Corollary 1 we bound the instantaneous regret (regret at some specific timepoint t ) by bounding the difference in SCM output, for the same action input, assuming the true
SCM vs the optimistic reparameterized SCM.
Then in Lemma 5 we use the result of Lemma 4 to show that our bound on instantaneous regret implies a bound on cumulative regret.
In Lemma 1, for convenience we will drop the explicit dependence of all quantities on t .
Lemma 1.
Assume some fixed set of actions a is chosen at any timepoint t .
Under Assumption 3, for any x generated by the true SCM Eq.
(2) , with probability at least 1 − δ there exists a set of functions η = { η i } mi =0 , where η i : Z i × A i
→ [ − 1 , 1] d , such that x = ˜ x
if ∀ i ω i
=
˜ ω
i .
Proof.
Since ω = ˜ ω we only need to prove that there exists some η such that ∀
i f i ( z i , a i ) =
µ i ( ˜ z i , ˜ a i ) + β Σ i ( ˜ z i , ˜ a i ) η i .
By Assumption 3, with probability 1 − δ , for all i = 0 , . . .
, m , we have an elementwise bound |
f i ( z i , a i ) − µ i ( z i , a i ) | ≤ β i σ i
( z i,t , a i,t ) .
Thus for each z i,t , a i,t there exists a vector η i with values in [ − 1 , 1] d such that f i ( z i , a i ) =
µ i ( z i , a i ) + β Σ i ( z i , a i ) η i .
Note that this is not quite what we need because the RHS contains z i and not ˜
z i .
We will now use an inductive argument on i that constructs each η i sequentially from i = 0 to m .
Base case: we must prove that for i =
we have f 0 ( z 0 , a 0 )
=
µ 0 (˜ z 0 , ˜ a 0 )
+ β t Σ
(˜ z 0 , ˜ a 0 ) η 0 .
We know z 0 = ˜
z 0 (both are the empty vector) and a = ˜ a by the assumption of a fixed action.
Then there exists some vector η 0 such that f 0 ( z 0 , a 0 )
=
µ 0 ( z 0 , a 0 ) + β Σ 0 ( z 0 , a 0 ) η 0 = µ
(˜ z 0 , ˜ a 0 )
+ β Σ 0 (˜ z 0 , ˜
a 0 ) η 0 .
Let η 0 ( · ) be the function that outputs the vector η 0 given input z 0 , a 0 and the base case is proven.
Now assume the inductive hypothesis: ∀ j < i we have f j ( z j , a j ) = µ j (˜ z j , ˜ a j ) + β Σ j (˜ z j , ˜ a j ) η j .
We want to show that this implies f i ( z i , a i ) =
µ i (˜ z i , ˜ a i ) + β Σ i (˜ z i , ˜ a i ) η i .
We know a i =
˜
a i by the assumption of a fixed action.
z i = ˜
z i because ˜ z i =
[˜ x pa i [1] , . . .
, ˜ x pa i [ | pa i | ] ] T and we selected each η j such that ˜ x
j = x j .
Then there exists some vector η i such that f i (
z i , a i ) =
µ i ( z i , a i ) + β Σ i ( z i , a i ))
η i = µ i (˜ z i , ˜ a i ) + β Σ i (˜ z i , ˜ a i ))
η i .
Let η i ( · ) output the vector η i given input z i , a i and the inductive step is proven.
Lemma 2.
Under Assumption 3, with probability at least (1 − δ ) ∀ t ≥
the instantaneous regret r t is bounded by
[Equation]
Proof.
The result follows directly from,
[Equation]
This is true by the definition of ˜ F t , a : ,t as the argmax of Eq.
(12) and that with probability at least (1 − δ )
we have F ∈ M T .
We now show how the observations under the true and optimistic dynamics differ for a fixed action a and fixed noise sequence ˜ ω
= ω .
Lemma 3.
Under Assumptions 1–3, let ¯ L f,t = 1 + L f + 2 β t L σ .
Then, for all iterations t > 0 , any functions η i : R p i ×
R q i
→ [ − 1 , 1] d i for all i and any sequence of ω
i with ˜ ω
i = ω
i , a ∈ A we have
[Equation]
Proof.
We prove by induction on i .
Base case.
Consider the base case i = 0 .
Because the nodes are topologically ordered we will have pa 0 = {} .
Its realization therefore depends only on the chosen action.
Formally, we assume z 0 = ∅ , x 0 = ˜ f
( z 0 , a 0 ) + ω 0 and ˜ x 0 = ˜ f
(˜ z 0 , ˜ a 0 )
+ ˜ ω 0 .
Since ω 0 ,t = ˜ ω 0 ,t , ∥ x 0 ,
[Equation]
[Equation]
In the following, we omit the dependence on the action a , e.g., using f i ( z i,t ) instead of f i ( z i,t , a i ) since we assume the actions to be the same for the process generating x
i,t and ˜ x i,t .
Induction step.
Now assuming that ∥ x i − 1 ,t
[Equation]
we prove a similar result for the i th node.
[Equation]
[Table]
[Equation]
where 1 follows the dynamics Eqs.
(2) and (11).
In 2 , we assume the noise to be equal and add and subtract the same term.
In 3 and 4 , we reorder terms and apply the triangle inequality.
In 5 and 6 , we rely on the calibrated uncertainty and Lipschitz dynamics, then collect terms and use diagonality of the matrix Σ i,t − 1 ( · ) .
In 7 and 8 , we add and subtract the same term and use the Lipschitz continuity of σ i,t − 1 .
Finally, in 9 we add 1 to ensure that when we later upper bound this term by taking the exponential of it.
10 applies the triangle inequality.
11 follows the inductive hypothesis, and 12 is due to the depth of at least one parent j being N j = N i − 1 .
Now we will reintroduce the dependence of quantities on a and since a was general, we substitute in the actual actions chosen at time t : a : ,t .
Corollary 1.
Under the assumptions of Lemma 3, for any sequence of η i ∈ [ − 1 , 1] d i , θ ∈ D , and t ≥ 1 we have that
[Equation]
Proof.
This follows from Lemma 3.
Y is just the final state in the network so
[Equation]
Lemma 4.
Under Assumption 3, let L Y,t =
β
t ¯ L Nf,t .
Then, with probability at least (1 − δ ) it holds for all t ≥ 0 that
[Equation]
Proof.
[Equation]
Last
Now we bound cumulative regret R T .
Lemma 5.
Under the assumption of Assumptions 1–3, with probability at least (1 − δ ) it holds for all t ≥ 0 that
[Equation]
Proof.
[Equation]
where 1 is due to Jensen’s inequality and 2 follows Lemma 4.
Similar to the equivalent lemma in
Curi et al.
(2020), this bound is dependent on the data observed by the iteration t , making it hard to interpret in a general case.
To this end, we further provide the worst-case bound dependent on the model complexity Γ T .
Lemma 6.
Under Assumptions 1–3, let z i,
t ∈
Z t , Z i,t − 1 ⊆
Z i,t , and a i,t ∈
A i for all i, t > 0 with compact sets
Z i,t and A i,t .
Then, with probability at least (1 − δ ) it holds for all t ≥ 0 that
[Equation]
33) where
[Equation] [Equation]
Proof.
Since z i,t ∈
Z i we have
[Equation] [Equation]
and the result follows.
Taking square roots and substituting in for L Y,T in terms of β T , L f and L σ in Eq.
(33) concludes the proof for Theorem 2.
A.2. P ROOF OF T HEOREM
We can bound Γ T in Theorem 2 to get a bound that depends on the specific GP model used for each f i .
We can show that for many commonly used kernels MCBO achieves sublinear (in T ) regret.
Theorem 1.
Consider the optimization problem in Eq.
(4) with SCM satisfying Assumptions 1– 3 where G is known but F is unknown.
Let { a t ∈ A , z t ∈ Z} t> 0 denote the sequence of actions chosen by MCBO and the parent variable realizations, respectively.
Suppose additionally that
A i is compact ∀ i .
Then for all T ≥ 1 , with probability at least 1 − δ , the cumulative regret of Algorithmis bounded by
[Equation]
Proof.
Step 1.
Some preliminaries relating mutual information I T ( x i, 1: T , f i, 1: T ) and maximum information gain γ T
In the following we consider the information gain for the node i , i.e., for x i, 1: T ∈ R d × T and f i, 1: T =
[ f i ( z i, 1 , a i, 1 ) , . . .
, f i ( z i,T , a i,T )]
⊤ evaluated at points A i
= { z i, 1: T , a i, 1: T } , A i ∈
Z i × A i .
In this step, for simplicity, when clear from context we will omit the i -notation in the derivation for the information gain.
Mutual information I T ( x 1: T , f 1: T ) is then defined as:
[Equation]
where H ( · ) denotes entropy.
For fitting the GPs, the following modelling assumptions are used: x t | f t ∼ N ( f t ( z t , a t ) , b I ) and x t | x 1: t − 1 , z 1: t , a 1: t ∼ N ( µ t −
[Equation]
( z t , a t ) , b I + Σ t − 1 ( z t , a t )) , where I ∈ R d × d , and
[Equation]
Our setup assumes that the components { x t,l } dl =0 are independent of each other given z t , a t .
There- fore from Srinivas et al.
we know that the mutual information for the i th GP model is:
[Equation] [Equation]
because per component:
[Equation]
Then, by the definition of maximum information gain ˜ γ i
[Equation]
T
[Equation]
l per node i in the graph:
[Equation]
Accordingly, we can write the maximum information gain between x i, 1: T and f i, 1: T as follows:
[Equation]
The upper bounds on γ i,T,l are provided in Srinivas et al.
(2010) for widely used kernels and scale sublinearly in T .
We use p i =
| pa ( i ) | d to represent the size of the z-input to a GP.
For the linear kernel γ i,T,l = O (( p i + q ) log T ) , and in case of the squared exponential kernel γ i,T,l = O (( p i + q )(log T ) p i + q +1 ) .
Therefore, the upper bounds for ˜ γ i,T scale with dimensionality d into account, e.g., for the linear kernel γ i,T =
O ( d i ( p i + q ) log T ) , and in case of the squared exponential kernel γ i,T = O (
[Equation]
p i +
[Equation]
(log
[Equation]
i + q +1 ) .
Step 2.
A bound for model complexity Γ T .
Here we bound the model complexity Eq.
[Equation]
For readability
[Equation]
in the following
[Equation]
we denote max A ( · ) :
[Equation]
where 1 bounds A and Z with a box.
2 is from the max over a sum.
3 is due to the assumption of x i,t being independent of A j , j ̸
= i , conditioned on A i . 4 is due to Jensen’s inequality.
5 is due to the fact that for any s 2 ∈ [0 , b − 2 ] we can bound s 2 ≤
[Equation]
Srinivas et
[Equation] [Equation]
This also holds for function s 2 ( · ) :
= b − 2 i σ 2 i, ( t − 1) ,l ( · ) since b − 2 i σ 2 i, ( t − 1) ,d ( · ) ≤ b −
i
k
( · , · ) ≤ b − 2 i because k i ( · , · ) < 1 ∀ i (bounded kernel assumption).
6 is due to Eqs.
(36) and (39).
Finally, in 7 we define γ T :=
max i ˜ γ
i,T being the maximum value of the maximum information gains over graph nodes.
Step 3.
Sublinearity w.r.t.
T of maximum information gain γ T .
Finally, we show that the upper bounds on γ T scale sublinearly in T .
That follows from ˜ γ i,T scaling sublinearly for most of the popularly used kernels.
In particular, a linear kernel leads to γ T =
O ( d ( Kd + q ) log T ) and a squared exponential kernel leads to γ T
= O
( d ( Kd + q )(log T )
Kd + q +1 ) (assuming output components are independent) since max i p i
=
Kd .
Since γ T scales sublinearly in T , so does the cumulative regret bound in Theorem 1.
A.2. D EPENDENCE OF β T ON T FOR P ARTICULAR K ERNELS
Note that for Assumption 3 to hold under our RKHS assumptions, β T might depend on T .
For a single observed variable indexed by i at time t , for Assumption 3 to hold we must have β T = ˜
O (cid:0)
B i + b i d √ γ i,t (cid:1) (Chowdhury & Gopalan (2019) equation 9).
For Assumption 3 to hold at time t for all i we can apply a union bound and see that it is sufficient for β T =
˜ O (cid:0) B + bd √ γ t (cid:1) where
B = max i B i and b = max i b i .
In Theorem 1, β T appears raised to the power N .
For γ T corresponding to the linear kernel and squared exponential kernel, this will still lead to sublinear regret regardless of N because γ T will be only logarithmic in T .
However, for a Mat´ern kernel, where the best known bound γ T
=
O ( p ( pT ) c log ( pT )) with 0 < c < 1 , the cumulative regret bound will not be sublinear if
N and c are sufficiently large.
It is, however, suspected that the upper bound on γ T for the Mat´ern kernel is loose (Scarlett et al., 2017).
This phenomena with the Mat ´ ern kernel is not new and appears in the guarantees of Curi et al.
(2020) which use GP models in model-based reinforcement learning.
A. M AXIMIZING THE A CQUISITION F UNCTION
As discussed, our theoretical results assume access to an oracle that can maximize Eq.
Here we discuss how we approximate this oracle in practice.
In noiseless settings, instead of parameterizing each η i as a neural network, we can parameterize it as a constant.
This is because with no noise, the inputs to η i ( z i and a i ) given a are fixed.
This keeps the space of parameters to optimize over small, meaning we can use an identical optimization procedure to that used in EIFN by Astudillo & Frazier (2021b) which is also an out-of-the-box optimizer in the BoTorch package Balandat et al.
For noisy settings where each η i is a neural network, we use our own optimizer.
For each initialization of η parameters, we perform stochastic gradient descent to optimize the η i parameters and a .
We can do this because Eq.
(12) is differentiable with respect to both a and the parameters of each η i .
After many repeated random initializations we will have many solution candidates.
We select the one with the highest acquisition function value.
We use a large number of different random initializations because the acquisition function may be very non-convex.
Other approaches, such as those considered in Curi et al.
(2020) for model-based reinforcement learning, could also be adapted to optimize our acquisition function.
When parameterizing each η i with a neural network, we always use a two layer feed-forward network with a ReLu non-linearity, To map the output into [ − 1 , 1] we put the output of the network through an element-wise Sigmoid.
On all noisy environments we estimate the expectation in the acquisition function (Eq.
(12)) using a Monte Carlo estimate with 32 repeats for each gradient step.
For noisy Dropwave we userepeats because the environment is particularly noisy compared to other noisy environments.
A. E XPERIMENTAL S ETUP
Here we give additional notes to explain the experimental setup and ensure full replicability.
The cross-validation for selecting β is performed across β = { 0 . 05 , 0 . 5 , 5 } .
For a given performance metric and task, for example average reward on Dropwave, we select the β that on average performs best across all other tasks for the same performance metric, called β ∗ .
We then report the results in terms of average reward for running the algorithm (UCB or MCBO) on Dropwave with β = β ∗
For MCBO and EIFN, we use identical values for all shared hyperparameters (e.g. GP kernels) and use the original hyperparameters from Astudillo & Frazier (2021b).
CBO methods do not have hyperparameters comparable with these methods because CBO methods do not model individual
ToyGraph
X X
Y
PSAGraph
X
X X
X X
Y
Dropwave aa
X
Y
Alpine aaaaaa
X
X X
X X
Y
Rosenbrock aaaaa
X X
X
Y
Ackley aaaaaa
X
X
Y binary tree aaaaaa
X
X X
X X
X
Y system observations (see Section 2.1).
We run the original EICBO source code effectively unmodified Aglietti et al.
(2020b).
For CBO tasks, if observational data is selected, we give the agent, for any algorithm, a single observational sample.
This is different to in Aglietti et al.
(2020b) where 20 observational samples are given.
There is no noisy version of Ackley because adding noise can make the environment unstable by producing very large or very small rewards.
For all environments and all X i , the noisy environment adds unit-Gaussian noise, except Dropwave where we scale this noise by 0 .
1 for reasons of stability.
When applying MCBO to function networks we allow a m to be nonzero, to match the function networks setting.
A. F URTHER E XPERIMENTAL A NALYSIS
Average expected reward In Figure 4 we show the deterministic function networks plots not in the main paper.
In Figure 5 we show the average reward for all β considered in the cross-validation.
[Figure]
[Figure]
Best expected reward Here we also report the best expected reward as a function of the number of rounds T .
The best expected reward at time T is defined by
[Equation]
This is similar to but not directly inversely related to simple regret.
Simple regret algorithms require an inference procedure to select a final action.
This procedure could be computationally expensive for the CBO setting.
For example, a reasonable choice would be to report a final action based upon maximizing a lower confidence bound of the objective, however a closed form expression for this does not exist for CBO.
Best expected reward instead assumes access to an oracle that computes the
[Figure]
highest expected reward of any action the algorithm has previously chosen.
Our plotting of the best expected reward is consistent with previous work on CBO (Aglietti et al., 2020b).
When performing the same cross-validation procedure as for the average expected reward case, we find that the performance of MCBO varies drastically across task.
This is shown in Figure 6.
Selecting β is known to be a difficulty with UCB-based method (Merrill et al., 2021).
Figureshows the performance in terms of both average and best reward for all 3 possibilities for β .
When selecting just the best beta among the 3 tried, we find that MCBO is very strong on all tasks (with the exception of best score on ToyGraph, where it underperforms the baselines).
The optimal β can vary by an order of magnitude or more across tasks in our case.
This is likely because all of our settings have very different functional relationships and graph structures.
As discussed in Section 4, this can be understood from our theoretical analysis.
The plots of average and best scores for other values of β (Figs. 5 and 7) suggest that β can be made larger for MCBO to trade-off exploration vs. exploitation depending on whether simple or cumulative regret is more of a priority.
Meanwhile, expected improvement methods are focused almost almost entirely on exploration.
Runtimes In Figure 8 we show the total runtime of each method for all 100 rounds.
EIFN and
MCBO run on equivalent hardware, which has 4 times more ram than the hardware used for UCB and EICBO.
MCBO generally has a much longer runtime than EIFN in noisy settings where η s are parameterized by neural networks.
In general in BO, because the unknown function is expensive to evaluate, we are less concerned about the time required to optimize the acquisition function.
Our implementation for noisy settings could likely be sped-up with more parallelism to make it com-
[Figure] [Figure]
parable to EIFN runtimes.
In noiseless settings, where the optimization methods used are roughly equivalent between EIFN and MCBO, the two methods have comparable runtimes
[Figure]
Non-monotonic regret of EICBO In Aglietti et al.
(2020b)
ToyGraph and PSAGraph are run with much larger initial sample sizes.
We replicated the experiments with the same initial samples sizes as in Aglietti et al.
(2020b)
and find the same qualitative results (non-montonicity) for the average reward case.
We use the smaller initial sample sizes in our paper to make the problem more difficult.