Paper: Adversarial Causal Bayesian Optimization (ICLR-2024)

Reviewer comment (Novelty):
I don't see any fundamental difference between the studied model and a standard bandit or Bayesian optimization problem, where part of the model is stochastic and part of it is controlled by an adversary. Therefore, apart from having a causal model in the story, the novelty of the contribution seems limited.

Author response:
**I don't see any fundamental difference between the studied model and a standard bandit or Bayesian optimization problem, where part of the model is stochastic and part of it is controlled by an adversary.**

We respectfully disagree with the reviewer and we hope that the points discussed below can further clarify the novelty of our approach. In particular, we believe there is significantly novelty compared to previous Bayesian optimization methods:
- Because some part of the model is adversarial, the learner must randomize its actions to achieve no regret. Although standard adversarial online learning algorithms (such as multiplicative weights) exist for this, they require full-information feedback which is not available in Bayesian optimization. 
- The only Bayesian optimization approach that can provably achieve no regret in such a setting is GP-MW. However, this algorithm does not build a causal graph and can thus be highly sample inefficient, as we show.
- In light of the above, CBO-MW is the first Bayesian optimization algorithm which can: 1) achieve no regret using 2) the learning of a causal reward model.  Similar to GP-MW, CBO-MW also computes optimistic estimates for the full information feedback. However, because such estimates are obtained through computing a counterfactual using a causal graph, we provide a practical subroutine to compute these (optimistic) counterfactuals in an efficient way (section 4.3).
- We provide a regret guarantee and can show significantly improved rates compared to GP-MW  (section 5)
- We show how to implement CBO-MW in a distributed way to improve the computational efficiency (section 6) and demonstrate good empirical performance (section 7). 

We hope the above points clarify the reviewer’s concerns. We are happy to expand more based on the reviewer’ feedback.

Reviewer comment (Theory):
It is mentioned in the problem statement that the adversary does not know the action to be performed by the agent. Could you explain what the choice of the adversary's action is based on? If the worst-case analysis is applied here, does it matter whether the adversary know the agent's action or not since the adversary will always act in the worst way anyhow? Or does the adversary choose the worst action based only on history actions? But then do they know the agent's algorithm or not? The assumption that they don't know the agent's action seems a bit odd.

Author response:
**Could you explain what the choice of the adversary's action is based on?**

From the text: “We assume the adversary does not have the power to know $a_{:, t}$ when selecting $a′_{:,t}$, but only has access to the history of interactions until round $t$.” Additionally, yes, we assume the adversary can know the agent’s algorithm (we’ll add a note to make this clear in the text). Since the agent can play a mixed strategy, the adversary can at best know the distribution the agent will select actions from at each round. This is not as strong of an adversary as knowing the agent’s exact actions, and makes the problem more intricate as the learner can achieve no regret if it’s mixed strategy is updated suitably (as we propose in the CBO-MW algorithm). We remark that this is a standard setup for the adversary in adversarial online learning.

If the adversary at time $t$ could see the agent’s action $a_t$ before selecting $a_t’$, it would be a Stackelberg game. We could also model this in our framework by directly modeling the adversary’s action as an observation $X_i$, since it is a direct reaction to the action we chose. We describe this in appendix A.1.1.

Reviewer comment (Novelty):
Could you explain the difference between your model and a model that combines stochanstic and adversary bandit?

Author response:
**Could you explain the difference between your model and a model that combines stochanstic and adversary bandit?**

Here we request further information from the reviewer. It is not clear to us what the point of comparison is because it is not obvious what it means to just combine a stochastic and adversarial bandit. We think if the claim is that the contribution is limited, it would be easier for us to give a helpful response if the reviewer 1) refers to specific works in the literature and then 2) describes how the delta between our work and these is trivial. We think this would allow us to more concretely discuss the contribution.

Reviewer comment (Novelty):
The related work ignores causal bandit literature?

Author response:
**“The related work ignores causal bandit literature?”**
After the related work sentence “Agliettiet al.(2020) propose the first CBO setting with hard interventions and an algorithm that uses the do-calculus to generalise from observational to interventional data”, we will add “The CBO line of work builds off the causal bandit setting [cite Lattimore], which similarly incorporates causal graph knowledge into the bandit setting usually considering discrete actions with categorical observations or linear mechanisms with continuous observations [cite Varici]..”.

Reviewer comment (Writing):
The graph notations are confusing. The typical graph has 1 root node. But in causal graphs, we may have multiple nodes without parents.

Author response:
**“The graph notations are confusing. The typical graph has 1 root node. But in causal graphs, we may have multiple nodes without parents.”**
We were not able to follow this comment. If you could expand on it we are happy to take a look. Based on one of your later comments (addressed below), we think you might be getting root and leaf node mixed up.

Reviewer comment (Theory):
The adversary cannot see the action taken by agent before taking its action. Is this adversary weak? You have considered that the agent can see adversary's action before choosing their own action set, but what about vice-versa?

Author response:
**“The adversary cannot see the action taken by agent before taking its action. Is this adversary weak? You have considered that the agent can see adversary's action before choosing their own action set, but what about vice-versa?”**
If the adversary at time $t$ could see our action $a_t$ and then respond with $a_t’$, it would be a Stackelberg game and not a simultaneous action game. Our setting, algorithm, and guarantees can actually also model this Stackelberg game setting. Since the adversary would respond directly to our action $a_t$, we could model the adversary action as a function of $a_t$ (and therefore treat it as just another $X$ in the graph). We discuss this in appendix A.1.1. 

**“You have considered that the agent can see adversary's action before choosing their own action set”**
This is not true. We consider a setting where the agent selects $a_t$ simultaneously with the adversary selecting $a_t’$, so the agent does not observe the adversary action beforehand. We believe that this is unambiguous in the paper from section 2 under “Problem Statement”.

Reviewer comment (Theory):
The additive term Beta^{N+1} does seem high.

Author response:
**“The additive term Beta^{N+1} does seem high.”**
After the regret guarantee we compare our regret guarantee with a non-causal approach and give a strong case for why we have a significant improvement. We are also to our knowledge the first to give any kind of guarantee at all for this particular setting (besides the guarantee you get from applying GP-MW which uses no causal information).

Reviewer comment (Novelty):
Note that "Causal Bandits for Linear Structural Equation Models" Varici et al 2023 show that the regret scales as length of the longest causal path in the graph for linear SCMs, whereas you consider N - length of path to root node. The former (not cited in your work), seems tighter.

Author response:
**“Note that "Causal Bandits for Linear Structural Equation Models" Varici et al 2023 show that the regret scales as length of the longest causal path in the graph for linear SCMs, whereas you consider N - length of path to root node. The former (not cited in your work), seems tighter.”**

We think that it is not meaningful to directly compare the guarantees from our paper and Varici et al.. Here are the two key differences in the setups:

- They consider linear mechanisms. We consider much more general models for each mechanism where they can be modeled with a GP (come from an RKHS with bounded norm). Linear models are closed under composition, so for their model the reward is still a linear function of the actions. For our setting, GPs are not closed under composition, so we get this emergent complexity where the reward as a function of actions is a much more complicated function class than just a single GP (it is a deep GP [1]). 
- They consider the stochastic setting and therefore a much “easier” notion of regret than the one we consider in the adversarial/non-stationary case (see equation 3). 

[1] Damianou, Andreas, and Neil D. Lawrence. "Deep gaussian processes." Artificial intelligence and statistics. PMLR, 2013.

We compare our regret bound to the only baseline we know of that can achieve a guarantee in this setting (GP-MW) and give an argument for why our guarantee is favorable. Varici et al 2023 considers a less general problem and while we think it is a great result, it is for a different setting to our result. 

We’ll add a citation of Varici et al 2023 to the related work (see comment above on causal bandits).

Reviewer comment (Theory):
Is the assumption of usage of only finite action spaces chosen from continuous Reals_[0,1] feasible? If we draw an epsilon net over [0,1], then the computation complexity of Alg 1 may rise.

Author response:
**“Is the assumption of usage of only finite action spaces chosen from continuous Reals_[0,1] feasible? If we draw an epsilon net over [0,1], then the computation complexity of Alg 1 may rise.”**

Let's say that for each $\cal{A_i}$ we have the same finite number of possible actions $K$. Now you are correct that the complexity of Algorithm 1 will become large because the computational complexity is order of $K^m$, since we need to update a weight for every possible action combination. This is the point of D-GP-MW which we introduce in the paper. This algorithm also achieves a regret guarantee (section 6 plus associated appendix), and has computational complexity only linear in $K$ (if we have a separate agent controlling each $a_i$).  As a specific example we apply the algorithm to the SMS setting in the experiments and it works well.

Reviewer comment (Experiments):
There are exponential combination of adversarial choices, for each of which a counterfactual computation may be taken up. This is computationally demanding.

Author response:
**“There are exponential combination of adversarial choices, for each of which a counterfactual computation may be taken up. This is computationally demanding.”**

This is not true. In algorithm 1, at time $t$ we compute a counterfactual *fixing* $a_t’$ and therefore only iterate over $\cal{A}$ (not $\cal{A’}$). See section 4.2: “...is the counterfactual of what would have happened, in expectation over noise,had $a_t′$  remained fixed but the algorithm selected $a$ instead of $a_t$”. Therefore increasing the size of the adversary’s action space has no impact on the computational complexity of the algorithm.

Reviewer comment (Novelty):
The main weakness is the naming of the method, which refers to CBO (Aglietti et al 2020) and hereby its relationship with the CBO setting. The paper claims to show a generalization of CBO, which it seems to be an *algorithm* proposed in Aglietti et al 2020 as a solution to the "Causal Global Optimization" (CGO) problem. Here, in the abstract but also in the main paper, (1) no mention to the CGO problem is made (2) CBO seems to refer to the "setting" (somehow as a replacement to CGO), while the algorithm proposed here is CBO-MW. But CBO is not a setting, as said, it's an algorithm to solve the CGO under certain assumptions. In CBO, there are intervention *sets* and intervention *values/levels* (continuous-valued). I cannot see any of these here throughout, so it's unclear whether interventions on multiple variables here are excluded or what. In general, are you trying to also solve the CGO problem (a suitably modified version under external interventions, of course) or not? Again on this point, the authors claim to compare using "previous CBO benchmarks", but there are no benchmarks actually from the CBO paper of Aglietti et al 2020, and the CBO-CW is actually *not* compared (nor experimentally, nor methodologically with a discussion) to CBO itself. There is also no "causal prior" associated to the GP as in CBO.

Author response:
We thank the reviewer for their review. 

**Relation to CBO**

We use ‘causal Bayesian optimization’ (CBO) to describe the general causal Bayesian optimization problem, which is separate from the specific algorithm proposed by Aglietti et al.  There have been several works studying this CBO setting since the original work of Aglietti et al. These works consider multiple variations of their setting: including soft intervention models (instead of hard interventions) and the cumulative regret metric (instead of the simulated regret metric). These are technical differences within a conceptually very similar set of problems. 

**The paper claims to show a generalization of CBO, which it seems to be an algorithm**

Thank you for pointing out that in Aglietti et al. the authors specifically distinguish between causal global optimization (CGO) as their setting, and causal Bayesian optimization (CBO) as their algorithm. Therefore when specifically describing the setting of Aglietti et al 2019 we will refer to it as CGO in our paper too. We will however continue to refer to the general problem of global optimization in the presence of a causal graph and prior over mechanisms as causal Bayesian optimization. We will also refer to the setting of our paper as ACBO. This is consistent with the wider BO literature where BO - global optimization with a prior - is used to describe a setting and usually the algorithm is described based on the specific acquisition function e.g. UCB, UEI etc. We note that the paper of MCBO, a baseline we compare to, also describes their setting as ‘CBO’. The main difference here is whether you consider the prior as part of the setting or the algorithm. In Aglietti et al. they use a prior as part of their solution to the CGO problem, but in many other parts of the BO literature, the prior is considered part of the problem setting. 

**...CBO, there are intervention sets and intervention values/levels**

CBO as in Aglietti et al. considers a hard intervention model. As discussed in the setup we consider a soft intervention model as in MCBO. Our model includes interventions on multiple variables. This explains the difference to what you read in Aglietti et al. Again we see this as more of a technical difference. 

**the authors claim to compare using "previous CBO benchmarks", but there are no benchmarks actually from the CBO paper of Aglietti et al 2020**

Firstly, CBO assumes hard interventions so cannot be directly applied at least to the experimental settings we consider. We compare against MCBO on the synthetic experiments. In the MCBO paper, MCBO is compared to CBO with hard interventions in the stochastic BO setting under the cumulative regret metric, and MCBO performs favorably. We say “adversarial versions of previously studied CBO benchmarks”, meaning that we modify some of the environments from the MCBO paper (a paper about CBO) for the adversarial setting. 

We hope the above points clarify the reviewer’s concerns and any misunderstandings. We are happy to expand more on this based on the reviewer’ feedback.

Reviewer comment (Novelty):
My understanding then from the above is that this work is actually not an extension of CBO at all (although I would like to hear from the authors), rather an extension of GP-MW which is mentioned a lot and compared to experimentally.

Author response:
Thank you for your response and constructive feedback.

**"it is confusing to say you are generalizing CBO throughout."**

We only explicitly use this wording in the abstract and it is true in the context of those sentences, since we do consider a more general version of "an agent intervenes on a structural causal model with known graph but unknown mechanisms to maximize a downstream reward variable". In the main body of the paper we believe that we give the more nuanced view that you are after:
"Aglietti et al.(2020) propose the first CBO setting with hard interventions and an algorithm that uses the do-calculus to generalise from observational to interventional data".  One thing that we would add here to the camera-ready would be to also note that Aglietti et al. considers unobserved confounding (which is related to why they need this causal prior that they use). 

**"cumulative regret vs simple regret"**
A sublinear cumulative regret implies a converging regret at any particular round. We believe that this is well understood in the BO literature. 

**Do you have to select which variables to intervene as well or it is given ?**
Yes this challenge can also be seen as a component of our setting. One can just consider one of the possible actions on each variable as the "not intervening on this variable" action. 

**"no causal priors here like in CBO"**
As mentioned above, we will add something to this sentence we quoted above to make it clear that Agliettti et al. are looking at a setting where unobserved confounding is present, which we hope would address your concern on this particular matter. 

We think that this minor change in the related work and our clarification here should be sufficient to address the raised concern. Please let us know if you agree and if it affects your score at all.

Reviewer comment (Evaluation):
Strengthening my belief with respect to the "distance" of this work from CBO, the authors here evaluate performance with the **cumulative** regret. This does not seem to be used at all in CBO, where instead a simple regret seems to be used (Aglietti et al 2020).

Author response:
Thank you for your response and constructive feedback.

**"it is confusing to say you are generalizing CBO throughout."**

We only explicitly use this wording in the abstract and it is true in the context of those sentences, since we do consider a more general version of "an agent intervenes on a structural causal model with known graph but unknown mechanisms to maximize a downstream reward variable". In the main body of the paper we believe that we give the more nuanced view that you are after:
"Aglietti et al.(2020) propose the first CBO setting with hard interventions and an algorithm that uses the do-calculus to generalise from observational to interventional data".  One thing that we would add here to the camera-ready would be to also note that Aglietti et al. considers unobserved confounding (which is related to why they need this causal prior that they use). 

**"cumulative regret vs simple regret"**
A sublinear cumulative regret implies a converging regret at any particular round. We believe that this is well understood in the BO literature. 

**Do you have to select which variables to intervene as well or it is given ?**
Yes this challenge can also be seen as a component of our setting. One can just consider one of the possible actions on each variable as the "not intervening on this variable" action. 

**"no causal priors here like in CBO"**
As mentioned above, we will add something to this sentence we quoted above to make it clear that Agliettti et al. are looking at a setting where unobserved confounding is present, which we hope would address your concern on this particular matter. 

We think that this minor change in the related work and our clarification here should be sufficient to address the raised concern. Please let us know if you agree and if it affects your score at all.