Paper: Learning Good Interventions in Causal Contextual Bandits with Adaptive Context (ICLR-2024)

Reviewer comment (Theory):
If I understood this paper correctly, all k contexts are not correlated. The contexts represent different users, and each user can have totally different causal graph and reward function. The proposed algorithm is learning the probabilities and reward functions independently among all contexts. In practice, the number of context could be very large, the proposed algorithm is not very practical as it requires a lot of explorations.

Author response:
1. The k contexts are not correlated. The contexts represent different users, and each user can have a totally different causal graph and reward function.

A1. In our manuscript, the term 'context' is used in a specific manner, which might not have been evidently clear. It does not refer to a single user, but rather to their demographic. Each demographic may have many users, and as such, the number of contexts is driven by the amount of information an advertiser receives about a user from the platform (which is limited). In practice, each context -- of which there are a finite number -- may contain a large number of users on the platform. Further we note that having correlated contexts may require additional assumptions as to how information is shared between contexts. Once such assumptions are incorporated, our framework is general enough to accomodate such a problem setup. Having said that, correlated contexts is not the problem we are addressing. We are happy to provide additional information or elaboration on this if it would be helpful.

Reviewer comment (Novelty):
The causal graph setting in this paper is not new and is very rudimental, i.e., the assumption of the graph for each context, which is similar to the original Lattimore 2016 paper and ignores many recent developments, e.g., Lu 2020, 2021, Adaptively Exploiting d-Separators with Causal Bandits 2022.

Author response:
2. The causal graph setting in this paper is not new.

A2. While newer causal graph settings have been considered in Bilodeau et al. (2022) and Lu et al. (2021), our contributions are altogether novel with regard to the decision problem considered. Having said that, we thank the reviewer for pointing us to these references. We will add, “There have been new lines of work extending the causal graph considered in this setting in Lu et al. (2021), alongside works that consider a d-separation between the intervention and the outcome (Bilodeau et al., 2022)."

Reviewer comment (Theory):
In terms of the analysis, how different is it compared to Subramanian 2022 work, e.g., can you explain why the stochastic transition to the contexts will make the analysis in the paper much more challenging?

Author response:
3. Why are stochastic transitions to the contexts difficult?

A3. We address this question in three parts: the difficulty in practice, the difficulty in the setting, and the difficulty in analysis.

First, we address the practical difficulty. We note that in practice, it is not possible to request a platform for exactly one type of user that we want. For example, if we request a demographic of new-parents, we may be referred to users who are either high-income or low-income (who both share the attribute of being a new parent). There are various other attributes of users that may or may not be requested and may be assigned to us. Therefore, assuming a deterministic transition to a context is practically infeasible.

Second, we address the difficulty in the setting. To compare the two settings, we carried out various experiments. We share the results below:
   1. Plot for regret with lambda in deterministic and nondeterministic settings:
      [Link to Plot](https://drive.google.com/file/d/1l3_jhuOS3xz_8LcQAaZhTbDfvNJxktHL/view?usp=sharing)
   
   2. Plot for regret with exploration budget in deterministic and nondeterministic settings:
      [Link to Plot](https://drive.google.com/file/d/1qi0ns6ZCP4RZP3RCJ8MymDafvGZRFfG3/view?usp=sharing)

   3. Plot for regret with the number of intermediate contexts:
      [Link to Plot](https://drive.google.com/file/d/1pJ-Ya3Fhclb0fowOWRY0nS_nb3Gk4y8g/view?usp=sharing)

We note that in each of these cases, the stochastic cases are harder.

We provide plots for several other experiments based on reviewer suggestions below, which we will be adding to the manuscript:
[Additional Experiments](https://drive.google.com/drive/folders/1VMkeenDM797NtsR25_Fnsc3t3yZkuqy1?usp=sharing)

Our anonymous GitHub repo contains the code required to replicate these experiments:
[causal contextual bandit GitHub repo](https://github.com/adaptiveContextualCausalBandits/aCCB)

Third, we address the difficulty in the analysis of the stochastic problem. Consider only two interventions at the start state, separated by an $\epsilon$ probability for the transition to a preferred context that has an optimal reward. Separating these two interventions at the start state becomes extremely difficult. The deterministic transitions entirely sidestep these concerns and can directly reach the preferred context. Therefore, the analysis in such a stochastic setting is hard.

Reviewer comment (Experiments):
Authors should compare the convex exploration with other exploration strategies, such as UCB-based or TS-based, instead of only uniform exploration.

Author response:
We thank the reviewer for these thoughtful comments. To compare our algorithm with these baselines, we carried out several new experiments, the results of which we share below.

# Variation of Expected Regret with Exploration Budget

| exploration_budget | UE				 	 | roundrobin_ucb | roundrobin_ts | ucb_over_intervention_pairs | ts_over_intervention_pairs | convex_explorer |
|--------------------|-----------------------|----------------|---------------|-----------------------------|----------------------------|-----------------|
| 100                | 1.0                   | 1.0            | 1.0           | 1.0                         | 1.0                        | 0.87            |
| 250                | 0.99                  | 0.95           | 0.96          | 1.0                         | 1.0                        | 0.44            |
| 500                | 0.71                  | 0.54           | 0.97          | 0.99                        | 1.0                        | 0.16            |
| 1,000              | 0.4                   | 0.32           | 0.96          | 0.79                        | 1.0                        | 0.02            |
| 2,500              | 0.02                  | 0.1            | 0.96          | 0.2                         | 1.0                        | 0.0             |
| 5,000              | 0.0                   | 0.04           | 0.94          | 0.13                        | 1.0                        | 0.0             |
| 7,500              | 0.0                   | 0.01           | 0.96          | 0.1                         | 1.0                        | 0.0             |
| 10,000             | 0.0                   | 0.0            | 0.93          | 0.08                        | 1.0                        | 0.0             |
| 12,500             | 0.0                   | 0.0            | 0.91          | 0.06                        | 1.0                        | 0.0             |
| 15,000             | 0.0                   | 0.0            | 0.93          | 0.06                        | 1.0                        | 0.0             |
| 20,000             | 0.0                   | 0.0            | 0.94          | 0.0                         | 1.0                        | 0.0             |
| 25,000             | 0.0                   | 0.0            | 0.96          | 0.0                         | 1.0                        | 0.0             |
| 100,000            | 0.0                   | 0.0            | 0.97          | 0.0                         | 1.0                        | 0.0             |


Firstly, we note that the uniform exploration algorithm (round robin) performs the best amongst the non-causal algorithms. UCB for choice of intervention at each context, with round-robin over the choice of initial intervention also works quite well, but takes a longer time to converge to the right solution. Thompson sampling based approaches do not work well. The works performing algorithms were those which treated the intervention pairs as a bandit, and tried to optimize over this larger bandit problem (ignoring the information available in the structure of the problem). We share the plot related to this table below:
https://drive.google.com/file/d/1qWSt7Kv-sEi85dD4sjflLnQqRC7V_TCN/view?usp=sharing (Note the log scaling over the x-axis).

We provide plots for several other experiments based on reviewer suggestions below, which we will be adding to the manuscipt:
https://drive.google.com/drive/folders/1VMkeenDM797NtsR25_Fnsc3t3yZkuqy1?usp=sharing

Our anonymous github repo contains the code required to replicate these experiments:
https://github.com/adaptiveContextualCausalBandits/aCCB

Reviewer comment (Theory):
What is the upper bound or lower bound of $\lambda$?

Author response:
The lower bound on $\lambda$ is $2k$ and upperbound is $nk$. While we have noted the upperbound in page 7 ("The upperbound on $\lambda$ is $nk$") and other places, we will clarify on the lower bound as well.

Reviewer comment (Experiments):
Moreover, how should λ be specifically adjusted, especially when dealing with entirely new contexts in a new scenario? Additionally, when λ is less than nk (e.g., λ=390), CE's performance is inferior to that of UE. How can this be explained?

Author response:
The $\lambda$ value is based on the instance given to us, and is not an adjustable hyperparameter. It is something intrinsic to the hardness of the problem given to us. In most instances, $\lambda$ is likely to be low given causal variables do not have $q$ values which are close to 0. 

A2b. In the theoretical analysis, we were not utilising the data from the first two parts of the algorithm (where the probability of transition, and causal parameter were being estimated). This is because these parts of the algorithm were only influencing the constants in the regret term. But, in a practical implementation, the data from these parts of the algorithm can also be used to estimate rewards. We have done so now, and the following are the updated results (with more baselines as well): 
https://drive.google.com/file/d/1YydC2e_NkCBWEp-IAPB0EXNPPzvwt3mU/view?usp=sharing

Further, we provide the data related to the figure here:

# Variation of Expected Regret with Lambda $\lambda$

| lambda    | UE			 		| roundrobin_ucb | roundrobin_ts | ucb_over_intervention_pairs | ts_over_intervention_pairs | convex_explorer |
|-----------|-----------------------|----------------|---------------|-----------------------------|----------------------------|-----------------|
| 20.000000 | 0.152000              | 0.327999       | 1.000005      | 0.395999                    | 1.000005                   | 0.000000        |
| 30.000000 | 0.196000              | 0.321999       | 1.000005      | 0.337999                    | 1.000005                   | 0.000000        |
| 40.000000 | 0.164000              | 0.333999       | 1.000005      | 0.391999                    | 1.000005                   | 0.002000        |
| 49.999996 | 0.140000              | 0.312000       | 1.000005      | 0.391999                    | 1.000005                   | 0.000000        |
| 60.000000 | 0.182000              | 0.339999       | 1.000005      | 0.361999                    | 1.000005                   | 0.002000        |
| 70.000000 | 0.180000              | 0.312000       | 1.000005      | 0.365999                    | 1.000005                   | 0.018000        |
| 79.999990 | 0.176000              | 0.310000       | 1.000005      | 0.349999                    | 1.000005                   | 0.038000        |
| 90.000000 | 0.192000              | 0.290000       | 1.000005      | 0.381999                    | 1.000005                   | 0.060000        |
| 100.000000| 0.174000              | 0.353999       | 1.000005      | 0.385999                    | 1.000005                   | 0.082000        |

We hope this clarifies that CE is not worse the UE (round robin) across domains.

Reviewer comment (Experiments):
There are too few experimental results, and is it possible to provide the source or generation rules of the experimental data? Although the theoretical aspects are solid, the experimental section needs further improvement, especially in terms of interpretability. The motivation given earlier pertains to a cold start scenario. Could authors provide further explanations for the experiments?

Author response:
We thank the reviewer for their well thought our suggestion in this point. We have carried out eight further experiments to this work currently, including adding more baselines and plotting the regret across various instances. 

We share the plots related to our new experiments here:
https://drive.google.com/drive/folders/1VMkeenDM797NtsR25_Fnsc3t3yZkuqy1

Our anonymous github repo contains the code required to replicate these experiments:
https://github.com/adaptiveContextualCausalBandits/aCCB

A3 To share our generation rules and provide further code-replicability, we share our entire code on our [causal contextual bandit GitHub repo](https://github.com/adaptiveContextualCausalBandits/aCCB). We will further add a full section expanding on our experimental setup in the supplementary material.

Reviewer comment (Theory):
Restricting to binary interventions and rewards (i.e., either 0 or 1) makes the problem easier to solve but limits the practical applications to problems with only binary interventions and rewards.

Author response:
A1. The binary intervention setting is not really restrictive  as the setting is general enough to capture all finite alphabets (over the variables). We will clarify this point further in the manuscript.

Reviewer comment (Theory):
The possible number of contexts can be very large (or even infinite), e.g., the number of users on the platform. Therefore, working with matrix P (where the number of columns is the same as the number of contexts) may be computationally challenging.

Author response:
A2. A contextual bandit setting need not hold one context per unit or per user in the system. Often, one may find many users pertain to the same context. For example, if one assume a combination of user-device and user-demographic to be a context, then not only is the number of contexts small, but further, the number of users per context may also be large. Crucially, irrespective of the number of contexts, we perform better than alternatives algorithms for this setting where regret is upper bounded by $\sqrt{nk/T}$ for n contexts and k interventions per context. Our regret is upper bounded by $\sqrt{\lambda/T}$ where $\lambda \ll nk$.

Reviewer comment (Theory):
The proposed algorithm is horizon-dependent as it needs to know the total number of rounds, T, upfront. It is unclear if the proposed algorithm will work for problems where T is very small.

Author response:
A3. Yes indeed. This is a standard assumption in the simple regret setting. These settings are applicable in domains where there is some budget given for exploring, in which time the near-optimal solution has to be found. For example, in medical domains, a certain number of exploratory tests are allowed after which the best variant of a drug has to be found. In the advertising domain, marketers often look at A/B tests where there is a certain budget for exploration after the marketer makes their decision on the best arm/intervention.

Reviewer comment (Writing):
Page 1, last paragraph: what are the other variables in this statement, "they get to observe the values of multiple other variables in the causal graph."?

Author response:
A4. Since variables in a causal graph are causally linked as in a causal bayesian network (CBN), by intervening on one variable in the graph, the values of other variables may be inferred. Specifically, by doing the do-nothing intervention, one may observe the values of multiple variables in the causal graph as well as the reward variable. Thereby a causal intervention is more potent than an arm-pull in a multi-armed bandit setting.

Reviewer comment (Theory):
Is there any relationship between $\lambda$ and effective dimension if the problem is modeled as a sparse bandit problem?

Author response:
A5. Indeed there is! As the reviewer has observed the causal parameter $\lambda$ is dependent on various features of the problem, including the number of contexts, and the causal threshold parameter $m$ at each context. Having said that, in our understanding, the modelling of the problem as a sparse bandit may not help. 

To experimentally verify this, we carried out an experiment where we compare our model (convex_explorer) with two baseline models (ucb_over_intervention_pairs,ts_over_intervention_pairs) which treat the problem as a large bandit problem. These models treat the pair of interventions at the start state, and at the intermediate context -- which may be thought of as the (user_demographic, advertisement_choice) pair -- as a large bandit problem. We note that such a formulation leads to sub-optimal regret as seen in the figure here: https://drive.google.com/file/d/1YydC2e_NkCBWEp-IAPB0EXNPPzvwt3mU/view?usp=sharing

We propose that such sub-optimal regret is because the information regarding the context is not shared, and hence the regret is high.