Original: C AUSAL C ONTEXTUAL B ANDITS WITH T ARGETED I NTERVENTIONS
Revised: C AUSAL C ONTEXTUAL B ANDITS WITH T ARGETED I NTERVENTIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study a contextual bandit setting where the learning agent has the ability to perform interventions on targeted subsets of the population, apart from possessing qualitative causal side-information.
Revised: We study a contextual bandit setting where the learning agent has the ability to perform interventions on targeted subsets of the population, apart from possessing qualitative causal side-information.

Original: This novel formalism captures intricacies in real-world scenarios such as software product experimentation where targeted experiments can be conducted.
Revised: This novel formalism captures intricacies in real-world scenarios such as software product experimentation where targeted experiments can be conducted.

Original: However, this fundamentally changes the set of options that the agent has, compared to standard contextual bandit settings, necessitating new techniques.
Revised: However, this fundamentally changes the set of options that the agent has, compared to standard contextual bandit settings, necessitating new techniques.

Original: This is also the first work that integrates causal sideinformation in a contextual bandit setting, where the agent aims to learn a policy that maps contexts to arms (as opposed to just identifying one best arm).
Revised: This is also the first work that integrates causal sideinformation in a contextual bandit setting, where the agent aims to learn a policy that maps contexts to arms (as opposed to just identifying one best arm).

Original: We propose a new algorithm, which we show empirically performs better than baselines on experiments that use purely synthetic data and on real world-inspired experiments.
Revised: We propose a new algorithm, which we show empirically performs better than baselines on experiments that use purely synthetic data and on real world-inspired experiments.

Original: We also prove a bound on regret that theoretically guards performance.
Revised: We also prove a bound on regret that theoretically guards performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Contextual bandits have been used as natural frameworks to model interactive decision making scenarios such as recommendation systems (Liu et al., 2018),
Revised: Contextual bandits have been used as natural frameworks to model interactive decision making scenarios such as recommendation systems (Liu et al., 2018),

Original: marketing campaign allocation (Sawant et al., 2018) and more (Bouneffouf & Rish, 2019).
Revised: marketing campaign allocation (Sawant et al., 2018) and more (Bouneffouf & Rish, 2019).

Original: In this framework, the learning agent repeatedly interacts with an environment with the aim of learning a near-optimal decision policy that maps a context space to a set of actions (also referred to as arms or interventions 1 ).
Revised: In this framework, the learning agent repeatedly interacts with an environment with the aim of learning a near-optimal decision policy that maps a context space to a set of actions (also referred to as arms or interventions 1 ).

Original: In the standard stochastic variant, in each round of interaction, the agent observes a context from the environment and, once the agent chooses an action, the environment returns a sampled reward that is a function of the current context and chosen action (Lattimore & Szepesv ´ ari, 2020).
Revised: In the standard stochastic variant, in each round of interaction, the agent observes a context from the environment and, once the agent chooses an action, the environment returns a sampled reward that is a function of the current context and chosen action (Lattimore & Szepesv ´ ari, 2020).

Original: The agent’s objective is to minimize some meaningfulmeasure of closeness of the policy to optimality (see Lattimore & Szepesv´ari (2020) for some standard definitions).
Revised: The agent’s objective is to minimize some meaningful

Original: One of the key issues in the wide application of contextual bandits (and reinforcement learning, in general (Dulac-Arnold et al., 2021)) is their need for a large number of samples that are costly to actualize in practice.
Revised: One of the key issues in the wide application of contextual bandits (and reinforcement learning, in general (Dulac-Arnold et al., 2021)) is their need for a large number of samples that are costly to actualize in practice.

Original: For example, each arm might correspond to performing a product experiment on users or to conducting a specific medical intervention.
Revised: For example, each arm might correspond to performing a product experiment on users or to conducting a specific medical intervention.

Original: However, we show that there are nuances in real-world situations which – though not fully captured by the standard formulations of contextual bandits – if modeled and leveraged, allow us to build methods that can improve the rate at which good policies can be identified.
Revised: However, we show that there are nuances in real-world situations which – though not fully captured by the standard formulations of contextual bandits – if modeled and leveraged, allow us to build methods that can improve the rate at which good policies can be identified.

Original: A motivating example Considerthe example of a sales-assistance software agent that is learning to suggest a campaign that is optimal for a given sales lead (defined by a set of features).
Revised: A motivating example Consider a sales-assistance software agent that is learning to suggest a campaign that is optimal for a given sales lead (defined by a set of features).

Original: With software products, there is often an opportunity to conduct experiments with variants of the software on defined subsets of the population of users (each specified by a set of characteristics, or context variables) to learn about resulting metrics; see Google (2021) for an example.
Revised: With software products, there is often an opportunity to conduct experiments with variants of the software on defined subsets of the population of users (each specified by a set of characteristics, or context variables) to learn about resulting metrics; see Google (2021) for an example.

Original: We can think of these experiments as constituting a training phase .
Revised: We can think of these experiments as constituting a training phase .

Original: More specifically, the agent, for example, can conduct a campaign (which is the intervention) on a specific subset of users (for example, the context value os=iOS could define a subset) and observe the outcome.
Revised: More specifically, the agent, for example, can conduct a campaign (i.e., the intervention) on a specific subset of users (for example, the context value os=iOS could define a subset) and observe the outcome.

Original: Thus, instead of necessarily performing an intervention on a randomly sampled user coming from the natural population distribution, the agent can instead choose to target the intervention on a randomly selected user with specific characteristics (as defined by an assignment of context variable values).
Revised: Thus, instead of necessarily performing an intervention on a randomly sampled user coming from the natural population distribution, the agent can instead choose to target the intervention on a randomly selected user with specific characteristics (as defined by an assignment of context variable values).

Original: We call this special type of interventions targeted interventions ,and they fundamentally change the set of options that the agent has in every training round.
Revised: This type of interventions, which we call targeted interventions , fundamentally changes the set of options that the agent has in every training round.

Original: In addition, the training phase often happens in a more lenient environment where the agent might have access to auxiliary context variables (such as IT-spend ) that are unavailable in the evaluation phase.
Revised: In addition, the training phase often happens in a more lenient environment where the agent might have access to auxiliary context variables (such as IT-spend ) that are unavailable in the evaluation phase.

Original: Further, there is also causal side-information sometimes available.
Revised: Further, there is also causal side-information sometimes available.

Original: For instance, we might know that emailsubject causes openemail , and not the other way around.
Revised: For instance, we might know that emailsubject causes openemail , and not the other way around.

Original: Encoding this qualitative side-information as a causal graph can help the agent make use of this structure.
Revised: Encoding this qualitative side-information as a causal graph can help the agent make use of this structure.

Original: Lattimore et al.
Revised: Lattimore et al.

Original: (2016) and Yabe et al.
Revised: (2016) and Yabe et al.

Original: (2018) demonstrate this in the best-arm identification case.
Revised: (2018) demonstrate this in the best-arm identification case.

Original: After training, there is an evaluation phase (e.g., when the agent is deployed), where the agent observes the main context variables and decides an action; its regret performance is measured at this point.
Revised: After training, there is an evaluation phase (e.g., when the agent is deployed), where the agent observes the main context variables and decides an action; its regret performance is measured at this point.

Original: Our framework Our framework captures the above intricacies (which are not restricted to software product experimentation).
Revised: Our framework Our framework captures the above intricacies (which are not restricted to software product experimentation).

Original: The formal treatment is in Section 2; we present an overview here.
Revised: The formal treatment is in Section 2; we present an overview here.

Original: The agent has T rounds of interaction that act as the training phase, followed by a ( T + 1) ’
Revised: The agent has T rounds of interaction that act as the training phase, followed by a ( T + 1) ’

Original: th round on which it is evaluated.
Revised: th round on which it is evaluated.

Original: In each of the T training rounds, it has the choice to perform either a targeted intervention or a standard interaction.
Revised: In each of the T training rounds, it has the choice to perform either a targeted intervention or a standard interaction.

Original: Further, in addition to the main context variables, the agent can observe a (possibly empty) set of additional auxiliary context variables during training; these auxiliary context variables are not observable during evaluation.
Revised: Further, in addition to the main context variables, the agent can observe a (possibly empty) set of additional auxiliary context variables during training; these auxiliary context variables are not observable during evaluation.

Original: All variables are made available to the agent at the end of every training round (this is similar to Lattimore et al.
Revised: All variables are made available to the agent at the end of every training round (this is similar to Lattimore et al.

Original: The agent also has access to a causal graph that models the qualitative causal relationships between the variables; there are very few assumptions made on this graph (discussed in Section 2.1).
Revised: The agent also has access to a causal graph that models the qualitative causal relationships between the variables; there are very few assumptions made on this graph (discussed in Section 2.1).

Original: The causal graph allows a factorized representation of the joint distribution of the variables, and as a result, enables information leakage – i.e., updating beliefs about several interventions after every single intervention.
Revised: The causal graph allows a factorized representation of the joint distribution of the variables, and as a result, enables information leakage – i.e., updating beliefs about several interventions after every single intervention.

Original: Also, importantly, we allow context variables to be categorical, and therefore the usual assumptions that enable generalization of the learned policy across contexts, such as linearity (e.g., in Dimakopoulou et al.
Revised: Also, importantly, we allow context variables to be categorical, and therefore the usual assumptions that enable generalization of the learned policy across contexts, such as linearity (e.g., in Dimakopoulou et al.

Original: (2019)), become invalid.
Revised: (2019)), become invalid.

Original: The agent’s problem is one of sample allocation – how to allocate samples across the T training rounds so as to learn a policy that minimizes regret in the ( T + 1) ’
Revised: The agent’s problem is one of sample allocation – how to allocate samples across the T training rounds so as to learn a policy that minimizes regret in the ( T + 1) ’

Original: th round that represents the evaluation phase.
Revised: th round that represents the evaluation phase.

Original: In the evaluation phase, the agent observes the main context variables from the environment, against which it chooses an action and receives a reward – much like in a standard contextual bandit setting.
Revised: In the evaluation phase, the agent observes the main context variables from the environment, against which it chooses an action and receives a reward – much like in a standard contextual bandit setting.

Original: Since targeted interventions are restricted to the training phase, it introduces a difference between the training and evaluation phases.
Revised: Since targeted interventions are restricted to the training phase, it introduces a difference between the training and evaluation phases.

Original: The agent’s challenge, therefore, is to learn policies that minimize regret in the evaluation phase using samples it collects in the training phase (from a combination of standard interactions and targeted interventions).
Revised: The agent’s challenge, therefore, is to learn policies that minimize regret in the evaluation phase using samples it collects in the training phase (from a combination of standard interactions and targeted interventions).

Original: The algorithm we propose utilizes a novel entropy- like measure called Unc that guides this sample allocation in a way that also exploits the information leakage.
Revised: The algorithm we propose utilizes a novel entropy- like measure called Unc that guides this sample allocation in a way that also exploits the information leakage.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: In this paper, we formalize this modified setting, which we call “causal contextual bandits with targeted interventions”, provide a novel algorithm and show both theoretical and experimental results.
Revised: In this paper, we formalize this modified setting, which we call “causal contextual bandits with targeted interventions”, provide a novel algorithm and show both theoretical and experimental results.

Original: Specifically, our contributions are:
Revised: Specifically, our contributions are:

Original: • We formalize the more nuanced contextual bandit setting described above (Sec. 2).
Revised: • We formalize the more nuanced contextual bandit setting described above (Sec. 2).

Original: This is the first work that we know of that formulates a contextual bandit setting with causal side-information.
Revised: This is the first work that we know of that formulates a contextual bandit setting with causal side-information.

Original: This is also the first paper we are aware of that introduces targeted interventions in a contextual bandit setting.
Revised: This is also the first paper we are aware of that introduces targeted interventions in a contextual bandit setting.

Original: • We propose a new algorithm based on minimizing a novel entropy- like measure (Sec. 3.1) •
Revised: • We propose a new algorithm based on minimizing a novel entropy- like measure (Sec. 3.1) •

Original: We prove a bound on its regret, providing a theoretical guard on its performance (Sec. 3.2) • We show results of experiments that use purely synthetic data (Sec. 4.1).
Revised: We prove a bound on its regret, providing a theoretical guard on its performance (Sec. 3.2) • We show results of experiments that use purely synthetic data (Sec. 4.1).

Original: The results demonstrate that our algorithm performs better than baselines.
Revised: The results demonstrate that our algorithm performs better than baselines.

Original: • We also run experiments that are inspired by proprietary data from a CRM software company (Sec.4.2).
Revised: • We also run experiments that are inspired by proprietary data from Freshworks Inc. (Sec.

Original: This provides evidence of our algorithm’s performance in more realistic scenarios as well.
Revised: 4.2), providing evidence of our algorithm’s performance in more realistic scenarios as well.

Original: Our motivation comes from real world settings where experiments are costly; therefore, we are interested in empirical behavior when the training budget T is relatively small.
Revised: Our motivation comes from real world settings where experiments are costly; therefore, we are interested in empirical behavior when the training budget T is relatively small.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Causal bandits have been studied in literature recently (Lattimore et al., 2016; Sen et al., 2017;
Revised: Causal bandits have been studied in literature recently (Lattimore et al., 2016; Sen et al., 2017;

Original: Yabe et al., 2018; Lu et al., 2020) and they leverage causal side-information to transfer knowledge across interventions.
Revised: Yabe et al., 2018; Lu et al., 2020) and they leverage causal side-information to transfer knowledge across interventions.

Original: They, however, have been studied only in a best arm identification setting, with each arm modeled as an intervention on a set of variables; the objective is to learn a single best arm.
Revised: They, however, have been studied only in a best arm identification setting, with each arm modeled as an intervention on a set of variables; the objective is to learn a single best arm.

Original: Our setting differs significantly since the agent attempts to learn a policy (by which we mean a mapping from contexts to arms) instead of a single best arm.
Revised: Our setting differs significantly since the agent attempts to learn a policy (by which we mean a mapping from contexts to arms) instead of a single best arm.

Original: Therefore, while the agent can perform targeted interventions that specify both the context and action, it is still attempting to learn an optimal context-action mapping.
Revised: Therefore, while the agent can perform targeted interventions that specify both the context and action, it is still attempting to learn an optimal context-action mapping.

Original: The contextual bandit literature is well-studied (see Zhou (2016) and Lattimore & Szepesv´ari (2020) for surveys), and has taken various approaches to enable knowledge transfer across contexts.
Revised: The contextual bandit literature is well-studied (see Zhou (2016) and Lattimore & Szepesv´ari (2020) for surveys), and has taken various approaches to enable knowledge transfer across contexts.

Original: For example, Dimakopoulou et al.
Revised: For example, Dimakopoulou et al.

Original: (2019) assume expected reward is a linear function of context, while
Revised: (2019) assume expected reward is a linear function of context, while

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2014) make assumptions about the existence of a certain oracle.
Revised: (2014) make assumptions about the existence of a certain oracle.

Original: However, there has not been work that has looked at contextual bandits which utilize causal side-information as a way to transfer knowledge across contexts, or considered targeted interventions in the space of options for the agent.
Revised: However, there has not been work that has looked at contextual bandits which utilize causal side-information as a way to transfer knowledge across contexts, or considered targeted interventions in the space of options for the agent.

Original: Our method utilizes the information leakage that the causal graph provides to not just learn about other interventions after every intervention, but also to be smarter about the choice of targeted interventions to conduct.
Revised: Our method utilizes the information leakage that the causal graph provides to not just learn about other interventions after every intervention, but also to be smarter about the choice of targeted interventions to conduct.

Original: Another related area is active learning (Settles, 2009).
Revised: Another related area is active learning (Settles, 2009).

Original: In active learning, a supervised learning agent gets the option to query an oracle actively to obtain the true labels for certain data points.
Revised: In active learning, a supervised learning agent gets the option to query an oracle actively to obtain the true labels for certain data points.

Original: However, it is in the supervised learning setting where the agent receives true label feedback, whereas in our setting the agent only receives bandit feedback (that is, only for the action that was taken).
Revised: However, it is in the supervised learning setting where the agent receives true label feedback, whereas in our setting the agent only receives bandit feedback (that is, only for the action that was taken).

Original: Nevertheless, our work can be thought of as infusing some elements of the active learning problem into the contextual bandits setting by providing the agent the ability to perform targeted interventions.
Revised: Nevertheless, our work can be thought of as infusing some elements of the active learning problem into the contextual bandits setting by providing the agent the ability to perform targeted interventions.

Original: There has been some work investigating the intersection of causality and bandits with the aim of transfer learning.
Revised: There has been some work investigating the intersection of causality and bandits with the aim of transfer learning.

Original: Zhang & Bareinboim (2017) study transfer of knowledge from offline data in the presence of unobserved confounders, in a non-contextual setting.
Revised: Zhang & Bareinboim (2017) study transfer of knowledge from offline data in the presence of unobserved confounders, in a non-contextual setting.

Original: As can be seen, this differs significantly from our setting.
Revised: As can be seen, this differs significantly from our setting.

Original:  F ORMALISM
Revised:  F ORMALISM

Original: We assume that the underlying environment is modeled as a causal model M , which is defined by a directed acyclic graph G over all variables – i.e., the variable to be intervened ( X ), the reward variable ( Y ), and the set of context variables ( C ) – and a joint probability distribution P that factorizes over G (Pearl, 2009; Koller & Friedman, 2009).
Revised: We assume that the underlying environment is modeled as a causal model M , which is defined by a directed acyclic graph G over all variables – i.e., the variable to be intervened ( X ), the reward variable ( Y ), and the set of context variables ( C ) – and a joint probability distribution P that factorizes over G (Pearl, 2009; Koller & Friedman, 2009).

Original: C is partitioned into the set of main context variables ( C tar ) and the set of auxiliary context variables ( C other ) .
Revised: C is partitioned into the set of main context variables ( C tar ) and the set of auxiliary context variables ( C other ) .

Original: G is sometimes called the causal graph or causal diagram of M .
Revised: G is sometimes called the causal graph or causal diagram of M .

Original: Each variable takes on a finite, known set of values; note that this is quite generic, and allows for categorical variables.
Revised: Each variable takes on a finite, known set of values; note that this is quite generic, and allows for categorical variables.

Original: An intervention do ( X = x ) on M involves removing all arrows from the parents of X into X , and setting X
Revised: An intervention do ( X = x ) on M involves removing all arrows from the parents of X into X , and setting X

Original: = x (Pearl, 2009).
Revised: = x (Pearl, 2009).

Original: The agent has access only to G and not to M ; therefore, the agent is not given any knowledge a priori about the underlying conditional probability distributions (CPDs) of the variables.
Revised: The agent has access only to G and not to M ; therefore, the agent is not given any knowledge a priori about the underlying conditional probability distributions (CPDs) of the variables.

Original: We specify a targeted intervention X = x conditioned on context C tar = c tar succinctly by ( x, c tar ) .
Revised: We specify a targeted intervention X = x conditioned on context C tar = c tar succinctly by ( x, c tar ) .

Original: While standard interventions (Pearl, 2009) result in a distribution of the form P ( .
Revised: While standard interventions (Pearl, 2009) result in a distribution of the form P ( .

Original: | do ( x )) , targeted interventions result in P ( .
Revised: | do ( x )) , targeted interventions result in P ( .

Original: | do ( x ) , C tar = c tar ) .
Revised: | do ( x ) , C tar = c tar ) .

Original: Table 1 summarizes the key notation used in this paper.
Revised: Table 1 summarizes the key notation used in this paper.

Original: The agent is allowed T training rounds of interaction with the environment, at the end of which it aims to have learned a policy ˆ ϕ : val ( C tar ) → val ( X ) .
Revised: The agent is allowed T training rounds of interaction with the environment, at the end of which it aims to have learned a policy ˆ ϕ : val ( C tar ) → val ( X ) .

Original: Specifically, in each of the T training rounds, the agent can choose to either
Revised: Specifically, in each of the T training rounds, the agent can choose to either

Original: • (Standard interaction) Observe context c tar ∼ P ( C tar ) , choose intervention x , and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar ) , (or
Revised: • (Standard interaction) Observe context c tar ∼ P ( C tar ) , choose intervention x , and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar ) , (or)

Original: [Table]
Revised: [Table]

Original: variablethat will be intervened
Revised: variable

Original: [Table]
Revised: [Table]

Original: tar ,
Revised: be

Original: [Table]
Revised: [Table]

Original: set of main context variables and set of auxiliary context variables, respectively; so the set of all context variables is C = C tar ∪ C other = { ..., C i , ... } .[Table] e.g., [Table]
Revised: set of main context variables and set of auxiliary context variables, respectively; so the set of all context variables is C = C tar ∪ C other = { ..., C i , ... } .

Original: variable’s value; e.g., c

Original: [Table]
Revised: [Table]

Original: a vector of random variable values; e.g., c tar denotes a specific choice of values taken by variables in C tar ( x
Revised: a vector of random variable values; e.g., c tar denotes a specific choice of values taken by variables in C tar ( x

Original: [Table]
Revised: [Table]

Original:  tar ) specifies targeted intervention X
Revised: c tar ) specifies targeted intervention X

Original: = x on subset defined by C tar = c tar ˆ
Revised: = x on subset defined by C tar = c tar ˆ

Original: [Table]
Revised: [Table]

Original: estimate of distribution P and expectation E based on current beliefs

Original: [Table]

Original: beliefs (vector of Dirichlet parameters) about parameters of P ( V | pa V ) ;θ ( v ) V | pa V is the entry corresponding to V = v .
Revised: beliefs (vector of Dirichlet parameters) about parameters of P ( V | pa V ) ;

Original: [Table]
Revised: [Table]

Original: set of values taken by the variable V , and set of variables V , respectively.
Revised: is the entry corresponding to

Original: [Table]
Revised: [Table]

Original: val
Revised: v .

Original: [Table]
Revised: [Table]

Original: |

Original: [Table] val [Table] |
Revised: set of values taken by the variable V , and set of variables V , respectively.

Original: [Table]
Revised: [Table]

Original: value of variables in PA V , the parents of V .
Revised: value of variables in PA V , the parents of V .

Original: For Y , we let PA Y denote its parents excluding X to keep proof easier to read.
Revised: For Y , we let PA Y denote its parents excluding X to keep proof easier to read.

Original: [Table]
Revised: [Table]

Original: if a is an assignment of values to A , then a ⟨B⟩ is assignment of those values to respective variables B ⊆ A .
Revised: if a is an assignment of values to A , then a ⟨B⟩ is assignment of those values to respective variables B ⊆ A .

Original: Set operations for readability, we use them on vectors as well, where there is no ambiguity.
Revised: Set operations for readability, we use them on vectors as well, where there is no ambiguity.

Original: • (Targeted intervention) Choose intervention x and a target subset given by context values C tar = c tar , and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar 
Revised: • (Targeted intervention) Choose intervention x and a target subset given by context values C tar = c tar , and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar )

Original: ) where, when there is no ambiguity, we use P interchangeably to mean either the joint distribution or a marginal distribution.
Revised: where, when there is no ambiguity, we use P interchangeably to mean either the joint distribution or a marginal distribution.

Original: Note that in both modes of interaction above, the intervention is only on
Revised: Note that in both modes of interaction above, the intervention is only on

Original: X ; they differ, however, in whether the agent observes the context values from the environment or whether it chooses the context values on which to condition the intervention.
Revised: X ; they differ, however, in whether the agent observes the context values from the environment or whether it chooses the context values on which to condition the intervention.

Original: After training, the agent is evaluated in the ( T + 1) ’
Revised: After training, the agent is evaluated in the ( T + 1) ’

Original: th round.
Revised: th round.

Original: Here, the agent is presented a query context c tar ∼ P ( C tar ) to which it responds with an action x = ˆ ϕ ( c tar ) using the learned policy, and receives a reward y ∼ P ( Y | do ( x ) , c tar ) from the environment.
Revised: Here, the agent is presented a query context c tar ∼ P ( C tar ) to which it responds with an action x = ˆ ϕ ( c tar ) using the learned policy, and receives a reward y ∼ P ( Y | do ( x ) , c tar ) from the environment.

Original: The objective of the agent is to learn a policy ˆ ϕ that minimizes simple-regret , which is defined as:
Revised: The objective of the agent is to learn a policy ˆ ϕ that minimizes simple-regret , which is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ϕ ∗ is an optimal policy, µ ∗ c tar ≜ E [ Y | do ( ϕ ∗ ( c tar )) , c tar ] and ˆ µ
Revised: where ϕ ∗ is an optimal policy, µ ∗ c tar ≜ E [ Y | do ( ϕ ∗ ( c tar )) , c tar ] and ˆ µ

Original: c tar ≜ E [ Y | do (ˆ ϕ ( c tar )) , c tar ] .
Revised: c tar ≜ E [ Y | do (ˆ ϕ ( c tar )) , c tar ] .

Original: 2. A SSUMPTIONS
Revised: 2. A SSUMPTIONS

Original: We are interested in DAGs G where there is an edge X
Revised: We are interested in DAGs G where there is an edge X

Original: → Y , and no other directed path between
Revised: → Y , and no other directed path between

Original: X and Y ; that is, X directly affects Y ; there is no context variable caused by X .
Revised: X and Y ; that is, X directly affects Y ; there is no context variable caused by X .

Original: This models the natural situation we encounter in settings such as a recommendation system where click-through rate ( Y ) of a user is directly affected by the user’s features (the context) and the recommendation ( X ); the recommendation does not cause any contexts, so doesn’t have a path to Y through them.
Revised: This models the natural situation we encounter in settings such as a recommendation system where click-through rate ( Y ) of a user is directly affected by the user’s features (the context) and the recommendation ( X ); the recommendation does not cause any contexts, so doesn’t have a path to Y through them.

Original: Purely interactive bandit settings, such as ours, are robust to unobserved confounders (UCs) between X and Y sincethe actions can be interpreted asdirectly sampling from the P ( .
Revised: We assume that there are no unobserved confounders (UCs); but purely interactive bandit settings, such as ours, are robust to direct UCs between X and Y since actions can be interpreted as sampling from the P ( .

Original: | do ( x )) distribution (Bareinboim et al., 2015; Guo et al., 2020).
Revised: | do ( x )) distribution (Bareinboim et al., 2015; Guo et al., 2020).

Original: Nevertheless, we do make an additional assumption to simplify the factorization in Section 3.2: ( A1 ) { C confounds C ′ ∈ C tar and Y } = ⇒ C ∈ C tar .
Revised: We make an additional assumption to simplify the factorization in Section 3.2: ( A1 ) { C confounds C ′ ∈ C tar and Y } = ⇒ C ∈ C tar .

Original: A sufficient condition for (A1) to be true is if C tar is ancestral (i.e., C tar contains all its ancestors).
Revised: A sufficient condition for (A1) to be true is if C tar is ancestral (i.e., C tar contains all its ancestors).

Original: This assumption does not preclude UCs (for example, the graph used in the synthetic experiments in Section 4.1 could have a UC between X and Y ), and can be relaxed in future extensions.
Revised: This assumption does not preclude UCs (for example, the graph used in the synthetic experiments in Section 4.1 could have a UC between X and Y ), and can be relaxed in the future.

Original:  S OLUTION APPROACH 3. A LGORITHM ( U NC CCB )
Revised:  S OLUTION APPROACH 3. A LGORITHM ( U NC CCB )

Original: Data: Graph G ; fraction α of Phase 1 rounds.
Revised: Data: Graph G ; fraction α of Phase 1 rounds.

Original: Initialization:
Revised: Initialization:

Original: For all V ∈ C ∪ { Y } , set θ V | pa V =
Revised: For all V ∈ C ∪ { Y } , set θ V | pa V =

Original: (1 , ..., 1) , for all values of pa V .
Revised: (1 , ..., 1) , for all values of pa V .

Original: 1 (Phase 1) for t = 1 . . .
Revised: 1 (Phase 1) for t = 1 . . .

Original: ⌈ αT ⌉ do
Revised: ⌈ αT ⌉ do

Original: Observe context c tar ∼ P ( C tar )
Revised: Observe context c tar ∼ P ( C tar )

Original: Choose x ∈ val ( X ) uniformly at random
Revised: Choose x ∈ val ( X ) uniformly at random

Original: Observe ( y, c other ) ∼ P
Revised: Observe ( y, c other ) ∼ P

Original: ( Y, C other | do ( x ) , c tar ) for V in C do updateBeliefs ( V, c ⟨ PA V ⟩ , c ⟨ V ⟩ ) //denote
Revised: ( Y, C other | do ( x ) , c tar ) for V in C do updateBeliefs ( V, c ⟨ PA V ⟩ , c ⟨ V ⟩ ) //denote

Original: c tar ∪ c other by c end updateBeliefs ( Y, ( c ⟨ PA Y ⟩ , x ) , c ⟨ Y ⟩ ) 9 end 10 (Phase 2) for t =
Revised: c tar ∪ c other by c end updateBeliefs ( Y, ( c ⟨ PA Y ⟩ , x ) , c ⟨ Y ⟩ ) 9 end 10 (Phase 2) for t =

Original: ⌈ αT ⌉ + 1 , . .
Revised: ⌈ αT ⌉ + 1 , . .

Original: , T do
Revised: , T do

Original: Choose x, c tar as: arg min x ∈ val ( X ) , c tar ∈ val ( C tar )   (cid:88) x ′ ∈ val ( X ) , c tar ′ ∈ val ( C tar )
Revised: Choose x, c tar as: arg min x ∈ val ( X ) , c tar ∈ val ( C tar )   (cid:88) x ′ ∈ val ( X ) , c tar ′ ∈ val ( C tar )

Original: Unc (cid:0) E
Revised: Unc (cid:0) E

Original: [ Y | do ( x ′ ) , c tar ′ ] (cid:12)(cid:12) x, c tar (cid:1)  
Revised: [ Y | do ( x ′ ) , c tar ′ ] (cid:12)(cid:12) x, c tar (cid:1)  

Original: Do targeted intervention ( x, c tar ) and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar ) for V in C other do updateBeliefs ( V, c ⟨ PA V ⟩ , c ⟨ V ⟩ ) end updateBeliefs ( Y, ( x, c ⟨ PA Y ⟩ ) , c ⟨ Y ⟩ ) 17 end
Revised: Do targeted intervention ( x, c tar ) and observe ( y, c other ) ∼ P ( Y, C other | do ( x ) , c tar ) for V in C other do updateBeliefs ( V, c ⟨ PA V ⟩ , c ⟨ V ⟩ ) end updateBeliefs ( Y, ( x, c ⟨ PA Y ⟩ ) , c ⟨ Y ⟩ ) 17 end

Original: Result: Final set of beliefs for all V, pa V : (cid:8) ..., θ V | pa V , ... (cid:9) 18 Procedure updateBeliefs( V, pa V , v )θ ( v ) V| pa V← θ ( v ) V | pa V +
Revised: Result: Final set of beliefs for all V, pa V : (cid:8) ..., θ V | pa V , ... (cid:9) 18 Procedure updateBeliefs( V, pa V , v ) v ) V pa V v ) V do do

Original: Data: Graph G , learned beliefs (cid:8) ..., θ V | pa V , ... (cid:9) , user context to be decided c tar 1 for every V, pa V do for v ∈ val ( V ) do

Original: Set
Revised: Set

Original: ˆ P( V = v | paV )
Revised: P pa

Original: = θ ( v ) V | pa V (cid:80) v ′ θ( v ′ )
Revised: ′ θ

Original: V | pa V end 5 end 6 for x ∈ val (X ) do
Revised: V | pa V end 5 end 6 for x ∈ val ( ) do

Original: Compute ˆψ ( x, c tar )≜ ˆ E[ Y | do (x ) , c tar ] using ˆ Pin Equation (2)
Revised: Compute ˆ c tar ) E do ( P

Original: 8 end
Revised: 8 end

Original: Result: Return
Revised: : Return

Original: We call our proposed algorithm Unc CCB .
Revised: We call our proposed algorithm Unc CCB .

Original: The training phase of Unc CCB (given as Algorithm 1a) consists of two phases.
Revised: The training phase of Unc CCB (given as Algorithm 1a) consists of two phases.

Original: In each round of Phase 1 (a fraction α of the rounds), it observes a context from the environment, then uniformly at random chooses an action, and finally observes the remaining variables.
Revised: In each round of Phase 1 (a fraction α of the rounds), it observes a context from the environment, then uniformly at random chooses an action, and finally observes the remaining variables.

Original: The observed values are used to perform standard Bayesian updates over the beliefs about parameters of all relevant CPDs.
Revised: The observed values are used to perform standard Bayesian updates over the beliefs about parameters of all relevant CPDs.

Original: Since we do not assume any a priori beliefs for the agent, this uniform exploration helps the agent enter Phase 2 with reasonable starting beliefs.
Revised: Since we do not assume any a priori beliefs for the agent, this uniform exploration helps the agent enter Phase 2 with reasonable starting beliefs.

Original: In Phase 2 (the remaining (1 − α ) fraction of the rounds) of training, the agent tries to allocate targeted interventions optimally.
Revised: In Phase 2 (the remaining (1 − α ) fraction of the rounds) of training, the agent tries to allocate targeted interventions optimally.

Original: To this end, it needs to trade-off between exploring new contexts and gaining more knowledge about already-explored contexts, while taking into account the information leakage resulting from the shared pathways in the causal graph (or equivalently, shared CPDs in the factorized representation of P ; Appendix E discusses a related subcase).
Revised: To this end, it needs to trade-off between exploring new contexts and gaining more knowledge about already-explored contexts, while taking into account the information leakage resulting from the shared pathways in the causal graph (or equivalently, shared CPDs in the factorized representation of P ; Appendix E discusses a related subcase).

Original: The novel, entropy-like,
Revised: The novel, entropy-like,

Original: Unc measure (discussed below) helps guide the agent in this decision in every round.
Revised: Unc measure (discussed below) helps guide the agent in this decision in every round.

Original: Algorithm 1b specifies the evaluation phase of Unc CCB , where the agent is evaluated on simple regret.
Revised: Algorithm 1b specifies the evaluation phase of Unc CCB , where the agent is evaluated on simple regret.

Original: Intuition behind the Unc measure As mentioned before, performing a targeted intervention given by ( x, c tar ) would allow the agent to update its beliefs about effects of other targeted interventions ( x ′ , c tar ′ ) .
Revised: Intuition behind the Unc measure As mentioned before, performing a targeted intervention given by ( x, c tar ) would allow the agent to update its beliefs about effects of other targeted interventions ( x ′ , c tar ′ ) .

Original: Intuitively, we would want the algorithm to allocate more samples to those targeted interventions that are most informative about the most “valuable” (in terms of reward Y ) targeted interventions.
Revised: Intuitively, we would want the algorithm to allocate more samples to those targeted interventions that are most informative about the most “valuable” (in terms of reward Y ) targeted interventions.

Original: Unc captures this intuition by providing a measure of the effect on the agent’s knowledge of E [ Y | do ( x ′ ) , c tar ′ ] when a targeted intervention ( x, c tar ) is performed.
Revised: Unc captures this intuition by providing a measure of the effect on the agent’s knowledge of E [ Y | do ( x ′ ) , c tar ′ ] when a targeted intervention ( x, c tar ) is performed.

Original: The agent, then aggregates this over all possible ( x ′ , c tar ′ ) , providing a measure of overall resulting knowledge from targeted intervention ( x, c tar ) ; this helps guide its sample allocation (see Line 11 of Algorithm 1a).
Revised: The agent, then aggregates this over all possible ( x ′ , c tar ′ ) , providing a measure of overall resulting knowledge from targeted intervention ( x, c tar ) ; this helps guide its sample allocation (see Line 11 of Algorithm 1a).

Original: Definition of the Unc measure We model the conditional distribution P ( V | pa V ) for any variable
Revised: Definition of the Unc measure We model the conditional distribution P ( V | pa V ) for any variable

Original: V as a categorical distribution whose parameters are sampled from a Dirichlet distribution.
Revised: V as a categorical distribution whose parameters are sampled from a Dirichlet distribution.

Original: That is, P ( V | pa V )
Revised: That is, P ( V | pa V )

Original: =
Revised: =

Original: Cat ( V ; b 1 , ..., b r ) , where ( b 1 , ..., b r )
Revised: Cat ( V ; b 1 , ..., b r ) , where ( b 1 , ..., b r )

Original: ∼ Dirc
Revised: ∼ Dirc

Original: ( θ V | pa V ) .
Revised: ( θ V | pa V ) .

Original: Here θ V | pa V is a vector of length, say, r .
Revised: Here θ V | pa V is a vector of length, say, r .

Original: Let denote θ V | pa V
Revised: Let denote θ V | pa V

Original: [ i ] denote the i ’th entry of θ V | pa V .
Revised: [ i ] denote the i ’th entry of θ V | pa V .

Original: We define an object called Ent that captures a measure of our knowledge of the CPD: 
Revised: We define an object called Ent that captures a measure of our knowledge of the CPD: Ent

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: Let the parents of V be PA V =
Revised: Let the parents of V be PA V =

Original: ( U 1 , ..., U p ) ; suppose they take a particular set of values pa V =
Revised: ( U 1 , ..., U p ) ; suppose they take a particular set of values pa V =

Original: ( u 1 , ..., u p ) .
Revised: ( u 1 , ..., u p ) .

Original: We define that a CPD P ( V | pa V ) is unaffected by targeted intervention ( x, c tar ) if there exists i ∈ { 1 , ..., p } such
Revised: We define that a CPD P ( V | pa V ) is unaffected by targeted intervention ( x, c tar ) if there exists i ∈ { 1 , ..., p } such

Original: that either (1) U
Revised: that either (1) U

Original: i is X and u i ̸
Revised: i is X and u i ̸

Original: = x , or (2) U i ∈
Revised: = x , or (2) U i ∈

Original: C tar and c tar ⟨ U i ⟩ ̸ = u i .
Revised: C tar and c tar ⟨ U i ⟩ ̸ = u i .

Original: In other words, the CPD P ( V | pa V ) is unaffected by ( x, c tar ) if doing this targeted intervention and observing all variables ( x, c tar , c other ) does not enable us to update beliefs about P ( V | pa V ) using knowledge of G .
Revised: In other words, the CPD P ( V | pa V ) is unaffected by ( x, c tar ) if doing this targeted intervention and observing all variables ( x, c tar , c other ) does not enable us to update beliefs about P ( V | pa V ) using knowledge of G .

Original: Now, define
Revised: Now, define

Original: [Equation]
Revised: [Equation]

Original: where Ent new is computed by averaging the resulting Ent values over the possible belief updates the agent might make after performing ( x, c tar ) .
Revised: where Ent new is computed by averaging the resulting Ent values over the possible belief updates the agent might make after performing ( x, c tar ) .

Original: That is, Ent new ( P ( V | pa V ))
Revised: That is, Ent new ( P ( V | pa V ))

Original: = 1 r (cid:80) i Ent (
Revised: = 1 r (cid:80) i Ent (

Original: Cat ( b ′ 1 , ..., b ′ r )) where ( b ′ 1 , ..., b ′ r ) ∼ Dirc (cid:0) ..., θ V | pa V [ i − 1] , θ V | pa V [ i ] + 1 , θ V | pa V [ i + 1] , ... (cid:1) .
Revised: Cat ( b ′ 1 , ..., b ′ r )) where ( b ′ 1 , ..., b ′ r ) ∼ Dirc (cid:0) ..., θ V | pa V [ i − 1] , θ V | pa V [ i ] + 1 , θ V | pa V [ i + 1] , ... (cid:1) .

Original: Letting c ′ denote c tar ′ ∪ c other ′ , we can now define Unc which captures our knowledge of E [ Y | do ( x ′ ) , c tar ′ ] if we perform some other targeted intervention given by ( x, c tar ) :
Revised: Letting c ′ denote c tar ′ ∪ c other ′ , we can now define Unc which captures our knowledge of E [ Y | do ( x ′ ) , c tar ′ ] if we perform some other targeted intervention given by ( x, c tar ) :

Original: [Equation]
Revised: [Equation]

Original: 3. R EGRET BOUND
Revised: 3. R EGRET BOUND

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any 0 < δ < 1 , with probability ≥ 1 − δ ,
Revised: For any 0 < δ < 1 , with probability ≥ 1 − δ ,

Original: Regret ≤ 3 E pa Y , c tar
Revised: Regret ≤ 3 E pa Y , c tar

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Due to space constraints, we only provide a proof sketch here.
Revised: Due to space constraints, we only provide a proof sketch here.

Original: The full proof is provided in
Revised: The full proof is provided in

Original: Appendix A as part of the Supplementary Text.
Revised: Appendix A as part of the Supplementary Text.

Original: For readability of the proof, we assume that Y and all variables in C are binary, whereas X can take on any finite set of values.
Revised: For readability of the proof, we assume that Y and all variables in C are binary, whereas X can take on any finite set of values.

Original: Under Assumption (A1) described in Section 2.1, we can factorize as follows:
Revised: Under Assumption (A1) described in Section 2.1, we can factorize as follows:

Original: [Equation]
Revised: [Equation]

Original: where c denotes c tar ∪ c other .
Revised: where c denotes c tar ∪ c other .

Original: The proof first bounds errors of estimates of ˆ P ( Y = 1 | x, c ⟨ PA Y ⟩ ) and ˆ P ( C = c
Revised: The proof first bounds errors of estimates of ˆ P ( Y = 1 | x, c ⟨ PA Y ⟩ ) and ˆ P ( C = c

Original: | c ⟨ PA C ⟩ ) with high probability using union bounds, concentration bounds, and the fact that the number of rounds in which the agent observes any set of context values is lower bounded by the number of times it sees it in Phase 1.
Revised: | c ⟨ PA C ⟩ ) with high probability using union bounds, concentration bounds, and the fact that the number of rounds in which the agent observes any set of context values is lower bounded by the number of times it sees it in Phase 1.

Original: We also make use of the fact that all variables are observed after every round.
Revised: We also make use of the fact that all variables are observed after every round.

Original: We then aggregate these bounds of individual CPD estimates to bound the overall estimate of ˆ E [ Y | do ( x ) , c tar ] , and then finally bound regret by utilizing the fact that the algorithm chooses arms based on the estimated ˆ E .
Revised: We then aggregate these bounds of individual CPD estimates to bound the overall estimate of ˆ E [ Y | do ( x ) , c tar ] , and then finally bound regret by utilizing the fact that the algorithm chooses arms based on the estimated ˆ E .

Original: Discussion Theorem 3.1 bounds the regret (with high probability).
Revised: Discussion Theorem 3.1 bounds the regret (with high probability).

Original: Note that T → ∞ =
Revised: Note that T → ∞ =

Original: ⇒
Revised: ⇒

Original: Regret → 0 , which shows that the regret bound has desirable limiting behavior.
Revised: Regret → 0 , which shows that the regret bound has desirable limiting behavior.

Original: 3 Further, regret is inversely related to √ T , similar to other simple-regret algorithms such as in Lattimore et al.
Revised: 2 Further, regret is inversely related to √ T , similar to other simple-regret algorithms such as in Lattimore et al.

Original: However, other terms that are a function of the space of possible interventions ( N C N X ) are different since we are in a contextual bandit setting (whereas Lattimore et al.
Revised: However, other terms that are a function of the space of possible interventions ( N C N X ) are different since we are in a contextual bandit setting (whereas Lattimore et al.

Original: (2016) is non-contextual); more specifically, regret is related proportionally to (cid:112) N X N C ln ( N X N C ) ; in fact, our regret bound is strictly tighter than this.
Revised: (2016) is non-contextual); more specifically, regret is related proportionally to (cid:112) N X N C ln ( N X N C ) ; in fact, our regret bound is strictly tighter than this.

Original: We prove the bound primarily to provide a theoretical guard on regret; we do not claim improved regret bounds since there is no comparable simple-regret contextual bandit regret bound.
Revised: We prove the bound primarily to provide a theoretical guard on regret; we do not claim improved regret bounds since there is no comparable simple-regret contextual bandit regret bound.

Original: We show improved performance over baselines empirically in Section 4.
Revised: We show improved performance over baselines empirically in Section 4.

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: Since there is no setting studied previously that directly maps to our setting, we adapt existing multi-armed bandit algorithms to make use of causal side-information and targeted interventions in a variety of ways to form a set of baselines against which we compare our algorithm’s performance.
Revised: Since there is no setting studied previously that directly maps to our setting, we adapt existing multi-armed bandit algorithms to make use of causal side-information and targeted interventions in a variety of ways to form a set of baselines against which we compare our algorithm’s performance.

Original: The baseline algorithms are the following.
Revised: The baseline algorithms are the following.

Original: Std TS and Std UniExp follow the standard bandit interaction protocol during training – observe context, choose action, receive reward.
Revised: Std TS and Std UniExp follow the standard bandit interaction protocol during training – observe context, choose action, receive reward.

Original: They treat each c tar as a separate problem and learn an optimal arm for each; they differ in that Std TS uses
Revised: They treat each c tar as a separate problem and learn an optimal arm for each; they differ in that Std TS uses

Original: Thompson Sampling (Agrawal & Goyal, 2012) for each c tar problem, whereas Std UniExp does uniform exploration over val ( X ) .
Revised: Thompson Sampling (Agrawal & Goyal, 2012) for each c tar problem, whereas Std UniExp does uniform exploration over val ( X ) .

Original: The other set of algorithms are TargInt TS , TargInt UniExp and TargInt TS UniExp .
Revised: The other set of algorithms are TargInt TS , TargInt UniExp and TargInt TS UniExp .

Original: These treat the space val ( X ) × val ( C tar ) as the set of arms for targeted interventions.
Revised: These treat the space val ( X ) × val ( C tar ) as the set of arms for targeted interventions.

Original: TargInt TS performs Thompson Sampling over this space, and TargInt UniExp performs uniform exploration over this space.
Revised: TargInt TS performs Thompson Sampling over this space, and TargInt UniExp performs uniform exploration over this space.

Original: TargInt TS UniExp is a variation of TargInt TS that, with some probability, 4 chooses a random targeted intervention; this infuses a controlled degree of uniform exploration into TargInt TS .
Revised: TargInt TS UniExp is a variation of TargInt TS that, with some probability, 3 chooses a random targeted intervention; this infuses a controlled degree of uniform exploration into TargInt TS .

Original: All five algorithms update CPD beliefs after each round.
Revised: All five algorithms update CPD beliefs after each round.

Original: No baseline is strictly stronger than another; their relative performance can vary based on the specific setting.
Revised: No baseline is strictly stronger than another; their relative performance can vary based on the specific setting.

Original: We also compare against a non-causal version of algorithm Std TS , which we call
Revised: We also compare against a non-causal version of algorithm Std TS , which we call

Original: NonCausal Std TS , to provide one non-causal baseline (i.e., which does not utilize the causal side-information).
Revised: NonCausal Std TS , to provide one non-causal baseline (i.e., which does not utilize the causal side-information).

Original: The evaluation phase for each baseline algorithm is analogous to Algorithm 1b – given a c tar , the action with the highest expected reward, based on beliefs at the end of the training phase, is chosen.
Revised: The evaluation phase for each baseline algorithm is analogous to Algorithm 1b – given a c tar , the action with the highest expected reward, based on beliefs at the end of the training phase, is chosen

Original: 4. P URELY SYNTHETIC DATA
Revised: 4. P URELY SYNTHETIC DATA

Original: We consider a causal model M whose causal graph G consists of the following edges: C 1 →
Revised: We consider a causal model M whose causal graph G consists of the following edges: C 1 →

Original: C 0 , C 

Revised: C 0 , C

Original: → X, C 

Revised: → X, C

Original: → Y, X → Y .
Revised: → Y, X → Y .

Original: We assume that C tar =
Revised: We assume that C tar =

Original: { C 1 } and that C other = { C 0 } .
Revised: { C 1 } and that C other = { C 0 } .

Original: We choose the fraction of Phase 1 rounds α = 0 .
Revised: We choose the fraction of Phase 1 rounds α = 0 .

Original: 5 as we empirically found that to be a good choice.
Revised: 5 as we empirically found that to be a good choice.

Original: More details and code are in the Supplementary Material.
Revised: More details and code are in the Supplementary Material.

Original: Additional experiments, analyzing the reason for why our algorithm performs better than baselines, are in Appendix B.
Revised: Additional experiments, analyzing the reason for why our algorithm performs better than baselines, are in Appendix B.

Original: Experiment 1 Given the wide range of possible settings arising from different parameterizations of M , the first experiment seeks to answer the question: “what is the expected performance of our algorithm over a set of representative settings ?”.
Revised: Experiment 1 Given the wide range of possible settings arising from different parameterizations of M , the first experiment seeks to answer the question: “what is the expected performance of our algorithm over a set of representative settings ?”.

Original: Each setting can be interpreted naturally when, for example, we think of C tar as the set of user features that a recommendation agent observes post-deployment.
Revised: Each setting can be interpreted naturally when, for example, we think of C tar as the set of user features that a recommendation agent observes post-deployment.

Original: The settings are chosen to capture the intuition that, typically, the agent sees highvalue contexts (i.e., contexts for which, if the agent learns the optimal action, can fetch high rewards) relatively less frequently, say, 20% of the time, but there can be variation in the number of different
Revised: The settings are chosen to capture the intuition that, typically, the agent sees highvalue contexts (i.e., contexts for which, if the agent learns the optimal action, can fetch high rewards) relatively less frequently, say, 20% of the time, but there can be variation in the number of different

Original: C tar values over which that probability mass is spread and in how “risky” they are (for example, very low rewards for non-optimal actions).
Revised: C tar values over which that probability mass is spread and in how “risky” they are (for example, very low rewards for non-optimal actions).

Original: In each run, the agent is presented with a randomly selected setting from this representative set.
Revised: In each run, the agent is presented with a randomly selected setting from this representative set.

Original: Results are averaged over 300 independent runs; error bars display ± 1 .
Revised: Results are averaged over 300 independent runs; error bars display ± 1 .

Original: 96 standard errors.
Revised: 96 standard errors.

Original: For details on specific parameterizations, refer Appendix D.
Revised: For details on specific parameterizations, refer Appendix D.

Original: Figure 1 provides the results of this experiment.
Revised: Figure 1 provides the results of this experiment.

Original: The axes are normalized since the representative set of settings could have different ranges for T and for regret.
Revised: The axes are normalized since the representative set of settings could have different ranges for T and for regret.

Original: We see that our algorithm performs better than all baselines, especially for lower values of T .
Revised: We see that our algorithm performs better than all baselines, especially for lower values of T .

Original: This, in fact, is a recurring theme – our algorithm’s performance improvement is more pronounced in low T-budget situations, which, as we stated in Section 1.1, is what we are interested in.
Revised: This, in fact, is a recurring theme – our algorithm’s performance improvement is more pronounced in low T-budget situations, which, as we stated in Section 1.1, is what we are interested in.

Original: Further, Appendix C contains plots for two individual settings from the set, one where Std TS and Std UniExp perform significantly better than TargInt TS , TargInt UniExp and TargInt TS UniExp , and another where it is the converse; in contrast, our algorithm performs close to the best baseline in both.
Revised: Further, Appendix C contains plots for two individual settings from the set, one where Std TS and Std UniExp perform significantly better than TargInt TS , TargInt UniExp and TargInt TS UniExp , and another where it is the converse; in contrast, our algorithm performs close to the best baseline in both.

Original: The intuition behind this is that the Unc measure incentivizes exploring new context values (due to its entropy-like nature) while weighting more the contexts that are likely to yield more rewards (based on our current estimates).
Revised: The intuition behind this is that the Unc measure incentivizes exploring new context values (due to its entropy-like nature) while weighting more the contexts that are likely to yield more rewards (based on our current estimates).

Original: Also, as expected, NonCausal Std TS performs the worst as it ignores the causal graph.
Revised: Also, as expected, NonCausal Std TS performs the worst as it ignores the causal graph.

Original: Experiment 2 To ensure that the results are not biased due to our choice of the representative set, the second experiment asks “what happens to performance when we directly randomize the parameters of the CPDs in each run, subject to realistic constraints?”.
Revised: Experiment 2 To ensure that the results are not biased due to our choice of the representative set, the second experiment asks “what happens to performance when we directly randomize the parameters of the CPDs in each run, subject to realistic constraints?”.

Original: Specifically, in each run, we (1) randomly pick an i ∈ { 1 , ..., ⌊ N C 1 / 2 ⌋} , (2) distribute 20% of the probability mass randomly over C 1 values { 1 , ..., i } , (3) distribute the remaining 80% of the mass over C 1 values { i +1 , ..., N C 1 } .
Revised: Specifically, in each run, we (1) randomly pick an i ∈ { 1 , ..., ⌊ N C 1 / 2 ⌋} , (2) distribute 20% of the probability mass randomly over C 1 values { 1 , ..., i } , (3) distribute the remaining 80% of the mass over C 1 values { i +1 , ..., N C 1 } .

Original: The c 1 values in the 20% bucket are higher-value, while those in the 80% bucket are lower-value.
Revised: The c 1 values in the 20% bucket are higher-value, while those in the 80% bucket are lower-value.

Original: Intuitively, this captures the commonly observed 80-20 pattern (for example, 20% of the users contribute to around 80% of the revenue), but randomizes the other aspects; this gives an estimate of how the algorithms would perform on expectation.
Revised: Intuitively, this captures the commonly observed 80-20 pattern (for example, 20% of the users contribute to around 80% of the revenue), but randomizes the other aspects; this gives an estimate of how the algorithms would perform on expectation.

Original: The results are averaged over 100 independent runs; error bars display ± 1 .
Revised: The results are averaged over 100 independent runs; error bars display ± 1 .

Original: 96 standard errors.
Revised: 96 standard errors.

Original: Figure 2 shows that our algorithm performs better than all baselines in this experiment, with a more pronounced improvement for the smaller values of T .
Revised: Figure 2 shows that our algorithm performs better than all baselines in this experiment, with a more pronounced improvement for the smaller values of T .

Original: For instance, when T = 24 , our algorithm’s mean regret is around 35% lower than that of the next best baseline.
Revised: For instance, when T = 24 , our algorithm’s mean regret is around 35% lower than that of the next best baseline.

Original: 4. CRM SALES DATA - INSPIRED EXPERIMENT
Revised: 4. CRM SALES DATA - INSPIRED EXPERIMENT

Original: Experiment 

Revised: Experiment

Original: This experiment seeks to answer the question: “how does the algorithm perform on realistic scenarios ?”.
Revised: This experiment seeks to answer the question: “how does the algorithm perform on realistic scenarios ?”.

Original: We setup the experiment inspired by a real world scenario.
Revised: We setup the experiment inspired by a real world scenario.

Original: Consider a bandit agent that could assist salespeople by learning to decide how many outgoing calls to make in an ongoing deal, given just the type of deal (new business, etc.)
Revised: Consider a bandit agent that could assist salespeople by learning to decide how many outgoing calls to make in an ongoing deal, given just the type of deal (new business, etc.)

Original: and size of customer (small, medium, etc.)
Revised: and size of customer (small, medium, etc.)

Original: , so as to maximize a reward metric (which we call the ‘expected deal value’).
Revised: , so as to maximize a reward metric (which we call the ‘expected deal value’).

Original: Deal type affects the reward via the source of the lead (for example, chat).
Revised: Deal type affects the reward via the source of the lead (for example, chat).

Original: The trained agent would be deployed internally or to the company’s clients, where it would generally not have access to lead source.
Revised: The trained agent would be deployed internally or to the company’s clients, where it would generally not have access to lead source.

Original: The causal graph relating these variables is given in Figure 3a, which was obtained using domain knowledge.
Revised: The causal graph relating these variables is given in Figure 3a, which was obtained using domain knowledge.

Original: ‘Deal type’ and ‘Customer size’ are C tar , ‘Lead source’ is C other , ‘# outgoing calls’ is X , and ‘Exp. deal value’ is Y .
Revised: ‘Deal type’ and ‘Customer size’ are C tar , ‘Lead source’ is C other , ‘# outgoing calls’ is X , and ‘Exp. deal value’ is Y .

Original: The parameters of each CPD corresponding to this causal graph was calibrated using proprietary sales data from a CRM company, after some adjustments using domain knowledge (for example, where there were coverage gaps).
Revised: The parameters of each CPD corresponding to this causal graph was calibrated using proprietary sales data from Freshworks Inc., after some adjustments using domain knowledge (for example, where there were coverage gaps).

Original: Note that the algorithms use only the CPDs and not this raw data; in any case, we make the dataset available in anonymized form at https://doi.org/10.5281/zenodo.5540348 under the ‘CC BY-NC 4.0’ license.
Revised: Note that the algorithms use only the CPDs and not this raw data; in any case, we make the dataset available in anonymized form at https://doi.org/10.5281/zenodo.5540348 under the ‘CC BY-NC 4.0’ license.

Original: Further, we also do a warm-start for all algorithms to account for the fact that, in practice, there is often some starting beliefs about the CPDs from past data, domain knowledge, etc., which can be encoded into the agent’s prior beliefs.
Revised: Further, we also do a warm-start for all algorithms to account for the fact that, in practice, there is often some starting beliefs about the CPDs from past data, domain knowledge, etc., which can be encoded into the agent’s prior beliefs.

Original: Specifically, we ran pure random exploration for 15 rounds at the beginning of each algorithm and updated all CPD beliefs; this simulates a warm-start.
Revised: Specifically, we ran pure random exploration for 15 rounds at the beginning of each algorithm and updated all CPD beliefs; this simulates a warm-start.

Original: Due to this, we used α = 0 for our algorithm.
Revised: Due to this, we used α = 0 for our algorithm.

Original: Results are averaged over 50 independent runs; error bars display ± 1 .
Revised: Results are averaged over 50 independent runs; error bars display ± 1 .

Original: 96 standard errors.
Revised: 96 standard errors.

Original: Figure 3 shows that our algorithm performs better than all baselines in this realistic setting as well.
Revised: Figure 3 shows that our algorithm performs better than all baselines in this realistic setting as well.

Original: The Supplementary Material provides the code.
Revised: The Supplementary Material provides the code.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION AND FUTURE DIRECTIONS
Revised:  C ONCLUSION AND FUTURE DIRECTIONS

Original: This work presented a contextual bandits formulation that captures real-world nuances such as the ability to conduct targeted interventions and the presence of causal side-information, along with a novel algorithm that exploits this to achieve improved sample efficiency.
Revised: This work presented a contextual bandits formulation that captures real-world nuances such as the ability to conduct targeted interventions and the presence of causal side-information, along with a novel algorithm that exploits this to achieve improved sample efficiency.

Original: In addition to synthetic experiments, we also performed real world-inspired experiments set up using actual CRM sales data.
Revised: In addition to synthetic experiments, we also performed real world-inspired experiments set up using actual CRM sales data.

Original: A useful direction of future investigation is the development of algorithms when G is unknown or partially known.
Revised: A useful direction of future investigation is the development of algorithms when G is unknown or partially known.

Original: Another important direction is the development of methods robust to distributional shifts.
Revised: Another important direction is the development of methods robust to distributional shifts.

Original: Distributional shifts have been studied for prediction tasks (Magliacane et al., 2018;
Revised: Distributional shifts have been studied for prediction tasks (Magliacane et al., 2018;

Original: Gong et al., 2016) and for causal effects (Bareinboim & Pearl, 2014; Correa & Bareinboim, 2020); it is an interesting question to study this in our setting, when G remains the same between training and evaluation but M changes.
Revised: Gong et al., 2016) and for causal effects (Bareinboim & Pearl, 2014; Correa & Bareinboim, 2020); it is an interesting question to study this in our setting, when G remains the same between training and evaluation but M changes.

Revised: A CKNOWLEDGEMENTS

Revised: This work was partly funded by Freshworks Inc. through a research grant to Balaraman Ravindran and Chandrasekar Subramanian.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: The full proof of Theorem 3.1 is provided in an appendix at the end of this PDF file (after the references) as part of the Supplementary Text.
Revised: The full proof of Theorem 3.1 is provided in an appendix at the end of this PDF file (after the references) as part of the Supplementary Text.

Original: The source code of all the experiments in the main paper, along with a README on how to run them, is provided as part of the Supplementary Materials in a file named Supplemental Code Paper1203.zip .
Revised: The source code of all the experiments in the main paper, along with a README on how to run them, is provided as part of the Supplementary Materials in a file named Supplemental Code Paper1203.zip .

Original: Further, even though the algorithms use only the CPDs and not the raw data, the dataset used to calibrate the real worldinspired experiments (see Section 4.2 of main paper) is made available in anonymized form at https://doi.org/10.5281/zenodo.5540348 under the ‘CC BY-NC 4.0’ license.
Revised: Further, even though the algorithms use only the CPDs and not the raw data, the dataset used to calibrate the real worldinspired experiments (see Section 4.2 of main paper) is made available in anonymized form at https://doi.org/10.5281/zenodo.5540348 under the ‘CC BY-NC 4.0’ license.

Original: The other experiments were synthetic and did not involve the use of any dataset.
Revised: The other experiments were synthetic and did not involve the use of any dataset.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: This work is a foundational work that provides an improved mathematical framework and algorithm for contextual bandits.
Revised: This work is a foundational work that provides an improved mathematical framework and algorithm for contextual bandits.

Original: Some experiments were set up utilizing data from a CRM software company (identity witheld for anonymity); the data is released in anonymized form with the consent of the company (link to dataset given in the main paper).
Revised: Some experiments were set up utilizing data from Freshworks Inc.; the data is released in anonymized form with the consent of the company (link to dataset given in the main paper).

Original: To the best of our knowledge, we do not believe there were any ethical issues associated with the development of this work.
Revised: To the best of our knowledge, we do not believe there were any ethical issues associated with the development of this work.

Original: Further, given the nature of the work as foundational and introducing a new algorithm (and not specific to an application), we do not foresee any specific potential negative ethical issues created by this work.
Revised: Further, given the nature of the work as foundational and introducing a new algorithm (and not specific to an application), we do not foresee any specific potential negative ethical issues created by this work.

Original: However, we do point out that researchers utilizing this method to their specific applications should adhere to ethical standards of their own (e.g., by avoiding targeting interventions on subpopulations based on racial attributes).
Revised: However, we do point out that researchers utilizing this method to their specific applications should adhere to ethical standards of their own (e.g., by avoiding targeting interventions on subpopulations based on racial attributes).

Original: A A PPENDIX (S UPPLEMENTARY T EXT ): P ROOF OF T HEOREM 3. 
Revised: A A PPENDIX (S UPPLEMENTARY T EXT ): P ROOF OF T HEOREM 3.

Original: This is the full version of the proof sketch presented in Section 3.2 of the paper.
Revised: This is the full version of the proof sketch presented in Section 3.2 of the paper.

Original: For readability of the proof, we assume that Y and all variables in C are binary, whereas X can have any finite set of values.
Revised: For readability of the proof, we assume that Y and all variables in C are binary, whereas X can have any finite set of values.

Original: In Sections A.1 through A.5, we derive the expression for bounding Regret ( c tar ) , which is the simple regret given C tar = c tar .
Revised: In Sections A.1 through A.5, we derive the expression for bounding Regret ( c tar ) , which is the simple regret given C tar = c tar .

Original: Then, in Section A.6, we derive the final expression for bounding overall Regret.
Revised: Then, in Section A.6, we derive the final expression for bounding overall Regret.

Original: In addition to the notation in Table 1 of the main paper, we use a few other notations to keep the proof more readable:
Revised: In addition to the notation in Table 1 of the main paper, we use a few other notations to keep the proof more readable:

Original: • c denotes c tar ∪ c other .
Revised: • c denotes c tar ∪ c other .

Original: • a ⟨B⟩ is shortened to b when it is unambiguous what a is; for example, c ⟨ PA V ⟩ is shortened to pa V when it is obvious that c is the current set of values taken by all the context variables.
Revised: • a ⟨B⟩ is shortened to b when it is unambiguous what a is; for example, c ⟨ PA V ⟩ is shortened to pa V when it is obvious that c is the current set of values taken by all the context variables.

Original: • We write P ( V = v | pa V ) interchangeably with P ( v | pa V ) since we use small letters (e
Revised: • We write P ( V = v | pa V ) interchangeably with P ( v | pa V ) since we use small letters (e

Original: g., v ) to denote the value taken by the random variable denoted by the respective capital letter (e.g., V ).
Revised: g., v ) to denote the value taken by the random variable denoted by the respective capital letter (e.g., V ).

Original: Now, the factorization under Assumption (A1) can be written in this simpler notation as:
Revised: Now, the factorization under Assumption (A1) can be written in this simpler notation as:

Original: [Equation]
Revised: [Equation]

Original: Recollect that we use set operations on vectors as well, wherever there is no ambiguity.
Revised: Recollect that we use set operations on vectors as well, wherever there is no ambiguity.

Original: A. B OUNDING THE ERROR OF P ( Y = 1 | x, pa Y ) ESTIMATES
Revised: A. B OUNDING THE ERROR OF P ( Y = 1 | x, pa Y ) ESTIMATES

Original: Let pa Y denote the value taken by PA Y (i.e., Y ’s parents), excluding x .
Revised: Let pa Y denote the value taken by PA Y (i.e., Y ’s parents), excluding x .

Original: Say, for a ( x, pa Y ) ,
Revised: Say, for a ( x, pa Y ) ,

Original: [Equation]
Revised: [Equation]

Original: Using the union bound, we can rewrite it as
Revised: Using the union bound, we can rewrite it as

Original: [Equation]
Revised: [Equation]

Original: In other words, with probability ≥ 1 − δ X, pa Y , the following event E X, pa Y is true:
Revised: In other words, with probability ≥ 1 − δ X, pa Y , the following event E X, pa Y is true:

Original: [Equation]
Revised: [Equation]

Original: We derive expressions for ϵ X, pa Y and δ X, pa Y in Lemma A.2.
Revised: We derive expressions for ϵ X, pa Y and δ X, pa Y in Lemma A.2.

Original: A. B OUNDING THE ERROR OF P ( C = c | PA C ) ESTIMATES
Revised: A. B OUNDING THE ERROR OF P ( C = c | PA C ) ESTIMATES

Original: Now, say, for a C ∈ C (taking value C = c ), and for given pa C ,
Revised: Now, say, for a C ∈ C (taking value C = c ), and for given pa C ,

Original: [Equation]
Revised: [Equation]

Original: Note that ˆ P (0 | pa C ) + ˆ P (1 | pa C )
Revised: Note that ˆ P (0 | pa C ) + ˆ P (1 | pa C )

Original: = 1 and P (0 | pa C )
Revised: = 1 and P (0 | pa C )

Original: + P (1 | pa C )
Revised: + P (1 | pa C )

Original: = 1 .
Revised: = 1 .

Original: Now, say the actual errors in estimates are given by ˜ ϵ c .
Revised: Now, say the actual errors in estimates are given by ˜ ϵ c .

Original: That is, ˜ ϵ c ≜
Revised: That is, ˜ ϵ c ≜

Original: ˆ P ( c | pa C ) − P ( c | pa C ) .
Revised: ˆ P ( c | pa C ) − P ( c | pa C ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Now, this means that
Revised: Now, this means that

Original: [Equation]
Revised: [Equation]

Original: a Therefore, the event | ˜ ϵ 0 | pa C | ≤ a is the same event as the event | ˜ ϵ 1 | pa C | ≤ a .
Revised: Therefore, the event | ˜ ϵ 0 | pa C | ≤ a is the same event as the event | ˜ ϵ 1 | pa C | ≤ a .

Original: Therefore, given a pa C , the following is true:
Revised: Therefore, given a pa C , the following is true:

Original: [Equation]
Revised: [Equation]

Original: ≥ 1 − δ C | pa C which is the same as
Revised: ≥ 1 − δ C | pa C which is the same as

Original: [Equation]
Revised: [Equation]

Original: where δ C | pa C is a function of ϵ C | pa C .
Revised: where δ C | pa C is a function of ϵ C | pa C .

Original: Note that we now have δ C | pa C instead of δ c | pa C .
Revised: Note that we now have δ C | pa C instead of δ c | pa C .

Original: In other words, given a pa C , with probability ≥ 1 − δ C | pa C , the following event E C | pa C is true:
Revised: In other words, given a pa C , with probability ≥ 1 − δ C | pa C , the following event E C | pa C is true:

Original: [Equation]
Revised: [Equation]

Original: We derive expressions for δ C | pa C and ϵ C | pa C in Lemma A.1.
Revised: We derive expressions for δ C | pa C and ϵ C | pa C in Lemma A.1.

Original: A. P UTTING THEM TOGETHER FOR THE OVERALL BOUND FOR R EGRET ( c tar )
Revised: A. P UTTING THEM TOGETHER FOR THE OVERALL BOUND FOR R EGRET ( c tar )

Original: Using the union bound, for any c tar , with probability ≥ 1 − (cid:80) pa Y δ X, pa Y − (cid:80) C ∈C (cid:80) pa C δ C | pa C , all of E X, pa Y and E C | pa C are true simultaneously ∀ pa Y and ∀ pa C ∀ C ∈
Revised: Using the union bound, for any c tar , with probability ≥ 1 − (cid:80) pa Y δ X, pa Y − (cid:80) C ∈C (cid:80) pa C δ C | pa C , all of E X, pa Y and E C | pa C are true simultaneously ∀ pa Y and ∀ pa C ∀ C ∈

Original: ( C \ PA Y ) .
Revised: ( C \ PA Y ) .

Original: Let’s call this event E .
Revised: Let’s call this event E .

Original: In other words, under event E , all estimates are bounded.
Revised: In other words, under event E , all estimates are bounded.

Original: Let a alg =
Revised: Let a alg =

Original: ˆ ϕ ( c tar ) denote the action chosen by the algorithm in the evaluation phase when presented with c tar ; and let a ∗ denote an optimal action for c tar .
Revised: ˆ ϕ ( c tar ) denote the action chosen by the algorithm in the evaluation phase when presented with c tar ; and let a ∗ denote an optimal action for c tar .

Original: Therefore, given C tar = c tar , under event E ,
Revised: Therefore, given C tar = c tar , under event E ,

Original: E [ Y | do ( a alg ) , c tar ] = (cid:88)
Revised: E [ Y | do ( a alg ) , c tar ] = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (Factorization due to G as earlier)
Revised: (Factorization due to G as earlier)

Original: [Equation]
Revised: [Equation]

Original: (Since ˜ ϵ C | pa C are actual errors.) ≥
Revised: (Since ˜ ϵ C | pa C are actual errors.) ≥

Original: [Equation]
Revised: [Equation]

Original: c other { 
Revised: c other { ˆ

Original: [Equation]
Revised: [Equation]

Original: (cid:89)
Revised: (cid:89)

Revised: [Equation]

Revised: other { ˆ

Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Revised: pa

Revised: [Equation]

Original: (Due to event E ) 
Revised: (Due to event E ) ≥

Original: [Equation]
Revised: [Equation]

Original: do
Revised: do

Original: [Equation]
Revised: [Equation]

Revised: − (cid:88)

Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x

Original: [Equation] a
Revised: = a

Original: [Equation]
Revised: [Equation]

Original: (Algebra discussed separately in Section A.3.1)
Revised: (Algebra discussed separately in Section A.3.1)

Original: [Equation]
Revised: [Equation]

Original: do
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x

Original: [Equation]
Revised: [Equation]

Original:  (Since Y is binary) ≥ ˆ E
Revised: a alg (Since Y is binary) ≥ ˆ E

Original: [ Y | do ( a ∗ ) , c tar ] − (cid:88) 
Revised: [ Y | do ( a ∗ ) , c tar ] − (cid:88) C

Original: [Equation]
Revised: [Equation]

Original:  (cid:88) 
Revised: Y (cid:88) pa

Revised: [Equation]

Revised: P

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12) 
Revised: (cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12) x

Original: [Equation]
Revised: [Equation]

Original: a alg (Due to proposed algorithm)
Revised: a alg (Due to proposed algorithm)

Original: Continuing and applying similar steps, we get
Revised: Continuing and applying similar steps, we get

Original: E [ Y | do ( a alg ) , c tar ] ≥ E
Revised: [Equation]

Original: [ Y | do ( a ∗ ) , c tar ] − 2 (cid:88) pa Y P
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Defining

Original: [Equation]
Revised: [Equation]

Original: (cid:88) C
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: C

Original: Defining ϵ ′ X ≜ (cid:88) pa Y

Original: P ( pa Y | c tar ) ϵ X, pa Y and ϵ ′ C ≜ (cid:88) pa C

Original: P ( pa C | c tar )

Original: [Equation]
Revised: [Equation]

Original: we get that,
Revised: we get that,

Original: [Equation]
Revised: [Equation]

Original: Therefore, under event E (that is, with probability ≥ 1 − (cid:80) pa Y δ X, pa Y − (cid:80) C ∈C (cid:80) pa C δ C | pa C ), for any given c tar ,
Revised: Therefore, under event E (that is, with probability ≥ 1 − (cid:80) pa Y δ X, pa Y − (cid:80) C ∈C (cid:80) pa C δ C | pa C ), for any given c tar ,

Original: [Equation]
Revised: [Equation]

Original: We derive expressions for each of these terms in Sections A.4 and A.5, and then arrive at the final bound in Section A.
Revised: We derive expressions for each of these terms in Sections A.4 and A.5, and then arrive at the final bound in Section A.

Original: [Equation]
Revised: [Equation]

Original: A.3. A LGEBRA FOR INTERMEDIATE EXPRESSION
Revised: A.3. A LGEBRA FOR INTERMEDIATE EXPRESSION

Original: We wish to show that
Revised: We wish to show that

Original: [Equation]
Revised: [Equation]

Original: Let us denote the expression on the left hand side by ( A ).
Revised: Let us denote the expression on the left hand side by ( A )

Revised: [Equation]

Original: Let c i ∈
Revised: Let c i ∈

Original: c other chosen in a reverse topological order; this ensures that a child is chosen before any of its parents.
Revised: c other chosen in a reverse topological order; this ensures that a child is chosen before any of its parents.

Original: Let C i be the corresponding variable in C other .
Revised: Let C i be the corresponding variable in C other .

Original: Then we can write
Revised: Then we can write

Original: [Equation]
Revised: [Equation]

Original: ≥

Original: [Equation] pa [Equation] where ˆ Q D [Y = 1 | x, c ] ≜
Revised: where ˆ Q D [

Original: [Equation]
Revised: [Equation]

Original: d [Equation] } and ˆ ϵ X, pa Ci ≜ [Equation]
Revised: = 1 | x

Original: c

Original: [Equation]
Revised: [Equation]

Original: ϵ

Original: [Equation]
Revised: [Equation]

Original: pa
Revised: and ˆ

Original: [Equation]
Revised: [Equation]

Original:  paC
Revised: X, pa

Original: [Equation]
Revised: [Equation]

Original: These make use of the fact that ˜ ϵ
Revised: These make use of the fact that ˜ ϵ

Original: c i | pa C i ≤ ϵ C i | pa C i , ˜ ϵ 

Revised: c i | pa C i ≤ ϵ C i | pa C i , ˜ ϵ

Original: | pa C i = − ˜
Revised: | pa C i = − ˜

Original: ϵ 1 | pa C i and that ˆ P ≤ 1 .
Revised: ϵ 1 | pa C i and that ˆ P ≤ 1 .

Original: We can keep reapplying the above steps in reverse topological order till we exhaust all
Revised: We can keep reapplying the above steps in reverse topological order till we exhaust all

Original: c i , and we’ll get
Revised: c i , and we’ll get

Original: [Equation]
Revised: [Equation]

Original: Noting that ˆ Q C other [ Y = 1 | x, c tar ] = ˆ P ( 
Revised: Noting that ˆ Q C other [ Y = 1 | x, c tar ] = ˆ P ( Y

Original: A. E XPRESSIONS FOR δ C | pa C AND ϵ C | pa C
Revised: A. E XPRESSIONS FOR δ C | pa C AND ϵ C | pa C

Original: Lemma A.1.
Revised: Lemma A.1.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let T pa C be the set of time indices during training when PA C = pa C was chosen/seen by the algorithm, and let T pa C = |T pa C | .
Revised: Let T pa C be the set of time indices during training when PA C = pa C was chosen/seen by the algorithm, and let T pa C = |T pa C | .

Original: Now, for a C that is observed and not chosen 6 , note that our estimate of ˆ P ( C = 1 | pa C ) is computed as ( θ (1) C | pa C + 1) /
Revised: Now, for a C that is observed and not chosen 5 , note that our estimate of ˆ P ( C = 1 | pa C ) is computed as ( θ (1) C | pa C + 1) /

Original: ( T pa C + 2) .
Revised: ( T pa C + 2) .

Original: We approximate 7 this as θ (1) C | pa C /T pa C .
Revised: We approximate 6 this as θ (1) C | pa C /T pa C .

Original: Using Hoeffding’s inequality, for any c ,
Revised: Using Hoeffding’s inequality, for any c ,

Original: [Equation]
Revised: [Equation]

Original: But since already observed, the two events (corresponding to c = 0 and c = 1 ) are the same, we have
Revised: But since already observed, the two events (corresponding to c = 0 and c = 1 ) are the same, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we let
Revised: Therefore, we let

Original: [Equation]
Revised: [Equation]

Original: Now, let T ′ ≜
Revised: Now, let T ′ ≜

Original: αT be the number of rounds of Phase 1.
Revised: αT be the number of rounds of Phase 1.

Original: Let ˜ T C | pa C be the number of times C was updated in Phase 1 as a results of encountering PA C = pa C .
Revised: Let ˜ T C | pa C be the number of times C was updated in Phase 1 as a results of encountering PA C = pa C .

Original: Note that the mean of ˜ T C | pa C is
Revised: Note that the mean of ˜ T C | pa C is

Original: T ′ P ( c tar ) P ( pa C | c tar ) =
Revised: T ′ P ( c tar ) P ( pa C | c tar ) =

Original: T ′ P ( pa C , c tar ) .
Revised: T ′ P ( pa C , c tar ) .

Original: By the Hoeffding’s inequality,
Revised: By the Hoeffding’s inequality,

Original: [Equation]
Revised: [Equation]

Original: By the union bound, we have (let’s call the event as E TC | PA C ),
Revised: By the union bound, we have (let’s call the event as E TC | PA C ),

Original: [Equation]
Revised: [Equation]

Original: where N PA C =
Revised: where N PA C =

Original: (cid:81) C ∈ PA C N C .
Revised: (cid:81) C ∈ PA C N C .

Original: Letting δ TC | PA C ≜ N PA C exp (cid:18) −
Revised: Letting δ TC | PA C ≜ N PA C exp (cid:18) −

Original: [Equation]
Revised: [Equation]

Original: (cid:19) , we have
Revised: (cid:19) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that T C | pa C ≥ ˜ T C | pa C .
Revised: Note that T C | pa C ≥ ˜ T C | pa C .

Original: Therefore, given that E TC | PA C is true, we have
Revised: Therefore, given that E TC | PA C is true, we have

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Definingδ C | pa C as
Revised: Defining

Original: [Equation]
Revised: [Equation]

Original: gives us
Revised: gives us

Original: [Equation]
Revised: [Equation]

Original: A. E XPRESSIONS FOR δ X, pa Y , ϵ X, pa Y , AND ϵ ′ x
Revised: A. E XPRESSIONS FOR δ X, pa Y , ϵ X, pa Y , AND ϵ ′ x

Original: Lemma A.2.
Revised: Lemma A.2.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As before, let T x, pa Y be the set of time indices during training when ( X, PA Y )
Revised: As before, let T x, pa Y be the set of time indices during training when ( X, PA Y )

Original: = ( x, pa Y ) was chosen/seen by the algorithm, and let T x, pa Y =
Revised: = ( x, pa Y ) was chosen/seen by the algorithm [Equation]

Original:  |T x, pa Y | .
Revised: and let T x, pa Y = |T x, pa Y | .

Original: As before, note that our estimate of ˆ P ( Y = 1 | x, pa Y ) is computed as ( θ (1) Y | x, pa Y + 1) /
Revised: As before, note that our estimate of ˆ P ( Y = 1 | x, pa Y ) is computed as ( θ (1) Y | x, pa Y + 1) /

Original: ( T x, pa Y + 2) , which we can approximate as θ (1) Y | x, pa Y /T x, pa Y .
Revised: ( T x, pa Y + 2) , which we can approximate as θ (1) Y | x, pa Y /T x, pa Y .

Original: Now, by Hoeffding’s inequality
Revised: Now, by Hoeffding’s inequality

Original: [Equation]
Revised: [Equation]

Original: During Phase 1, let ˜ T x, pa Y be the random variable denoting the number of times that ( X, PA Y )
Revised: During Phase 1, let ˜ T x, pa Y be the random variable denoting the number of times that ( X, PA Y )

Original: =
Revised: =

Original: ( x, pa Y ) was seen/chosen.
Revised: ( x, pa Y ) was seen/chosen.

Original: Note that, due to the nature of Phase 1 (i.e., each x is chosen with probability 1 N X ), the mean of ˜ T x, pa Y is T ′ P ( pa Y , c tar ) 1 N X .
Revised: Note that, due to the nature of Phase 1 (i.e., each x is chosen with probability 1 N X ), the mean of ˜ T x, pa Y is T ′ P ( pa Y , c tar ) 1 N X .

Original: Therefore, using the union bound and the Hoeffding’s inequality,
Revised: Therefore, using the union bound and the Hoeffding’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Since ∀ ( x, pa Y ) , T x, pa Y ≥ ˜ T x, pa Y , we have (let’s call this event E TX,PA Y ),
Revised: Since ∀ ( x, pa Y ) , T x, pa Y ≥ ˜ T x, pa Y , we have (let’s call this event E TX,PA Y ),

Original: P(cid:20) ∀ ( x, pa Y ) , T x, pa Y ≥
Revised: P

Original: [Equation]
Revised: [Equation]

Original: P
Revised: ∀ ( x, pa Y ) , T x, pa Y ≥

Original: [Equation]
Revised: [Equation]

Original: pa
Revised: pa

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: Letting δ TX,PA Y ≜ N X N PA Y exp (cid:16) − 2 T ′ (cid:0) ϵ TX,PA Y (cid:1) 2 (cid:17) , we have
Revised: Letting δ TX,PA Y ≜ N X N PA Y exp (cid:16) − 2 T ′ (cid:0) ϵ TX,PA Y (cid:1) 2 (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we can get that, for a given pa Y , ∀ x , (let’s call it event E X, pa Y ),
Revised: Now, we can get that, for a given pa Y , ∀ x , (let’s call it event E X, pa Y ),

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Setting
Revised: Setting

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: A. D ERIVING THE FINAL REGRET BOUND
Revised: A. D ERIVING THE FINAL REGRET BOUND

Original: The probability that events E , E TPA C ( ∀ C ∈ C ) and E TX,PA Y are all simultaneously true is
Revised: The probability that events E , E TPA C ( ∀ C ∈ C ) and E TX,PA Y are all simultaneously true is

Original: [Equation]
Revised: [Equation]

Original: Substituting Equations (4) and (5) back into Equation (3)
Revised: Substituting Equations (4) and (5) back into Equation (3)

Original: [Equation]
Revised: [Equation]

Original: we get that, with the above probability [Equation] for a given c tar ,
Revised: we get that, with the above probability, for a given c tar ,

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: = (cid:118)(cid:117)(cid:117)(cid:116)(cid:20) T ′ (cid:21) ln (cid:32) N PA C δ TC | PA C (cid:33) and ϵ TX,PA Y =
Revised: (cid:118)(cid:117)(cid:117)(cid:116)(cid:20) T ′ (cid:21) ln (cid:32) N PA C δ TC | PA C (cid:33) and ϵ TX,PA Y =

Original: (cid:118)(cid:117)(cid:117)(cid:116)(cid:20) T ′ (cid:21) ln
Revised: (cid:118)(cid:117)(cid:117)(cid:116)(cid:20) T ′ (cid:21) ln

Original: (cid:32) N X N PA Y δ TX,PA Y (cid:33)
Revised: (cid:32) N X N PA Y δ TX,PA Y (cid:33)

Original: We set ˜ δ
Revised: We set ˜ δ

Original: = δ X, pa Y =
Revised: = δ X, pa Y =

Original: δ TX,PA Y =
Revised: δ TX,PA Y =

Original: δ C | pa C =
Revised: δ C | pa C =

Original: δ TPA C , ∀ C, pa Y , pa C .
Revised: δ TPA C , ∀ C, pa Y , pa C .

Original: Thus, for a given c tar , with probability ≥ 1 − N PA Y ˜ δ − ˜ δ − N C\ PA Y ˜ δ − |C \ PA Y | ˜ δ ,
Revised: Thus, for a given c tar , with probability ≥ 1 − N PA Y ˜ δ − ˜ δ − N C\ PA Y ˜ δ − |C \ PA Y | ˜ δ ,

Original: Regret ( c tar ) ≤ (cid:88) pa Y
Revised: Regret ( c tar ) ≤ (cid:88) pa Y

Original: P ( pa Y | c tar ) (cid:118)(cid:117)(cid:117)(cid:116)(cid:34)
Revised: P ( pa Y | c tar ) (cid:118)(cid:117)(cid:117)(cid:116)(cid:34)

Original: T ′ N X P ( pa Y , c tar ) − ϵ TX,PA Y (cid:35) ln (cid:18) 2 N X ˜ δ (cid:19) + (cid:88)
Revised: T ′ N X P ( pa Y , c tar ) − ϵ TX,PA Y (cid:35) ln (cid:18) 2 N X ˜ δ (cid:19) + (cid:88)

Original: C ∈C other (cid:88) pa C
Revised: C ∈C other (cid:88) pa C

Original: P ( pa C | c tar ) (cid:118)(cid:117)(cid:117)(cid:116)(cid:34)
Revised: P ( pa C | c tar ) (cid:118)(cid:117)(cid:117)(cid:116)(cid:34)

Original: T ′ P ( pa C , c tar ) − ϵ TC | PA C (cid:35) ln (cid:18)˜ δ (cid:19)
Revised: T ′ P ( pa C , c tar ) − ϵ TC | PA C (cid:35) ln (cid:18)˜ δ (cid:19)

Original: where ϵ TC | PA C = (cid:115) (cid:20) T ′ (cid:21) ln (cid:18) N PA C ˜
Revised: where ϵ TC | PA C = (cid:115) (cid:20) T ′ (cid:21) ln (cid:18) N PA C ˜

Original: δ
Revised: δ

Original: (cid:19) and ϵ TX,PA Y =
Revised: (cid:19) and ϵ TX,PA Y =

Original: (cid:115)(cid:20) T ′ (cid:21) ln (cid:18) N X N PA Y ˜ δ (cid:19)
Revised: (cid:115)(cid:20) T ′ (cid:21) ln (cid:18) N X N PA Y ˜ δ (cid:19)

Original: Now, noting that N PA Y ˜ δ + ˜ δ
Revised: Now, noting that N PA Y ˜ δ + ˜ δ

Original: + N C\ PA Y ˜ δ
Revised: + N C\ PA Y ˜ δ

Original: + |C \ PA Y | ˜ δ ≤ ( N C + |C| )
Revised: + |C \ PA Y | ˜ δ ≤ ( N C + |C| )

Original: ˜ δ , we set δ =
Revised: ˜ δ , we set δ =

Original: ( N C + |C| )
Revised: ( N C + |C| )

Original: ˜ δ .
Revised: ˜ δ .

Original: So we get that with probability ≥ 1 − δ ,
Revised: So we get that with probability ≥ 1 − δ ,

Original: Regret ( c tar ) ≤
Revised: Regret ( c tar ) ≤

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Substituting Equation (6) into the definition of Regret in Section 2 of the main paper, we get the expression in Theorem 3.1.
Revised: Substituting Equation (6) into the definition of Regret in Section 2 of the main paper, we get the expression in Theorem 3.1.

Original: B A PPENDIX (S UPPLEMENTARY T EXT ): A DDITIONAL EXPERIMENT – UNDERSTANDING THE REASON FOR BETTER PERFORMANCE
Revised: B A PPENDIX (S UPPLEMENTARY T EXT ): A DDITIONAL EXPERIMENT – UNDERSTANDING THE REASON FOR BETTER PERFORMANCE

Original: [Figure]
Revised: [Figure]

Original: In this analysis, we try to “ understand why our algorithm exhibits better performance than baselines.”
Revised: In this analysis, we try to “ understand why our algorithm exhibits better performance than baselines.”

Original: We had pointed out intuitively that the Unc measure trades off between exploring new contexts and learning more about explored contexts based on (a) current knowledge of the various context-action pairs, and (b) current estimates of the ‘value’ (i.e., rewards obtainable) of context-action pairs.
Revised: We had pointed out intuitively that the Unc measure trades off between exploring new contexts and learning more about explored contexts based on (a) current knowledge of the various context-action pairs, and (b) current estimates of the ‘value’ (i.e., rewards obtainable) of context-action pairs.

Original: This section attempts to zoom into this behavior.
Revised: This section attempts to zoom into this behavior.

Original: Specifically, we consider one of the settings (setting #1) from the representative set used in Experiment 1; it has N C tar = 10 , all equally likely, but C tar ∈ { 0 , 1 } are more valuable than others (i.e., learning the right actions for these contexts can give higher rewards); see code ( config setting1.py ) for more details.
Revised: Specifically, we consider one of the settings (setting #1) from the representative set used in Experiment 1; it has N C tar = 10 , all equally likely, but C tar ∈ { 0 , 1 } are more valuable than others (i.e., learning the right actions for these contexts can give higher rewards); see code ( config setting1.py ) for more details.

Original: We zoom into the case when T = 15 .
Revised: We zoom into the case when T = 15 .

Original: We doindependent runs, count the number of times every possible C tar value is encountered or chosen, and plot the frequencies.
Revised: We doindependent runs, count the number of times every possible C tar value is encountered or chosen, and plot the frequencies.

Original: We compare our algorithm’s behavior with two representative baselines – Algorithm Std UniExp (which does standard contextual bandit interactions) and Algorithm
Revised: We compare our algorithm’s behavior with two representative baselines – Algorithm Std UniExp (which does standard contextual bandit interactions) and Algorithm

Original: TargInt UniExp (which does targeted interventions); the effect is similar for the other two baselines as well.
Revised: TargInt UniExp (which does targeted interventions); the effect is similar for the other two baselines as well.

Original: As seen in Figure 4, our algorithm chooses the higher value contexts ( C tar ∈ { 0 , 1 } ) with relatively higher frequency than the baselines, while still ensuring good exploration of other contexts – in line with the intuition discussed in the main paper.
Revised: As seen in Figure 4, our algorithm chooses the higher value contexts ( C tar ∈ { 0 , 1 } ) with relatively higher frequency than the baselines, while still ensuring good exploration of other contexts – in line with the intuition discussed in the main paper.

Original: C A PPENDIX (S UPPLEMENTARY T EXT ): M ORE DETAILS ABOUT E XPERIMENT
Revised: C A PPENDIX (S UPPLEMENTARY T EXT ): M ORE DETAILS ABOUT E XPERIMENT

Original: In the discussion about Experiment 1 in the main paper, it was mentioned that Appendix C contains plots for two individual settings from the representative set – one where Algorithms Std TS and Std UniExp perform significantly better than Algorithms TargInt TS , TargInt UniExp and
Revised: In the discussion about Experiment 1 in the main paper, it was mentioned that Appendix C contains plots for two individual settings from the representative set – one where Algorithms Std TS and Std UniExp perform significantly better than Algorithms TargInt TS , TargInt UniExp and

Original: TargInt TS UniExp , and another where it is the converse – while our algorithm performs close to the best baseline in both.
Revised: TargInt TS UniExp , and another where it is the converse – while our algorithm performs close to the best baseline in both.

Original: Figure 5 presents those plots.
Revised: Figure 5 presents those plots.

Original: Each result is based on 50 runs.
Revised: Each result is based on 50 runs.

Original: D A PPENDIX (S UPPLEMENTARY T EXT ): R EGARDING PARAMETERIZATIONS USED IN THE EXPERIMENTS
Revised: D A PPENDIX (S UPPLEMENTARY T EXT ): R EGARDING PARAMETERIZATIONS USED IN THE EXPERIMENTS

Original: For the specific parameterizations of all settings used in all the experiments, refer to the README file in Supplemental Code Paper1203.zip as part of the Supplemental Material.
Revised: For the specific parameterizations of all settings used in all the experiments, refer to the README file in Supplemental Code Paper1203.zip as part of the Supplemental Material.

Original: E A PPENDIX (S UPPLEMENTARY T EXT ): M ORE DISCUSSION ON THE CASE WHERE C other = ∅
Revised: E A PPENDIX (S UPPLEMENTARY T EXT ): M ORE DISCUSSION ON THE CASE WHERE C other = ∅

Original: If C other = ∅ (that is, if it is empty), it might appear as if there would not be any information leakage.
Revised: If C other = ∅ (that is, if it is empty), it might appear as if there would not be any information leakage.

Original: However, this is not true, and information leakage can still exist.
Revised: However, this is not true, and information leakage can still exist.

Original: For instance, consider the same graph as in used in Section 4.1.
Revised: For instance, consider the same graph as in used in Section 4.1.

Original: But now suppose that C tar = { C 1 , C 0 } , whereas C other = ∅ .
Revised: But now suppose that C tar = { C 1 , C 0 } , whereas C other = ∅ .

Original: Consider two targeted interventions, ( x, c 0 , c 1 ) and ( x, c 0 , c ′ 1 ) , where c 1 ̸ = c ′ 1 .
Revised: Consider two targeted interventions, ( x, c 0 , c 1 ) and ( x, c 0 , c ′ 1 ) , where c 1 ̸ = c ′ 1 .

Original: Note that the distribution of Y after these two targeted interventions would, respectively, be
Revised: Note that the distribution of Y after these two targeted interventions would, respectively, be

Original: [Equation]
Revised: [Equation]

Original: The common term P ( Y | x, c 0 ) enables information leakage between these two targeted interventions.
Revised: The common term P ( Y | x, c 0 ) enables information leakage between these two targeted interventions.

Original: To see this, note that if the agent conducts targeted intervention ( x, c 0 , c 1 ) , then it is able to update its beliefs about P ( Y | x, c 0 ) which improves its estimates of ( x, c 0 , c ′ 1 ) as well due to the shared term.
Revised: To see this, note that if the agent conducts targeted intervention ( x, c 0 , c 1 ) , then it is able to update its beliefs about P ( Y | x, c 0 ) which improves its estimates of ( x, c 0 , c ′ 1 ) as well due to the shared term.

Original: So
Revised: So

Original: C other = ∅ does not preclude information leakage.
Revised: C other = ∅ does not preclude information leakage.

Original: Instead, suppose that we are considering the interventions ( x, c 0 , c 1 ) and ( x, c ′ 0 , c 1 ) , where c 0 ̸ = c ′ 0 .
Revised: Instead, suppose that we are considering the interventions ( x, c 0 , c 1 ) and ( x, c ′ 0 , c 1 ) , where c 0 ̸ = c ′ 0 .

Original: In this case, information leakage gains will not be possible.
Revised: In this case, information leakage gains will not be possible.

Original: However, our algorithm would still be able to achieve better performance compared to baselines by balancing between exploring new ( x, c tar ) pairs and learning more about already explored “valuable” ( x, c tar ) pairs.
Revised: However, our algorithm would still be able to achieve better performance compared to baselines by balancing between exploring new ( x, c tar ) pairs and learning more about already explored “valuable” ( x, c tar ) pairs.

Original: To see this, note that in the definition of Unc in Section 3.1 the “information” component is weighted by the “value” component (the last two terms, namely, ˆ P ( c ′ )
Revised: To see this, note that in the definition of Unc in Section 3.1 the “information” component is weighted by the “value” component (the last two terms, namely, ˆ P ( c ′ )

Original: · ˆ E [ Y | c ′ , do ( x ′ )] ).
Revised: · ˆ E [ Y | c ′ , do ( x ′ )] ).

Original: [Figure]
Revised: [Figure]

Original: F A PPENDIX (S UPPLEMENTARY T EXT ): R EGARDING THE CHOICE OF THE PROBABILITY OF UNIFORM EXPLORATION IN T ARG I NT TS U NI E XP
Revised: F A PPENDIX (S UPPLEMENTARY T EXT ): R EGARDING THE CHOICE OF THE PROBABILITY OF UNIFORM EXPLORATION IN T ARG I NT TS U NI E XP

Original: Thompson Sampling aims to minimize cumulative regret in a best-arm identification setting (Agrawal & Goyal, 2012).
Revised: Thompson Sampling aims to minimize cumulative regret in a best-arm identification setting (Agrawal & Goyal, 2012).

Original: For cumulative regret minimization, it performs better than simple exploration algorithms such as ϵ -greedy, by controlling the exploration based on the agent’s current beliefs.
Revised: For cumulative regret minimization, it performs better than simple exploration algorithms such as ϵ -greedy, by controlling the exploration based on the agent’s current beliefs.

Original: However, as discussed in the main part of the paper, in our setting, the agent is faced with the task of learning a policy , while balancing exploration of new contexts and learning more about already explored contexts.
Revised: However, as discussed in the main part of the paper, in our setting, the agent is faced with the task of learning a policy , while balancing exploration of new contexts and learning more about already explored contexts.

Original: Thus, apart from having standard Thompson Sampling as a baseline ( TargInt TS ), we also create a variation of this (called TargInt TS UniExp ) where, in each round, the algorithm, with some probability λ , chooses a targeted intervention uniformly at random (from the space val ( X ) × val ( C tar ) ) for exploration; with probability 1 − λ , it chooses the targeted intervention given by Thompson Sampling.
Revised: Thus, apart from having standard Thompson Sampling as a baseline ( TargInt TS ), we also create a variation of this (called TargInt TS UniExp ) where, in each round, the algorithm, with some probability λ , chooses a targeted intervention uniformly at random (from the space val ( X ) × val ( C tar ) ) for exploration; with probability 1 − λ , it chooses the targeted intervention given by Thompson Sampling.

Original: This probability λ allows us to directly control the degree of uniform exploration.
Revised: This probability λ allows us to directly control the degree of uniform exploration.

Original: In Experiment 1 (Section 4.1), we chose the probability of uniform exploration λ = 0 .
Revised: In Experiment 1 (Section 4.1), we chose the probability of uniform exploration λ = 0 .

Original: 5 for
Revised: 5 for

Original: TargInt TS UniExp .
Revised: TargInt TS UniExp .

Original: This was to act as a midpoint between the TargInt TS algorithm
Revised: This was to act as a midpoint between the TargInt TS algorithm

Original: (attempting to optimize cumulative regret) and TargInt UniExp (which does uniform exploration).
Revised: (attempting to optimize cumulative regret) and TargInt UniExp (which does uniform exploration).

Original: We noted that our algorithm continues to perform better than all baselines.
Revised: We noted that our algorithm continues to perform better than all baselines.

Original: Figure 6 shows that this continues to be true even when λ in TargInt TS UniExp is not as large as 0.5, by setting λ = 0 .
Revised: Figure 6 shows that this continues to be true even when λ in TargInt TS UniExp is not as large as 0.5, by setting λ = 0 .