Original: C AUSAL I MITATION L EARNING VIA I NVERSE R EINFORCEMENT L EARNING
Revised: C AUSAL I MITATION L EARNING VIA I NVERSE R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the most common ways children learn when unfamiliar with the environment is by mimicking adults.
Revised: One of the most common ways children learn when unfamiliar with the environment is by mimicking adults.

Original: Imitation learning concerns an imitator learning to behave in an unknown environment from an expert’s demonstration; reward signals remain latent to the imitator.
Revised: Imitation learning concerns an imitator learning to behave in an unknown environment from an expert’s demonstration; reward signals remain latent to the imitator.

Original: This paper studies imitation learning through causal lenses and extends the analysis and tools developed for behavior cloning (Zhang, Kumor,
Revised: This paper studies imitation learning through causal lenses and extends the analysis and tools developed for behavior cloning (Zhang, Kumor,

Original: Bareinboim, 2020) to inverse reinforcement learning.
Revised: Bareinboim, 2020) to inverse reinforcement learning.

Original: First, we propose novel graphical conditions that allow the imitator to learn a policy performing as well as the expert’s behavior policy, even when the imitator and the expert’s state-action space disagree, and unobserved confounders (UCs) are present.
Revised: First, we propose novel graphical conditions that allow the imitator to learn a policy performing as well as the expert’s behavior policy, even when the imitator and the expert’s state-action space disagree, and unobserved confounders (UCs) are present.

Original: When provided with parametric knowledge about the unknown reward function, such a policy may outperform the expert’s.
Revised: When provided with parametric knowledge about the unknown reward function, such a policy may outperform the expert’s.

Original: Also, our method is easily extensible and allows one to leverage existing IRL algorithms even when UCs are present, including the multiplicative-weights algorithm (MWAL) (Syed & Schapire, 2008) and the generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016).
Revised: Also, our method is easily extensible and allows one to leverage existing IRL algorithms even when UCs are present, including the multiplicative-weights algorithm (MWAL) (Syed & Schapire, 2008) and the generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016).

Original: Finally, we validate our framework by simulations using real-world and synthetic data.
Revised: Finally, we validate our framework by simulations using real-world and synthetic data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning (RL) has been deployed and shown to perform extremely well in highly complex environments in the past decades (Sutton & Barto, 1998; Mnih et al., 2013; Silver et al., 2016; Berner et al., 2019).
Revised: Reinforcement Learning (RL) has been deployed and shown to perform extremely well in highly complex environments in the past decades (Sutton & Barto, 1998; Mnih et al., 2013; Silver et al., 2016; Berner et al., 2019).

Original: One of the critical assumptions behind many of the classical RL algorithms is that the reward signal is fully observed, and the reward function could be well-specified.
Revised: One of the critical assumptions behind many of the classical RL algorithms is that the reward signal is fully observed, and the reward function could be well-specified.

Original: In many real-world applications, however, it might be impractical to design a suitable reward function that evaluates each and every scenario (Randløv & Alstrøm, 1998; Ng et al., 1999).
Revised: In many real-world applications, however, it might be impractical to design a suitable reward function that evaluates each and every scenario (Randløv & Alstrøm, 1998; Ng et al., 1999).

Original: For example, in the context of human driving, it is challenging to design a precise reward function, and experimenting in the environment could be ill-advised; still, watching expert drivers operating is usually feasible.
Revised: For example, in the context of human driving, it is challenging to design a precise reward function, and experimenting in the environment could be ill-advised; still, watching expert drivers operating is usually feasible.

Original: In machine learning, the imitation learning paradigm investigates the problem of how an agent should behave and learn in an environment with an unknown reward function by observing demonstrations from a human expert (Argall et al., 2009; Billard et al., 2008; Hussein et al., 2017; Osa et al., 2018).
Revised: In machine learning, the imitation learning paradigm investigates the problem of how an agent should behave and learn in an environment with an unknown reward function by observing demonstrations from a human expert (Argall et al., 2009; Billard et al., 2008; Hussein et al., 2017; Osa et al., 2018).

Original: There are two major learning modalities that implements IL – behavioral cloning (BC) (Widrow, 1964; Pomerleau, 1989; Muller et al., 2006; M¨ulling et al., 2013; Mahler & Goldberg, 2017) and inverse reinforcement learning (IRL)
Revised: There are two major learning modalities that implements IL – behavioral cloning (BC) (Widrow, 1964; Pomerleau, 1989; Muller et al., 2006; M¨ulling et al., 2013; Mahler & Goldberg, 2017) and inverse reinforcement learning (IRL)

Original: Ng et al.
Revised: Ng et al.

Original: ; Ziebart et al.
Revised: ; Ziebart et al.

Original: ; Ho & Ermon (2016);
Revised: ; Ho & Ermon (2016);

Original: Fu et al.
Revised: Fu et al.

Original: BC methods directly mimic the expert’s behavior policy by learning a mapping from observed states to the expert’s action via supervised learning.
Revised: BC methods directly mimic the expert’s behavior policy by learning a mapping from observed states to the expert’s action via supervised learning.

Original: Alternatively, IRL methods first learn a potential reward function under which the expert’s behavior policy is optimal.
Revised: Alternatively, IRL methods first learn a potential reward function under which the expert’s behavior policy is optimal.

Original: The imitator then obtains a policy by employing standard RL methods to maximize the learned reward function.
Revised: The imitator then obtains a policy by employing standard RL methods to maximize the learned reward function.

Original: Under some common assumptions, both BC and IRL are able to obtain policies that achieve the expert’s performance (Kumor et al., 2021; Swamy et al., 2021).
Revised: Under some common assumptions, both BC and IRL are able to obtain policies that achieve the expert’s performance (Kumor et al., 2021; Swamy et al., 2021).

Original: Moreover, when additional parametric knowledge about the reward function is provided, IRL may produce a policy that outperforms the expert’s in the underlying environment (Syed & Schapire, 2008;
Revised: Moreover, when additional parametric knowledge about the reward function is provided, IRL may produce a policy that outperforms the expert’s in the underlying environment (Syed & Schapire, 2008;

Original: Li et al., 2017; Yu et al., 2020).
Revised: Li et al., 2017; Yu et al., 2020).

Original: For concreteness, consider a learning scenario depicted in Fig.
Revised: For concreteness, consider a learning scenario depicted in Fig.

Original: 1a, describing trajectories of humandriven cars collected by drones flying over highways (Krajewski et al., 2018; Etesami & Geiger, 2020).
Revised: 1a, describing trajectories of humandriven cars collected by drones flying over highways (Krajewski et al., 2018; Etesami & Geiger, 2020).

Original: Using such data, we want to learn a policy X ← π ( Z ) deciding on the acceleration (action)
Revised: Using such data, we want to learn a policy X ← π ( Z ) deciding on the acceleration (action)

Original: X ∈ { 0 , 1 } of the demonstrator car based on velocities and locations Z of surrounding cars.
Revised: X ∈ { 0 , 1 } of the demonstrator car based on velocities and locations Z of surrounding cars.

Original: The driving performance is measured by a latent reward signal Y .
Revised: The driving performance is measured by a latent reward signal Y .

Original: Consider an instance where Y ←
Revised: Consider an instance where Y ←

Original: X
Revised: X

Original: Z
Revised: Z

Original: Y
Revised: Y

Original: U (a)
Revised: U (a)

Original: X
Revised: X

Original: Z
Revised: Z

Original: Y
Revised: Y

Original: U U
Revised: U U

Original: U(b)
Revised: U(b)

Original: Z
Revised: Z

Original: X
Revised: X

Original: Z
Revised: Z

Original: X
Revised: X

Original: Y
Revised: Y

Original: U (c)
Revised: U (c)

Original: Z
Revised: Z

Original: X
Revised: X

Original: Y
Revised: Y

Original: Z
Revised: Z

Original: X
Revised: X

Original: Y
Revised: Y

Original: Z
Revised: Z

Original: X
Revised: X

Original: Y U (d)
Revised: Y U (d)

Original: (1 − X )
Revised: (1 − X )

Original: Z + X
Revised: Z + X

Original: (1 − Z ) and values of Z are drawn uniformly
Revised: (1 − Z ) and values of Z are drawn uniformly

Original: A human expert generates demonstrations following a behavior policy P ( X ) which selects the value of X uniformly at random.
Revised: A human expert generates demonstrations following a behavior policy P ( X ) which selects the value of X uniformly at random.

Original: Evaluating the performance of the expert gives the expected reward E
Revised: Evaluating the performance of the expert gives the expected reward E

Original: [ Y ] = 0 . 5 .
Revised: [ Y ] = 0 . 5 .

Original: Now we apply standard IRL algorithms to learn a policy
Revised: Now we apply standard IRL algorithms to learn a policy

Original: X ← π ( Z ) so that the imitator’s driving performance, denoted by E [ Y | do ( π )] , is at least as good as the expert’s performance E
Revised: X ← π ( Z ) so that the imitator’s driving performance, denoted by E [ Y | do ( π )] , is at least as good as the expert’s performance E

Original: [ Y ] .
Revised: [ Y ] .

Original: Detailed derivations of the IRL policy are shown in Appendix A. Note that E [ Y | z, x ] = x
Revised: Detailed derivations of the IRL policy are shown in Appendix A. Note that E [ Y | z, x ] = x

Original: + z − 2 xz belongs to a family of reward functions
Revised: + z − 2 xz belongs to a family of reward functions

Original: f Y ( x, z ) =
Revised: f Y ( x, z ) =

Original: αx + βz − γxz , where 0 < α < γ .
Revised: αx + βz − γxz , where 0 < α < γ .

Original: A typical IRL imitator solves a minimax problem min π max f Y E [ f Y ( X, Z )]
Revised: A typical IRL imitator solves a minimax problem min π max f Y E [ f Y ( X, Z )]

Original: − E [ f Y ( X, Z ) | do ( π )] .
Revised: − E [ f Y ( X, Z ) | do ( π )] .

Original: The inner step “guesses” a reward function being optimized by the expert; while the outer step learns a policy maximizing the learned reward function.
Revised: The inner step “guesses” a reward function being optimized by the expert; while the outer step learns a policy maximizing the learned reward function.

Original: Applying these steps leads to an imitating policy π ∗ :
Revised: Applying these steps leads to an imitating policy π ∗ :

Original: X ← ¬ Z with the expected reward E [ Y | do ( π ∗ )] = 1 , which outperforms the sub-optimal expert.
Revised: X ← ¬ Z with the expected reward E [ Y | do ( π ∗ )] = 1 , which outperforms the sub-optimal expert.

Original: Despite the performance guarantees provided by existing imitation methods, both BC and IRL rely on the assumption that the expert’s input observations match those available to the imitator.
Revised: Despite the performance guarantees provided by existing imitation methods, both BC and IRL rely on the assumption that the expert’s input observations match those available to the imitator.

Original: More recently, there exists an emerging line of research under the rubric of causal imitation learning that augments the imitation paradigm to account for environments consisting of arbitrary causal mechanisms and the aforementioned mismatch between expert and imitator’s sensory capabilities (de Haan et al., 2019;
Revised: More recently, there exists an emerging line of research under the rubric of causal imitation learning that augments the imitation paradigm to account for environments consisting of arbitrary causal mechanisms and the aforementioned mismatch between expert and imitator’s sensory capabilities (de Haan et al., 2019;

Original: Zhang et al., 2020; Etesami & Geiger, 2020; Kumor et al., 2021).
Revised: Zhang et al., 2020; Etesami & Geiger, 2020; Kumor et al., 2021).

Original: Closest to our work, Zhang et al.
Revised: Closest to our work, Zhang et al.

Original: (2020); Kumor et al.
Revised: (2020); Kumor et al.

Original: (2021) derived graphical criteria that completely characterize when and how BC could lead to successful imitation even when the agents perceive reality differently.
Revised: (2021) derived graphical criteria that completely characterize when and how BC could lead to successful imitation even when the agents perceive reality differently.

Original: Still, it is unclear how to perform IRL-type training if some expert’s observed states remain latent to the imitator, which leads to the presence of unobserved confounding (UCs) in expert’s demonstrations.
Revised: Still, it is unclear how to perform IRL-type training if some expert’s observed states remain latent to the imitator, which leads to the presence of unobserved confounding (UCs) in expert’s demonstrations.

Original: Perhaps surprisingly, naively applying IRL methods when UCs are present does not necessarily lead to satisfactory performance, even when the expert itself behaves optimally.
Revised: Perhaps surprisingly, naively applying IRL methods when UCs are present does not necessarily lead to satisfactory performance, even when the expert itself behaves optimally.

Original: To witness, we now modify the previous highway driving scenario to demonstrate the challenges of
Revised: To witness, we now modify the previous highway driving scenario to demonstrate the challenges of

Original: UCs.
Revised: UCs.

Original: In reality, covariates Z (i.e., velocities and location) are also affected by the car horn U 1 of surrounding vehicles and the wind condition U 2 .
Revised: In reality, covariates Z (i.e., velocities and location) are also affected by the car horn U 1 of surrounding vehicles and the wind condition U 2 .

Original: However, due to the different perspectives of drones (recording from the top), such critical information (i.e, U 1 , U 2 ) is not recorded by the camera and thus remains unobserved.
Revised: However, due to the different perspectives of drones (recording from the top), such critical information (i.e, U 1 , U 2 ) is not recorded by the camera and thus remains unobserved.

Original: Fig.
Revised: Fig.

Original: 1b graphically describes this modified learning setting.
Revised: 1b graphically describes this modified learning setting.

Original: More specifically, consider an instance where Z ← U 1 ⊕ U 2 , Y ← ¬ X ⊕ Z ⊕ U 2 ; ⊕ is the exclusive-or operator; and values of U 1 and U 2 are drawn uniformly over { 0 , 1 } .
Revised: More specifically, consider an instance where Z ← U 1 ⊕ U 2 , Y ← ¬ X ⊕ Z ⊕ U 2 ; ⊕ is the exclusive-or operator; and values of U 1 and U 2 are drawn uniformly over { 0 , 1 } .

Original: An expert driver, being able to hear the car horn U 1 , follows a behavior policy X ← U 1 and achieves the optimal performance E
Revised: An expert driver, being able to hear the car horn U 1 , follows a behavior policy X ← U 1 and achieves the optimal performance E

Original: [ Y ] = 1 .
Revised: [ Y ] = 1 .

Original: Meanwhile, observe that E [ Y | z, x ] = 1 belongs to a family of reward functions f Y ( x, z )
Revised: Meanwhile, observe that E [ Y | z, x ] = 1 belongs to a family of reward functions f Y ( x, z )

Original: =
Revised: =

Original: α (where α > 0 ).
Revised: α (where α > 0 ).

Original: Solving min π max f Y E [ f Y ( X, Z )] − E [ f Y ( X, Z ) | do ( π )] leads to an IRL policy π ∗ with expected reward E [ Y | do ( π ∗ )]
Revised: Solving min π max f Y E [ f Y ( X, Z )] − E [ f Y ( X, Z ) | do ( π )] leads to an IRL policy π ∗ with expected reward E [ Y | do ( π ∗ )]

Original: = 0 .
Revised: = 0 .

Original: 5 , which is far from the expert’s optimal performance E
Revised: 5 , which is far from the expert’s optimal performance E

Original: [ Y ] = 1 .
Revised: [ Y ] = 1 .

Original: After all, a question that naturally arises is, under what conditions an IRL imitator procedure can perform well when UCs are present, and there is a mismatch between the perception of the two agents?
Revised: After all, a question that naturally arises is, under what conditions an IRL imitator procedure can perform well when UCs are present, and there is a mismatch between the perception of the two agents?

Original: In this paper, we answer this question and, more broadly, investigate the challenge of performing IRL through causal lenses.
Revised: In this paper, we answer this question and, more broadly, investigate the challenge of performing IRL through causal lenses.

Original: In particular, our contributions are summarized as follows.
Revised: In particular, our contributions are summarized as follows.

Original: (1) We provide a novel, causal formulation of the inverse reinforcement learning problem.
Revised: (1) We provide a novel, causal formulation of the inverse reinforcement learning problem.

Original: This formulation allows one to formally study and understand the conditions under which an IRL policy is learnable, including in settings where UCs cannot be ruled out a priori.
Revised: This formulation allows one to formally study and understand the conditions under which an IRL policy is learnable, including in settings where UCs cannot be ruled out a priori.

Original: (2) We derive a new graphical condition for deciding whether an imitating policy can be computed from the available data and knowledge, which provides a robust generalization of current IRL algorithms to non-Markovian settings, including GAIL (Ho & Ermon, 2016) and MWAL (Syed & Schapire, 2008).
Revised: (2) We derive a new graphical condition for deciding whether an imitating policy can be computed from the available data and knowledge, which provides a robust generalization of current IRL algorithms to non-Markovian settings, including GAIL (Ho & Ermon, 2016) and MWAL (Syed & Schapire, 2008).

Original: (3) Finally, we move beyond this graphical condition and develop an effective IRL algorithm for structural causal models (Pearl, 2000) with arbitrary causal relationships.
Revised: (3) Finally, we move beyond this graphical condition and develop an effective IRL algorithm for structural causal models (Pearl, 2000) with arbitrary causal relationships.

Original: Due to the space constraints, all proofs are provided in Appendix B. For a more detailed survey on imitation learning and causal inference, we refer readers to Appendix E.
Revised: Due to the space constraints, all proofs are provided in Appendix B. For a more detailed survey on imitation learning and causal inference, we refer readers to Appendix E.

Original: 1. P RELIMINARIES
Revised: 1. P RELIMINARIES

Original: We use capital letters to denote random variables ( X ) and small letters for their values ( x ).
Revised: We use capital letters to denote random variables ( X ) and small letters for their values ( x ).

Original: D X represents the domain of X and P X the space of probability distributions over D X .
Revised: D X represents the domain of X and P X the space of probability distributions over D X .

Original: For a set X , let | X | denote its dimension.
Revised: For a set X , let | X | denote its dimension.

Original: The probability distribution over variables X is denoted by P ( X ) .
Revised: The probability distribution over variables X is denoted by P ( X ) .

Original: Similarly, P ( Y | X ) represents a set of conditional distributions P ( Y | X
Revised: Similarly, P ( Y | X ) represents a set of conditional distributions P ( Y | X

Original: = x ) for all realizations x .
Revised: = x ) for all realizations x .

Original: We use abbreviations P ( x ) for probabilities P ( X = x ) ; so does P ( Y = y | X = x ) =
Revised: We use abbreviations P ( x ) for probabilities P ( X = x ) ; so does P ( Y = y | X = x ) =

Original: P ( y | x ) .
Revised: P ( y | x ) .

Original: Finally, indicator function 1 { Z = z } returns 1 if Z = z holds true; otherwise 0 .
Revised: Finally, indicator function 1 { Z = z } returns 1 if Z = z holds true; otherwise 0 .

Original: The basic semantic framework of our analysis rests on structural causal models (SCMs) (Pearl, 2000, Ch. 7).
Revised: The basic semantic framework of our analysis rests on structural causal models (SCMs) (Pearl, 2000, Ch. 7).

Original: An SCM M is a tuple ⟨ U , V , F , P ( U ) ⟩ with V the set of endogenous, and U exogenous variables.
Revised: An SCM M is a tuple ⟨ U , V , F , P ( U ) ⟩ with V the set of endogenous, and U exogenous variables.

Original: F is a set of structural functions s.t. for f V ∈ F , V ← f V ( pa V , u V ) , with
Revised: F is a set of structural functions s.t. for f V ∈ F , V ← f V ( pa V , u V ) , with

Original: PA V ⊆ V , U V ⊆ U .
Revised: PA V ⊆ V , U V ⊆ U .

Original: Values of U are drawn from an exogenous distribution P ( U ) , inducing distribution P ( V ) over endogenous variables V .
Revised: Values of U are drawn from an exogenous distribution P ( U ) , inducing distribution P ( V ) over endogenous variables V .

Original: Since the learner can observe only a subset of endogenous variables, we split V into a partition O ∪ L where variable O ⊆
Revised: Since the learner can observe only a subset of endogenous variables, we split V into a partition O ∪ L where variable O ⊆

Original: V are observed and
Revised: V are observed and

Original: L = V \ O remain latent to the leaner.
Revised: L = V \ O remain latent to the leaner.

Original: The marginal distribution P ( O ) is thus referred to as the observational distribution .
Revised: The marginal distribution P ( O ) is thus referred to as the observational distribution .

Original: An  intervention on a subset X ⊆ V , denoted by do ( x ) , is an operation where values of X are set to constants x , replacing the functions
Revised: An atomic intervention on a subset X ⊆ V , denoted by do ( x ) , is an operation where values of X are set to constants x , replacing the functions

Original: f X =
Revised: f X =

Original: { f X : ∀ X ∈ X } that would normally determine their values.
Revised: { f X : ∀ X ∈ X } that would normally determine their values.

Original: For an SCM M , let M x be a submodel of M induced by intervention do ( x ) .
Revised: For an SCM M , let M x be a submodel of M induced by intervention do ( x ) .

Original: For a set Y ⊆ V , the interventional distribution P ( s | do ( x )) induced by do ( x ) is defined as the distribution over Y in the submodel M x , i.e., P M ( Y | do ( x ))
Revised: For a set Y ⊆ V , the interventional distribution P ( s | do ( x )) induced by do ( x ) is defined as the distribution over Y in the submodel M x , i.e., P M ( Y | do ( x ))

Original: ≜ P M x ( Y ) .
Revised: ≜ P M x ( Y ) .

Original: We leave
Revised: We leave

Original: M implicit when it is obvious from the context.
Revised: M implicit when it is obvious from the context.

Original: Each SCM M is associated with a causal diagram G which is a directed acyclic graph where (e.g., see Fig. 1) solid nodes represent observed variables O , dashed nodes represent latent variables L , and arrows represent the arguments PA V of each function f V ∈ F .
Revised: Each SCM M is associated with a causal diagram G which is a directed acyclic graph where (e.g., see Fig. 1) solid nodes represent observed variables O , dashed nodes represent latent variables L , and arrows represent the arguments PA V of each function f V ∈ F .

Original: Exogenous variables U are not explicitly shown; a bi-directed arrow between nodes V i and V j indicates the presence of an unobserved confounder (UC) affecting both V i and V j .
Revised: Exogenous variables U are not explicitly shown; a bi-directed arrow between nodes V i and V j indicates the presence of an unobserved confounder (UC) affecting both V i and V j .

Original: We will use family abbreviations to represent graphical relationships such as parents, children, descendants, and ancestors.
Revised: We will use family abbreviations to represent graphical relationships such as parents, children, descendants, and ancestors.

Original: For example, the set of parent nodes of X in G is denoted by pa ( X ) G = ∪ X ∈ X pa ( X ) G ; ch , de and an are similarly defined.
Revised: For example, the set of parent nodes of X in G is denoted by pa ( X ) G = ∪ X ∈ X pa ( X ) G ; ch , de and an are similarly defined.

Original: Capitalized versions Pa , Ch , De , An include the argument as well, e.g. Pa ( X ) G = pa ( X ) G ∪ X .
Revised: Capitalized versions Pa , Ch , De , An include the argument as well, e.g. Pa ( X ) G = pa ( X ) G ∪ X .

Original: For a subset X ⊆ V , the subgraph obtained from G with edges outgoing from X / incoming into X removed is written as G X / G X respectively.
Revised: For a subset X ⊆ V , the subgraph obtained from G with edges outgoing from X / incoming into X removed is written as G X / G X respectively.

Original: G [ X ] is a subgraph of G containing only nodes X and edges among them.
Revised: G [ X ] is a subgraph of G containing only nodes X and edges among them.

Original: A path from a node X to a node Y in G is a sequence of edges, which does not include a particular node more than once.
Revised: A path from a node X to a node Y in G is a sequence of edges, which does not include a particular node more than once.

Original: Two sets of nodes X , Y are said to be d-separated by a third set Z in a DAG G , denoted by ( X ⊥⊥ Y | Z ) G , if every edge path from nodes in X to nodes in
Revised: Two sets of nodes X , Y are said to be d-separated by a third set Z in a DAG G , denoted by ( X ⊥⊥ Y | Z ) G , if every edge path from nodes in X to nodes in

Original: Y is “blocked” by nodes in Z .
Revised: Y is “blocked” by nodes in Z .

Original: The criterion of blockage follows (Pearl, 2000, Def. 1.2.3).
Revised: The criterion of blockage follows (Pearl, 2000, Def. 1.2.3).

Original: For a more detailed survey on SCMs, we refer readers to (Pearl, 2000; Bareinboim et al., 2022).
Revised: For a more detailed survey on SCMs, we refer readers to (Pearl, 2000; Bareinboim et al., 2022).

Original:  C AUSAL I NVERSE R EINFORCEMENT L EARNING
Revised:  C AUSAL I NVERSE R EINFORCEMENT L EARNING

Original: We investigate the sequential decision-making setting concerning a set of actions X , a series of covariates Z , and a latent reward Y in an SCM M .
Revised: We investigate the sequential decision-making setting concerning a set of actions X , a series of covariates Z , and a latent reward Y in an SCM M .

Original: An expert (e.g., a physician, driver), operating in
Revised: An expert (e.g., a physician, driver), operating in

Original: SCM M , selects actions following a behavior policy , which is the collection of structural functions
Revised: SCM M , selects actions following a behavior policy , which is the collection of structural functions

Original: f X = { f X | X ∈ X } .
Revised: f X = { f X | X ∈ X } .

Original: The expert’s performance is evaluated as the expected reward E [ Y ] .
Revised: The expert’s performance is evaluated as the expected reward E [ Y ] .

Original: On the other hand, a learning agent (i.e., the imitator) intervenes on actions X following an ordering
Revised: On the other hand, a learning agent (i.e., the imitator) intervenes on actions X following an ordering

Original: X 1 ≺ · · · ≺ X n ; each action X
Revised: X 1 ≺ · · · ≺ X n ; each action X

Original: i is associated with a set of features PA ∗ i ⊆
Revised: i is associated with a set of features PA ∗ i ⊆

Original: O \ { X i } .
Revised: O \ { X i } .

Original: A policy π over actions X is a sequence of decision rules π
Revised: A policy π over actions X is a sequence of decision rules π

Original: = { π 1 , . .
Revised: = { π 1 , . .

Original: π n } .
Revised: π n } .

Original: Each decision rule π i ( X i | Z i ) is a probability distribution over an action X i ∈ X , conditioning on values of a set of covariates
Revised: Each decision rule π i ( X i | Z i ) is a probability distribution over an action X i ∈ X , conditioning on values of a set of covariates

Original: Z i ⊆ PA ∗ i .
Revised: Z i ⊆ PA ∗ i .

Original: The collection of such policies π ∼ S are also referred to as dynamic treatment regimes (Murphy et al., 2001; Chakraborty & Murphy, 2014), which generalize personalized medicine to time-varying treatment settings in healthcare, in which treatment is repeatedly tailored to a patient’s dynamic state.
Revised: The collection of such policies π ∼ S are also referred to as dynamic treatment regimes (Murphy et al., 2001; Chakraborty & Murphy, 2014), which generalize personalized medicine to time-varying treatment settings in healthcare, in which treatment is repeatedly tailored to a patient’s dynamic state.

Original: The state-action domain of a policy π is characterized using a policy scope S .
Revised: The state-action domain of a policy π is characterized using a policy scope S .

Original: Definition 1 (Lee & Bareinboim (2020)) .
Revised: Definition 1 (Lee & Bareinboim (2020)) .

Original: For an SCM M , a policy scope S (for short, scope) over actions X is a sequence of tuples {⟨ X i , Z i ⟩} ni =1 where Z i ⊆ PA ∗ i for every X i ∈ X .
Revised: For an SCM M , a policy scope S (for short, scope) over actions X is a sequence of tuples {⟨ X i , Z i ⟩} ni =1 where Z i ⊆ PA ∗ i for every X i ∈ X .

Original: We will consistently use π ∼ S to denote a policy π associated with a scope S .
Revised: We will consistently use π ∼ S to denote a policy π associated with a scope S .

Original: For example, consider a policy scope S
Revised: For example, consider a policy scope S

Original: = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} over actions X 1 , X 2 in Fig.
Revised: = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} over actions X 1 , X 2 in Fig.

Original: 1c.
Revised: 1c.

Original: A policy π ∼ S is a sequence of distributions
Revised: A policy π ∼ S is a sequence of distributions

Original: π = { π 1 ( X 1 | Z 1 ) , π 2 ( X 2 | Z 2 ) } .
Revised: π = { π 1 ( X 1 | Z 1 ) , π 2 ( X 2 | Z 2 ) } .

Original: An intervention on actions X following a policy π , denoted by do ( π ) , entails a submodel M π from a SCM M where structural functions f X associated with X (i.e., the expert’s behavior policy) are replaced with decision rules
Revised: A policy intervention on actions X following a policy π , denoted by do ( π ) , entails a submodel M π from a SCM M where structural functions f X associated with X (i.e., the expert’s behavior policy) are replaced with decision rules

Original: X i ∼ π
Revised: X i ∼ π

Original: i
Revised: i

Original: X i | Z i ) for every X i ∈
Revised: X i | Z i ) for every X i ∈

Original: X .
Revised: X .

Original: A critical assumption throughout this paper is that submodel M π does not contain any cycles.
Revised: A critical assumption throughout this paper is that submodel M π does not contain any cycles.

Original: That is, the causal diagram G S associated with M π is a directed acyclic graph.
Revised: That is, the causal diagram G S associated with M π is a directed acyclic graph.

Original: Similarly, the interventional distribution P ( V | do ( π )) induced by policy π is defined as the joint distribution over variables V in M π .
Revised: Similarly, the interventional distribution P ( V | do ( π )) induced by policy π is defined as the joint distribution over variables V in M π .

Original: The goal of the agent is to learn an imitating policy π ∗ that achieves the expert’s performance,
Revised: The goal of the agent is to learn an imitating policy π ∗ that achieves the expert’s performance,

Original: [Equation]
Revised: [Equation]

Original: In words, the right hand side is the expert’s performance that the agent wants to achieve, while the left hand side is the real reward experienced by the agent.
Revised: In words, the right hand side is the expert’s performance that the agent wants to achieve, while the left hand side is the real reward experienced by the agent.

Original: Throughout this paper, detailed parametrizations of the underlying SCM M are assumed to be unknown to the agent.
Revised: Throughout this paper, detailed parametrizations of the underlying SCM M are assumed to be unknown to the agent.

Original: Instead, the agent has access to a causal diagram G associated with M , and observed trajectories of the expert’s demonstrations, summarized as the observational distribution P ( O ) .
Revised: Instead, the agent has access to a causal diagram G associated with M , and observed trajectories of the expert’s demonstrations, summarized as the observational distribution P ( O ) .

Original: The challenge in imitation learning settings arises from the fact that the reward Y is not specified and latent, i.e., Y ̸∈ O .
Revised: The challenge in imitation learning settings arises from the fact that the reward Y is not specified and latent, i.e., Y ̸∈ O .

Original: This precludes approaches that attempt to identify E [ Y | do ( π )] directly from the demonstration data (e.g., through the do- or soft-do-calculus Pearl (2000); Correa & Bareinboim (2020)).
Revised: This precludes approaches that attempt to identify E [ Y | do ( π )] directly from the demonstration data (e.g., through the do- or soft-do-calculus Pearl (2000); Correa & Bareinboim (2020)).

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020); Kumor et al.
Revised: (2020); Kumor et al.

Original: (2021) provide a graphical condition that is sufficient for learning an imitating policy via behavioral cloning (BC) provided with a causal diagram G .
Revised: (2021) provide a graphical condition that is sufficient for learning an imitating policy via behavioral cloning (BC) provided with a causal diagram G .

Original: For a policy scope S = {⟨ X i , Z i ⟩} ni =1 , let G ( i ) , i = 1 , . . .
Revised: For a policy scope S = {⟨ X i , Z i ⟩} ni =1 , let G ( i ) , i = 1 , . . .

Original: , n , denote a manipulated graph obtained from G by the following steps: for all j = i + 1 , . .
Revised: , n , denote a manipulated graph obtained from G by the following steps: for all j = i + 1 , . .

Original: . , n , (1) remove arrows coming into every action X j ; and (2) add direct arrows from nodes in Z j to X j .
Revised: . , n , (1) remove arrows coming into every action X j ; and (2) add direct arrows from nodes in Z j to X j .

Original: Formally, the sequential π -backdoor criterion is defined as:
Revised: Formally, the sequential π -backdoor criterion is defined as:

Original: Definition 2 (Kumor et al.
Revised: Definition 2 (Kumor et al.

Original: Given a causal diagram G , a policy scope S = {⟨ X i , Z i ⟩} ni =1 is said to satisfy the sequential π -backdoor criterion in G (for short, π -backdoor admissible) if at each
Revised: Given a causal diagram G , a policy scope S = {⟨ X i , Z i ⟩} ni =1 is said to satisfy the sequential π -backdoor criterion in G (for short, π -backdoor admissible) if at each

Original: X i ∈ X , one of the following conditions hold:
Revised: X i ∈ X , one of the following conditions hold:

Original: X i is not an ancestor of Y in G ( i ) , i.e., X ̸∈ An ( Y ) G ( i ) .
Revised: X i is not an ancestor of Y in G ( i ) , i.e., X ̸∈ An ( Y ) G ( i ) .

Original: Z i blocks all backdoor path from X i to Y in G ( i ) , i.e., ( Y ⊥⊥ X i | Z i ) in G ( i ) X i .
Revised: Z i blocks all backdoor path from X i to Y in G ( i ) , i.e., ( Y ⊥⊥ X i | Z i ) in G ( i ) X i .

Original: (Kumor et al., 2021) showed that whenever a π -backdoor admissible scope S is available, one could learn an imitating policy π ∗ ∼ S by setting π ∗ i ( x i | z i ) =
Revised: (Kumor et al. , 2021) showed that whenever a π -backdoor admissible scope S is available, one could learn an imitating policy π ∗ ∼ S by setting π ∗ i ( x i | z i ) =

Original: P ( x i |
Revised: P ( x i |

Original: z i ) for every action X i ∈ X .
Revised: z i ) for every action X i ∈ X .

Original: The conditional distribution P ( x i |
Revised: The conditional distribution P ( x i |

Original: z i ) is obtainable via standard BC algorithms.
Revised: z i ) is obtainable via standard BC algorithms.

Original: For instance, consider the causal diagram G in Fig. 1c.
Revised: For instance, consider the causal diagram G in Fig. 1c.

Original: Scope S
Revised: Scope S

Original: = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} is π -backdoor admissible since ( X 1 ⊥⊥ Y | Z 1 ) and ( X 2 ⊥⊥ Y | Z 2 ) hold in G , which is a super graph containing both manipulated diagrams G (1) and G (2) .
Revised: = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} is π -backdoor admissible since ( X 1 ⊥⊥ Y | Z 1 ) and ( X 2 ⊥⊥ Y | Z 2 ) hold in G , which is a super graph containing both manipulated diagrams G (1) and G (2) .

Original: An imitating policy π ∗ = { π ∗ 1 , π ∗ 2 } is thus obtainable by setting π ∗ 1 ( X 1 | Z 1 )
Revised: An imitating policy π ∗ = { π ∗ 1 , π ∗ 2 } is thus obtainable by setting π ∗ 1 ( X 1 | Z 1 )

Original: =
Revised: =

Original: P ( X 1 | Z 1 ) and π ∗ 2 ( X 2 | Z 2 ) =
Revised: P ( X 1 | Z 1 ) and π ∗ 2 ( X 2 | Z 2 ) =

Original: P ( X 2 | Z 2 ) .
Revised: P ( X 2 | Z 2 ) .

Original: While impressive, a caveat of their results is that the performance of the imitator is restricted by that of the expert, i.e., E [ Y | do ( π ∗ )]
Revised: While impressive, a caveat of their results is that the performance of the imitator is restricted by that of the expert, i.e., E [ Y | do ( π ∗ )]

Original: =
Revised: =

Original: E [ Y ] .
Revised: E [ Y ] .

Original: In other words, causal BC provides an efficient way to mimic the expert’s performance.
Revised: In other words, causal BC provides an efficient way to mimic the expert’s performance.

Original: If the expert’s behavior is far from optimal, the same will hold for the learning agent.
Revised: If the expert’s behavior is far from optimal, the same will hold for the learning agent.

Original: 2. M INIMAL S EQUENTIAL B ACKDOOR C RITERION
Revised: 2. M INIMAL S EQUENTIAL B ACKDOOR C RITERION

Original: To circumvent this issue, we take a somewhat different approach to causal imitation by incorporating the principle of inverse reinforcement learning (IRL).
Revised: To circumvent this issue, we take a somewhat different approach to causal imitation by incorporating the principle of inverse reinforcement learning (IRL).

Original: Following the game-theoretic approach (Syed & Schapire, 2008), we formulate the problem as learning to play a two-player zero-sum game in which the agent chooses a policy, and the nature chooses an SCM instance.
Revised: Following the game-theoretic approach (Syed & Schapire, 2008), we formulate the problem as learning to play a two-player zero-sum game in which the agent chooses a policy, and the nature chooses an SCM instance.

Original: A key property of this algorithm is that it allows us to incorporate prior parametric knowledge about the latent reward signal.
Revised: A key property of this algorithm is that it allows us to incorporate prior parametric knowledge about the latent reward signal.

Original: When such knowledge is informative, our algorithm is about to obtain a policy that could significantly outperform the expert with respect to the unknown causal environment, while at the same time are guaranteed to be no worse.
Revised: When such knowledge is informative, our algorithm is about to obtain a policy that could significantly outperform the expert with respect to the unknown causal environment, while at the same time are guaranteed to be no worse.

Original: Formally, let M = {∀ M | G M = G , P M ( O )
Revised: Formally, let M = {∀ M | G M = G , P M ( O )

Original: = P ( O ) } denote the set of SCMs compatible with both the causal diagram G and the observational distribution P ( O ) .
Revised: = P ( O ) } denote the set of SCMs compatible with both the causal diagram G and the observational distribution P ( O ) .

Original: Now consider the optimization problem defined as follows.
Revised: Now consider the optimization problem defined as follows.

Original: For a policy scope S ,
Revised: For a policy scope S ,

Original: [Equation]
Revised: [Equation]

Original: The inner maximization in the above equation can be viewed as an causal IRL step where we attempt to “guess” a worst-case SCM ˆ M compatible with G and P ( O ) that prioritizes the expert’s policy.
Revised: The inner maximization in the above equation can be viewed as an causal IRL step where we attempt to “guess” a worst-case SCM ˆ M compatible with G and P ( O ) that prioritizes the expert’s policy.

Original: That is, the gap in the performance between the expert’s and the imitator’s policies is maximized.
Revised: That is, the gap in the performance between the expert’s and the imitator’s policies is maximized.

Original: Meanwhile, since the expert’s reward E M [ Y ] is not affected by the imitator’s policy π , the outer minimization is equivalent to a planning step that finds a policy π ∗ optimizing the learned SCM ˆ M .
Revised: Meanwhile, since the expert’s reward E M [ Y ] is not affected by the imitator’s policy π , the outer minimization is equivalent to a planning step that finds a policy π ∗ optimizing the learned SCM ˆ M .

Original: Obviously, the solution π ∗ is an imitating policy if gap ν ∗ = 0 .
Revised: Obviously, the solution π ∗ is an imitating policy if gap ν ∗ = 0 .

Original: In cases where the expert is sub-optimal, i.e., E ˆ M [ Y ] <
Revised: In cases where the expert is sub-optimal, i.e., E ˆ M [ Y ] <

Original: E ˆ M [ Y | do ( π )] for some policies π , we may have ν ∗ < 0 .
Revised: E ˆ M [ Y | do ( π )] for some policies π , we may have ν ∗ < 0 .

Original: That is, the policy π ∗ will dominate the expert’s policy f X regardless of parametrizations of SCM M in the worst-case scenario.
Revised: That is, the policy π ∗ will dominate the expert’s policy f X regardless of parametrizations of SCM M in the worst-case scenario.

Original: In other words, π ∗ to some extent ignores the sub-optimal expert, and instead exploits prior knowledge about the underlying model.
Revised: In other words, π ∗ to some extent ignores the sub-optimal expert, and instead exploits prior knowledge about the underlying model.

Original: Despite the clear semantics in terms of causal models, the optimization problem in Eq.
Revised: Despite the clear semantics in terms of causal models, the optimization problem in Eq.

Original: (2) requires the learner to search over all possible SCMs compatible with the causal diagram G and observational distribution P ( O ) .
Revised: (2) requires the learner to search over all possible SCMs compatible with the causal diagram G and observational distribution P ( O ) .

Original: In principle, it entails a quite challenging search since one does not have access to the parametric forms of the underlying structural functions F nor the exogenous distribution P ( U ) .
Revised: In principle, it entails a quite challenging search since one does not have access to the parametric forms of the underlying structural functions F nor the exogenous distribution P ( U ) .

Original: It is not clear how the existing optimization procedures can be used.
Revised: It is not clear how the existing optimization procedures can be used.

Original: In this paper, we will develop novel methods to circumvent this issue, thus leading to effective imitating policies.
Revised: In this paper, we will develop novel methods to circumvent this issue, thus leading to effective imitating policies.

Original: Our first algorithm relies on a refinement of the sequential π -backdoor, based on the concept of minimality.
Revised: Our first algorithm relies on a refinement of the sequential π -backdoor, based on the concept of minimality.

Original: A subscope S ′ of a policy scope S = {⟨ X i , Z i ⟩} ni =1 , denoted by S ′ ⊆ S , is a sequence {⟨ X i , Z ′ i ⟩} ni =1 where Z ′ i ⊆
Revised: A subscope S ′ of a policy scope S = {⟨ X i , Z i ⟩} ni =1 , denoted by S ′ ⊆ S , is a sequence {⟨ X i , Z ′ i ⟩} ni =1 where Z ′ i ⊆

Original: Z i for every X i ∈
Revised: Z i for every X i ∈

Original: X .
Revised: X .

Original: A proper subscope S ′ ⊂ S is a subscope in S other than S itself.
Revised: A proper subscope S ′ ⊂ S is a subscope in S other than S itself.

Original: The minimal π -backdoor admissible scope is defined as follows.
Revised: The minimal π -backdoor admissible scope is defined as follows.

Original: Definition 3.
Revised: Definition 3.

Original: Given a causal diagram G , a π -backdoor admissible scope S is said to be minimal if there exists no proper subscope S ′ ⊂
Revised: Given a causal diagram G , a π -backdoor admissible scope S is said to be minimal if there exists no proper subscope S ′ ⊂

Original: S satisfying the sequential π -backdoor in G .
Revised: S satisfying the sequential π -backdoor in G .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a causal diagram G , if there exists a minimal π -backdoor admissible scope
Revised: Given a causal diagram G , if there exists a minimal π -backdoor admissible scope

Original: S = {⟨ X i , Z i ⟩} ni =1 in G , consider the following conditions:
Revised: S = {⟨ X i , Z i ⟩} ni =1 in G , consider the following conditions:

Original: Let effective actions X ∗ =
Revised: Let effective actions X ∗ =

Original: X ∩ An ( Y ) G S and effective covariates Z ∗ =
Revised: X ∩ An ( Y ) G S and effective covariates Z ∗ =

Original: (cid:83) X i ∈
Revised: (cid:83) X i ∈

Original: X ∗ Z i ; 2.
Revised: X ∗ Z i ; 2.

Original: For i = 1 , . . . , n + 1 , let X ∗ <
Revised: For i = 1 , . . . , n + 1 , let X ∗ <

Original: i = {∀ X j ∈ X ∗ | j < i } and Z ∗ <
Revised: i = {∀ X j ∈ X ∗ | j < i } and Z ∗ <

Original: i =
Revised: i =

Original: (cid:83) X j ∈ X ∗ <
Revised: (cid:83) X j ∈ X ∗ <

Original: i Z j .
Revised: i Z j .

Original: Then, for any policy π ∼ S , the expected reward E [ Y | do ( π )] is computable from P ( O , Y ) as:
Revised: Then, for any policy π ∼ S , the expected reward E [ Y | do ( π )] is computable from P ( O , Y ) as:

Original: [Equation]
Revised: [Equation]

Original: where the occupancy measure ρ π ( x ∗ , z ∗
Revised: where the occupancy measure ρ π ( x ∗ , z ∗

Original: [Equation]
Revised: [Equation]

Original: (cid:81)
Revised: (cid:81)

Original: [Equation]
Revised: [Equation]

Original: To illustrate, consider again the causal diagram G in Fig.
Revised: To illustrate, consider again the causal diagram G in Fig.

Original: 1c; the manipulated diagram G (2) =
Revised: 1c; the manipulated diagram G (2) =

Original: G and G (1) is obtained from G by removing Z 2 ↔ X 2 .
Revised: G and G (1) is obtained from G by removing Z 2 ↔ X 2 .

Original: While scope S 1 = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} satisfies the sequential π -backdoor, it is not minimal since ( X 1 ⊥⊥ Y ) in G (1) X 1 .
Revised: While scope S 1 = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} satisfies the sequential π -backdoor, it is not minimal since ( X 1 ⊥⊥ Y ) in G (1) X 1 .

Original: On the other hand,
Revised: On the other hand,

Original: S 2 = {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} is minimal π -backdoor admissible since ( X 2 ⊥⊥ Y | Z 2 ) holds true in
Revised: S 2 = {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} is minimal π -backdoor admissible since ( X 2 ⊥⊥ Y | Z 2 ) holds true in

Original: G (2) X 2 ; and the covariate set { Z 2 } is minimal due to the presence of the backdoor path X 2 ← Z 2 → Y .
Revised: G (2) X 2 ; and the covariate set { Z 2 } is minimal due to the presence of the backdoor path X 2 ← Z 2 → Y .

Original: Let us focus on the minimal π -backdoor admissible scope S 2 .
Revised: Let us focus on the minimal π -backdoor admissible scope S 2 .

Original: Note that G S 2 is a subgraph obtained from G by removing the bi-directed arrow Z 2 ↔ X 2 .
Revised: Note that G S 2 is a subgraph obtained from G by removing the bi-directed arrow Z 2 ↔ X 2 .

Original: We must have effective actions X ∗ =
Revised: We must have effective actions X ∗ =

Original: { X 1 , X 2 } and effective covariates Z ∗ =
Revised: { X 1 , X 2 } and effective covariates Z ∗ =

Original: { Z 2 } .
Revised: { Z 2 } .

Original: Therefore, Z ∗ < 1 =
Revised: Therefore, Z ∗ < 1 =

Original: Z ∗ < 2 = ∅ and Z ∗ < 3 =
Revised: Z ∗ < 2 = ∅ and Z ∗ < 3 =

Original: { Z 2 } .
Revised: { Z 2 } .

Original: For any policy π ∼ S 2 , Thm. 1 implies E [ Y | do ( π )]
Revised: For any policy π ∼ S 2 , Thm. 1 implies E [ Y | do ( π )]

Original: = (cid:80) x 1 ,x 2 ,z 2 E [ Y | x 1 , x 2 , z 2 ] P ( z 2 | x 1 ) π 2 ( x 2 | z 2 ) π ( x 1 ) .
Revised: = (cid:80) x 1 ,x 2 ,z 2 E [ Y | x 1 , x 2 , z 2 ] P ( z 2 | x 1 ) π 2 ( x 2 | z 2 ) π ( x 1 ) .

Original: On the other hand, the same result in Thm.
Revised: On the other hand, the same result in Thm.

Original: 1 does not necessarily hold for a non-minimal π -backdoor admissible scope.
Revised: 1 does not necessarily hold for a non-minimal π -backdoor admissible scope.

Original: For instance, consider again the non-minimal scope S 1 = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} .
Revised: For instance, consider again the non-minimal scope S 1 = {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} .

Original: The expected reward E [ Y | do ( π )] of a policy π ∼ S 2 is not computable from Eq.
Revised: The expected reward E [ Y | do ( π )] of a policy π ∼ S 2 is not computable from Eq.

Original: (3), and is ultimately not identifiable from distribution P ( O , Y ) in G (Tian, 2008).
Revised: (3), and is ultimately not identifiable from distribution P ( O , Y ) in G (Tian, 2008).

Original: 2. I MITATION VIA I NVERSE R EINFORCEMENT L EARNING
Revised: 2. I MITATION VIA I NVERSE R EINFORCEMENT L EARNING

Original: Once a minimal π -backdoor admissible scope S is found, there exist effective procedures to solve for an imitating policy in Eq.
Revised: Once a minimal π -backdoor admissible scope S is found, there exist effective procedures to solve for an imitating policy in Eq.

Original: Let R be a hypothesis class containing all expected rewards E M [ Y | x ∗ , z ∗ ] compatible with candidate SCMs M ∈ M , i.e., R = { E M [ Y | x ∗ , z ∗ ] | ∀ M ∈ M } .
Revised: Let R be a hypothesis class containing all expected rewards E M [ Y | x ∗ , z ∗ ] compatible with candidate SCMs M ∈ M , i.e., R = { E M [ Y | x ∗ , z ∗ ] | ∀ M ∈ M } .

Original: Applying the identification formula in Thm.
Revised: Applying the identification formula in Thm.

Original: 1 reduces the optimization problem in Eq.
Revised: 1 reduces the optimization problem in Eq.

Original: (2) as follows:
Revised: (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: where the expert’s occupancy measure ρ ( x ∗ , z ∗ ) =
Revised: where the expert’s occupancy measure ρ ( x ∗ , z ∗ ) =

Original: P ( x ∗ , z ∗ ) and the agent’s occupancy measure ρ π ( x ∗ , z ∗ ) is given by Eq.
Revised: P ( x ∗ , z ∗ ) and the agent’s occupancy measure ρ π ( x ∗ , z ∗ ) is given by Eq.

Original: The above minimax problem is solvable using standard IRL algorithms.
Revised: The above minimax problem is solvable using standard IRL algorithms.

Original: The identification result in Thm.
Revised: The identification result in Thm.

Original: 1 ensures that the learned policy applies to any SCM compatible with the causal diagram and the observational data, thus robust to the unobserved confounding bias in the expert’s demonstrations.
Revised: 1 ensures that the learned policy applies to any SCM compatible with the causal diagram and the observational data, thus robust to the unobserved confounding bias in the expert’s demonstrations.

Original: Henceforth, we will consistently refer to Eq.
Revised: Henceforth, we will consistently refer to Eq.

Original: (4) as the canonical equation of causal IRL .
Revised: (4) as the canonical equation of causal IRL .

Original: The remainder of this section demonstrates this computation procedure using some of the state-of-the-art IRL algorithms with common choices of parametric reward functions.
Revised: The remainder of this section demonstrates this computation procedure using some of the state-of-the-art IRL algorithms with common choices of parametric reward functions.

Original: We refer readers to Appendix C for more discussion on the pseudo-code and implementation details.
Revised: We refer readers to Appendix C for more discussion on the pseudo-code and implementation details.

Original: Causal MWAL (Abbeel & Ng, 2004; Syed & Schapire, 2008) study IRL in Markov decision processes where the reward function r ( x ∗ , z ∗ ) is a linear combination of k -length feature expectations vectors
Revised: Causal MWAL (Abbeel & Ng, 2004; Syed & Schapire, 2008) study IRL in Markov decision processes where the reward function r ( x ∗ , z ∗ ) is a linear combination of k -length feature expectations vectors

Original: ϕ ( x ∗ , z ∗ ) .
Revised: ϕ ( x ∗ , z ∗ ) .

Original: Particularly, let r ( x ∗ , z ∗ ) =
Revised: Particularly, let r ( x ∗ , z ∗ ) =

Original: w · ϕ ( x ∗ , z ∗ ) for a coefficient vector w contained in a convex set S k
Revised: w · ϕ ( x ∗ , z ∗ ) for a coefficient vector w contained in a convex set S k

Original: = (cid:8) w ∈ R k | ∥ w ∥ 1 = 1 and w ⪰ 0 (cid:9) .
Revised: = (cid:8) w ∈ R k | ∥ w ∥ 1 = 1 and w ⪰ 0 (cid:9) .

Original: Let ϕ ( i ) be the i -th component of feature vector ϕ and let deterministic policies with scope S be ordered by π (1) , . .
Revised: Let ϕ ( i ) be the i -th component of feature vector ϕ and let deterministic policies with scope S be ordered by π (1) , . .

Original: π ( n ) .
Revised: π ( n ) .

Original: The canonical equation in Eq.
Revised: The canonical equation in Eq.

Original: (4) is reducible to a two-person zero-sum matrix game under the linearity.
Revised: (4) is reducible to a two-person zero-sum matrix game under the linearity.

Original: Proposition 1.
Revised: Proposition 1.

Original: For a hypothesis class R = { r = w · ϕ | w ∈ S k } , the solution ν ∗ of the canonical equation in Eq.
Revised: For a hypothesis class R = { r = w · ϕ | w ∈ S k } , the solution ν ∗ of the canonical equation in Eq.

Original: (4) is obtainable by solving the following minimax problem:
Revised: (4) is obtainable by solving the following minimax problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where G is a k × n matrix given by G ( i, j ) = (cid:80) x ∗ , z ∗ ϕ (
Revised: ) where G is a k × n matrix given by G ( i, j ) = (cid:80) x ∗ , z ∗ ϕ (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: There exist effective multiplicative weights algorithms for solving the matrix game in Eq.
Revised: There exist effective multiplicative weights algorithms for solving the matrix game in Eq.

Original: (5), including MW (Freund & Schapire, 1999) and MWAL (Syed & Schapire, 2008).
Revised: (5), including MW (Freund & Schapire, 1999) and MWAL (Syed & Schapire, 2008).

Original: Causal GAIL (Ho & Ermon, 2016) introduces the GAIL algorithm for learning an imitating policy in Markov decision processes with a general family of non-linear reward functions.
Revised: Causal GAIL (Ho & Ermon, 2016) introduces the GAIL algorithm for learning an imitating policy in Markov decision processes with a general family of non-linear reward functions.

Original: In particular, r ( x ∗ , z ∗ ) takes values in the real space R , i.e., r ∈ R X ∗ , Z ∗ where R X ∗ , Z ∗ =
Revised: In particular, r ( x ∗ , z ∗ ) takes values in the real space R , i.e., r ∈ R X ∗ , Z ∗ where R X ∗ , Z ∗ =

Original: { r : D X ∗ × D Z ∗ (cid:55)→ R } .
Revised: { r : D X ∗ × D Z ∗ (cid:55)→ R } .

Original: The complexity of reward function r is penalized by a convex regularization function ψ ( r ) , i.e.,
Revised: The complexity of reward function r is penalized by a convex regularization function ψ ( r ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Henceforth, we will consistently refer to Eq.
Revised: Henceforth, we will consistently refer to Eq.

Original: (6) as the penalized canonical equation of causal IRL.
Revised: (6) as the penalized canonical equation of causal IRL.

Original: It is often preferable to solve its conjugate form.
Revised: It is often preferable to solve its conjugate form.

Original: Formally,
Revised: Formally,

Original: Proposition 2.
Revised: Proposition 2.

Original: For a hypothesis class R = { r : D X ∗ ×
Revised: For a hypothesis class R = { r : D X ∗ ×

Original: D Z ∗ (cid:55)→ R } regularized by ψ , the solution ν ∗ of the penalized canonical equation in Eq.
Revised: D Z ∗ (cid:55)→ R } regularized by ψ , the solution ν ∗ of the penalized canonical equation in Eq.

Original: (6) is obtainable by solving the following problem:
Revised: (6) is obtainable by solving the following problem:

Original: [Equation]
Revised: [Equation]

Original: (7) where ψ ∗ be a conjugate function of ψ and is given by ψ ∗ = max r ∈ R X
Revised: (7) where ψ ∗ be a conjugate function of ψ and is given by ψ ∗ = max r ∈ R X

Original: × Z a ⊤ r − ψ ( r ) .
Revised: × Z a ⊤ r − ψ ( r ) .

Original: Eq.
Revised: Eq.

Original: (7) seeks a policy π which minimizes the divergence of the occupancy measures between the imitator and the expert, as measured by the function ψ ∗ .
Revised: (7) seeks a policy π which minimizes the divergence of the occupancy measures between the imitator and the expert, as measured by the function ψ ∗ .

Original: The computational framework of generative adversarial networks (Goodfellow et al., 2014) provides an effective approach to solve such a matching problem, e.g., the GAIL algorithm (Ho & Ermon, 2016).
Revised: The computational framework of generative adversarial networks (Goodfellow et al., 2014) provides an effective approach to solve such a matching problem, e.g., the GAIL algorithm (Ho & Ermon, 2016).

Original:  C AUSAL I MITATION WITHOUT S EQUENTIAL B ACKDOOR
Revised:  C AUSAL I MITATION WITHOUT S EQUENTIAL B ACKDOOR

Original: In this section, we investigate causal IRL beyond the condition of minimal sequential π -backdoor.
Revised: In this section, we investigate causal IRL beyond the condition of minimal sequential π -backdoor.

Original: Observe that the key to the reduction of the canonical causal IRL equation in Eq.
Revised: Observe that the key to the reduction of the canonical causal IRL equation in Eq.

Original: (4) lies in the identification of expected rewards
Revised: (4) lies in the identification of expected rewards

Original: E
Revised: E

Original: [ Y | do ( π )] had the latent reward Y been observed.
Revised: [ Y | do ( π )] had the latent reward Y been observed.

Original: Next we will study general conditions under which E
Revised: Next we will study general conditions under which E

Original: [ Y | do ( π )] is uniquely discernible from distribution
Revised: [ Y | do ( π )] is uniquely discernible from distribution

Original: P ( O , Y ) in the causal diagram G , called the identifiability of causal effects (Pearl, 2000, Def. 3.2.4).
Revised: P ( O , Y ) in the causal diagram G , called the identifiability of causal effects (Pearl, 2000, Def. 3.2.4).

Original: Definition 4 (Identifiability) .
Revised: Definition 4 (Identifiability) .

Original: Given a causal diagram G and a policy π ∼ S , the expected reward E
Revised: Given a causal diagram G and a policy π ∼ S , the expected reward E

Original: [ Y | do ( π )] is said to be identifiable from distribution P ( O , Y ) in G if E [ Y | do ( π )] is uniquely computable from P ( O , Y ) in any SCM M compatible with G .
Revised: [ Y | do ( π )] is said to be identifiable from distribution P ( O , Y ) in G if E [ Y | do ( π )] is uniquely computable from P ( O , Y ) in any SCM M compatible with G .

Original: We say a policy scope S is identifiable (from P ( O , Y ) in G ) if for all policies π ∼ S , the corresponding expected rewards E
Revised: We say a policy scope S is identifiable (from P ( O , Y ) in G ) if for all policies π ∼ S , the corresponding expected rewards E

Original: [ Y | do ( π )] are identifiable from P ( O , Y ) in G .
Revised: [ Y | do ( π )] are identifiable from P ( O , Y ) in G .

Original: Our next result shows that whenever an identifiable policy scope S is found, one could always reduce the causal IRL problem to the canonical optimization equation in Eq.
Revised: Our next result shows that whenever an identifiable policy scope S is found, one could always reduce the causal IRL problem to the canonical optimization equation in Eq.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if for any policy π ∼ S ,
Revised: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if for any policy π ∼ S ,

Original: the expected reward E
Revised: the expected reward E

Original: [ Y | do ( π )] is computable from P ( O , Y ) as
Revised: [ Y | do ( π )] is computable from P ( O , Y ) as

Original: [Equation]
Revised: [Equation]

Original: where subsets X ∗ ⊆
Revised: where subsets X ∗ ⊆

Original: X , Z ∗ ⊆
Revised: X , Z ∗ ⊆

Original: O \ X ; and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is a function of the observational distribution P ( O ) and policy π .
Revised: O \ X ; and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is a function of the observational distribution P ( O ) and policy π .

Original: [Figure]
Revised: [Figure]

Original: Thm. 2 suggests a general procedure to learn an imitating policy via causal IRL.
Revised: Thm. 2 suggests a general procedure to learn an imitating policy via causal IRL.

Original: Whenever an identifiable scope S is found, the identification formula in Eq.
Revised: Whenever an identifiable scope S is found, the identification formula in Eq.

Original: (8) permits one to reduce the optimization problem in Eq.
Revised: (8) permits one to reduce the optimization problem in Eq.

Original: (2) to the canonical equation in Eq.
Revised: (2) to the canonical equation in Eq.

Original: One could thus obtain an imitating policy π ∼ S by solving Eq.
Revised: One could thus obtain an imitating policy π ∼ S by solving Eq.

Original: (4) where the expert’s occupancy measure ρ ( x ∗ , z ∗ ) =
Revised: (4) where the expert’s occupancy measure ρ ( x ∗ , z ∗ ) =

Original: P ( x ∗ , z ∗ ) and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is given by Eq.
Revised: P ( x ∗ , z ∗ ) and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is given by Eq.

Original: As an example, consider the frontdoor diagram described in Fig.
Revised: As an example, consider the frontdoor diagram described in Fig.

Original: 2a and a policy scope
Revised: 2a and a policy scope

Original: S = {⟨ X, ∅⟩} .
Revised: S = {⟨ X, ∅⟩} .

Original: The expected reward E [ Y | do ( π )]
Revised: The expected reward E [ Y | do ( π )]

Original: = (cid:80) x
Revised: = (cid:80) x

Original: ′ E
Revised: ′ E

Original: [ Y | do ( x ′ )]
Revised: [ Y | do ( x ′ )]

Original: π ( x ′ ) and E [ Y | do ( x ′ )] is identifiable from P ( X, Y, Z ) using the frontdoor adjustment formula (Pearl, 2000, Thm. 3.3.4).
Revised: π ( x ′ ) and E [ Y | do ( x ′ )] is identifiable from P ( X, Y, Z ) using the frontdoor adjustment formula (Pearl, 2000, Thm. 3.3.4).

Original: The expected reward E [ Y | do ( π )] of any policy π ( X ) could be written as:
Revised: The expected reward E [ Y | do ( π )] of any policy π ( X ) could be written as:

Original: [Equation]
Revised: [Equation]

Original: Let occupancy measures ρ ( x, z )
Revised: Let occupancy measures ρ ( x, z )

Original: =
Revised: =

Original: P ( x, z ) and ρ π ( x, z ) = P ( x ) (cid:80) x
Revised: P ( x, z ) and ρ π ( x, z ) = P ( x ) (cid:80) x

Original: ′ P
Revised: ′ P

Original: z | x ′ )
Revised: z | x ′ )

Original: π ( x ′ ) .
Revised: π ( x ′ ) .

Original: We could thus learn an imitating policy in the frontdoor diagram by solving the canonical equation given by:
Revised: We could thus learn an imitating policy in the frontdoor diagram by solving the canonical equation given by:

Original: [Equation]
Revised: [Equation]

Original: where R is a hypothesis class of the reward function
Revised: where R is a hypothesis class of the reward function

Original: r ( x, z ) ≜
Revised: r ( x, z ) ≜

Original: E [ Y | x, z ] .
Revised: E [ Y | x, z ] .

Original: The solution π ∗ ( X ) is an imitating policy performing at least as well as the expert’s behavior policy if the gap ν ∗ ≤ 0 .
Revised: The solution π ∗ ( X ) is an imitating policy performing at least as well as the expert’s behavior policy if the gap ν ∗ ≤ 0 .

Original: Next, we will describe how to obtain the identification formula in Eq.
Revised: Next, we will describe how to obtain the identification formula in Eq.

Original: (8) provided with an identifiable scope S .
Revised: (8) provided with an identifiable scope S .

Original: Without loss of generality, we will assume that the reward Y is the only endogenous variable that is latent in the causal diagram G ,
Revised: Without loss of generality, we will assume that the reward Y is the only endogenous variable that is latent in the causal diagram G ,

Original: i.e., V =
Revised: i.e., V =

Original: O ∪ { Y } .
Revised: O ∪ { Y } .

Original: 1 We will utilize a special type of clustering of nodes in the causal diagram G , called the confounded component (for short, c-component).
Revised: 1 We will utilize a special type of clustering of nodes in the causal diagram G , called the confounded component (for short, c-component).

Original: Definition 5 (C-component (Tian & Pearl, 2002)) .
Revised: Definition 5 (C-component (Tian & Pearl, 2002)) .

Original: For a causal diagram G , a subset C ⊆ V is a c-component if any pair V
Revised: For a causal diagram G , a subset C ⊆ V is a c-component if any pair V

Original: i , V j ∈ C is connected by a bi-directed path in G .
Revised: i , V j ∈ C is connected by a bi-directed path in G .

Original: For instance, the frontdoor diagram in Fig.
Revised: For instance, the frontdoor diagram in Fig.

Original: 2a contains two c-components C 1 =
Revised: 2a contains two c-components C 1 =

Original: { X, Y } and
Revised: { X, Y } and

Original: C 2 = { Z } .
Revised: C 2 = { Z } .

Original: We will utilize a sound and complete procedure
Revised: We will utilize a sound and complete procedure

Original: I DENTIFY (Tian, 2002; 2008) for identifying causal effects
Revised: I DENTIFY (Tian, 2002; 2008) for identifying causal effects

Original: E [ Y | do ( π )] of an arbitrary policy π ∼ S .
Revised: E [ Y | do ( π )] of an arbitrary policy π ∼ S .

Original: Particularly, I DENTIFY takes as input the causal diagram G , a reward Y , and a policy scope S .
Revised: Particularly, I DENTIFY takes as input the causal diagram G , a reward Y , and a policy scope S .

Original: It returns an identification formula for E [ Y | do ( π )] from P ( O , Y ) if expected rewards of all policies π ∼ S are identifiable.
Revised: It returns an identification formula for E [ Y | do ( π )] from P ( O , Y ) if expected rewards of all policies π ∼ S are identifiable.

Original: Otherwise,
Revised: Otherwise,

Original: I DENTIFY ( G , Y, S ) =
Revised: I DENTIFY ( G , Y, S ) =

Original: “ F AIL ” .
Revised: “ F AIL ” .

Original: Details of I DENTIFY are shown in (Zhang et al., 2020, Appendix
Revised: Details of I DENTIFY are shown in (Zhang et al., 2020, Appendix

Original: B).
Revised: B).

Original: Recall that G S is the causal diagram of submodel M π induced by policy π ∼ S .
Revised: Recall that G S is the causal diagram of submodel M π induced by policy π ∼ S .

Original: Fig.
Revised: Fig.

Original: 2b shows diagram G S obtained from the frontdoor graph G and scope S
Revised: 2b shows diagram G S obtained from the frontdoor graph G and scope S

Original: = {⟨ X, ∅⟩} described in Fig.
Revised: = {⟨ X, ∅⟩} described in Fig.

Original: 2a.
Revised: 2a.

Original: Let
Revised: Let

Original: Z Y =
Revised: Z Y =

Original: An ( Y ) be ancestors of Y in G S .
Revised: An ( Y ) be ancestors of Y in G S .

Original: Our next result shows that I DENTIFY ( G , Y, S ) is ensured to find an identification formula of the form in Eq.
Revised: Our next result shows that I DENTIFY ( G , Y, S ) is ensured to find an identification formula of the form in Eq.

Original: (8) when it is identifiable.
Revised: (8) when it is identifiable.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if I DENTIFY ( G , Y, S ) ̸
Revised: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if I DENTIFY ( G , Y, S ) ̸

Original: = “ F AIL ” .
Revised: = “ F AIL ” .

Original: Moreover, I DENTIFY ( G , Y, S ) returns an identification formula of the form in Eq.
Revised: Moreover, I DENTIFY ( G , Y, S ) returns an identification formula of the form in Eq.

Original: (8) where X ∗ =
Revised: (8) where X ∗ =

Original: Pa ( C Y ) ∩ X and Z ∗ =
Revised: Pa ( C Y ) ∩ X and Z ∗ =

Original: Pa ( C Y ) \ ( { Y } ∪ X ) ; and C Y is a c-component containing reward Y in subgraph G
Revised: Pa ( C Y ) \ ( { Y } ∪ X ) ; and C Y is a c-component containing reward Y in subgraph G

Original: [ An ( Z Y )] .
Revised: [ An ( Z Y )] .

Original: For example, for the frontdoor diagram G in Fig.
Revised: For example, for the frontdoor diagram G in Fig.

Original: 2a, the manipulated diagram G S with scope S = {⟨ X, ∅⟩} is described in Fig.
Revised: 2a, the manipulated diagram G S with scope S = {⟨ X, ∅⟩} is described in Fig.

Original: 2b.
Revised: 2b.

Original: Since Z Y =
Revised: Since Z Y =

Original: An ( Y ) G S =
Revised: An ( Y ) G S =

Original: { X, Z, Y } , C Y is thus given by { X, Y } .
Revised: { X, Z, Y } , C Y is thus given by { X, Y } .

Original: Lem.
Revised: Lem.

Original: 1 implies that X ∗ =
Revised: 1 implies that X ∗ =

Original: Pa ( { X, Y } ) ∩ { X } = { X } and Z ∗ =
Revised: Pa ( { X, Y } ) ∩ { X } = { X } and Z ∗ =

Original: Pa ( { X, Y } ) \ { X, Y } = { Z } .
Revised: Pa ( { X, Y } ) \ { X, Y } = { Z } .

Original: Applying I DENTIFY ( G , Y, {⟨ X, ∅} ) returns the frontdoor adjustment formula in Eq.
Revised: Applying I DENTIFY ( G , Y, {⟨ X, ∅} ) returns the frontdoor adjustment formula in Eq.

Original: 3. S EARCHING FOR I DENTIFIABLE P OLICY S COPES
Revised: 3. S EARCHING FOR I DENTIFIABLE P OLICY S COPES

Original: The remainder of this section describes an effective algorithm to find identifiable policy scopes S had the latent reward signal Y been observed.
Revised: The remainder of this section describes an effective algorithm to find identifiable policy scopes S had the latent reward signal Y been observed.

Original: Let S denote the collection of all identifiable policy scopes
Revised: Let S denote the collection of all identifiable policy scopes

Original: S from distribution P ( O , Y ) in the causal diagram G .
Revised: S from distribution P ( O , Y ) in the causal diagram G .

Original: Our algorithm L IST
Revised: Our algorithm L IST

Original: I D S COPE , described in
Revised: I D S COPE , described in

Original: Alg. 1, enumerates elements in S .
Revised: Alg. 1, enumerates elements in S .

Original: It takes as input a causal diagram G , a reward signal Y , and subsets
Revised: It takes as input a causal diagram G , a reward signal Y , and subsets

Original: L = ∅ and R =
Revised: L = ∅ and R =

Original: (cid:83) ni =1 PA ∗ i .
Revised: (cid:83) ni =1 PA ∗ i .

Original: More specifically, L IST I D S COPE maintains two scopes S l ⊆ S r
Revised: More specifically, L IST I D S COPE maintains two scopes S l ⊆ S r

Original: (Step 2).
Revised: (Step 2).

Original: It performs backtrack search to find identifiable scopes S in G such that S l ⊆ S ⊆ S r .
Revised: It performs backtrack search to find identifiable scopes S in G such that S l ⊆ S ⊆ S r .

Original: It aborts branches that either (1) all subscopes in S r are identifiable (Step 3); or (2) all subscopes containing
Revised: It aborts branches that either (1) all subscopes in S r are identifiable (Step 3); or (2) all subscopes containing

Original: S l are non-identifiable (Step 6).
Revised: S l are non-identifiable (Step 6).

Original: The following proposition supports our aborting criterion.
Revised: The following proposition supports our aborting criterion.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given a causal diagram G , for policy scopes S ′ ⊆ S , S ′ is identifiable from distribution
Revised: Given a causal diagram G , for policy scopes S ′ ⊆ S , S ′ is identifiable from distribution

Original: P ( O , Y ) in G if S is identifiable from P ( O , Y ) in G .
Revised: P ( O , Y ) in G if S is identifiable from P ( O , Y ) in G .

Original: 1: Input: G , Y and subsets L ⊆ R 2: Let scopes S r = {⟨ X i , R ∩ PA ∗ i ⟩} ni =1 and
Revised: 1: Input: G , Y and subsets L ⊆ R 2: Let scopes S r = {⟨ X i , R ∩ PA ∗ i ⟩} ni =1 and

Original: S l = {⟨ X i , L ∩ PA ∗ i ⟩} ni
Revised: S l = {⟨ X i , L ∩ PA ∗ i ⟩} ni

Original: =1 .
Revised: =1 .

Original: 3: if I DENTIFY ( G , Y, S r ) ̸ = “ F AIL ′′ then 4:
Revised: 3: if I DENTIFY ( G , Y, S r ) ̸ = “ F AIL ′′ then 4:

Original: Output S r .
Revised: Output S r .

Original: 5: end if 6: if I DENTIFY ( G , Y, S l )
Revised: 5: end if 6: if I DENTIFY ( G , Y, S l )

Original: ̸ = “ F AIL ′′ then 7:
Revised: ̸ = “ F AIL ′′ then 7:

Original: Pick an arbitrary V ∈ R \ L .
Revised: Pick an arbitrary V ∈ R \ L .

Original: L IST I D S COPE ( G , Y, L ∪ { V } , R ) .
Revised: L IST I D S COPE ( G , Y, L ∪ { V } , R ) .

Original: L IST I D S COPE ( G , Y, L , R \ { V } ) .
Revised: L IST I D S COPE ( G , Y, L , R \ { V } ) .

Original: 10: end if
Revised: 10: end if

Original: At Step 7, L IST I D S COPE picks an arbitrary variable V that is included in input covariates R but not in L .
Revised: At Step 7, L IST I D S COPE picks an arbitrary variable V that is included in input covariates R but not in L .

Original: It then recursively returns all identifiable policy scopes S in G : the first recursive call returns scopes taking V as an input for some actions X i ∈ X and the second call return all scopes that do not consider V when selecting values for all actions X .
Revised: It then recursively returns all identifiable policy scopes S in G : the first recursive call returns scopes taking V as an input for some actions X i ∈ X and the second call return all scopes that do not consider V when selecting values for all actions X .

Original: We say a policy π is associated with a collection of policy scopes S , denoted by π ∼ S , if there exists S ∈ S so that π ∼ S .
Revised: We say a policy π is associated with a collection of policy scopes S , denoted by π ∼ S , if there exists S ∈ S so that π ∼ S .

Original: It is possible to show that L IST I D S COPE produces a collection of identifiable scopes that is sufficient for the imitation task.
Revised: It is possible to show that L IST I D S COPE produces a collection of identifiable scopes that is sufficient for the imitation task.

Original: Theorem 3.
Revised: Theorem 3.

Original: For a causal diagram G and a reward Y , L IST I D S COPE ( G , Y, ∅ , (cid:83) ni =1 PA ∗ i ) enumerates a subset S ∗ ⊆ S
Revised: For a causal diagram G and a reward Y , L IST I D S COPE ( G , Y, ∅ , (cid:83) ni =1 PA ∗ i ) enumerates a subset S ∗ ⊆ S

Original: so that for any π ∼ S , there is π ∗ ∼ S ∗ where E [ Y | do ( π )]
Revised: so that for any π ∼ S , there is π ∗ ∼ S ∗ where E [ Y | do ( π )]

Original: =
Revised: =

Original: E [ Y | do ( π ∗ )] .
Revised: E [ Y | do ( π ∗ )] .

Original: Moreover, L IST I D S COPE outputs identifiable policy scopes with a polynomial delay.
Revised: Moreover, L IST I D S COPE outputs identifiable policy scopes with a polynomial delay.

Original: This follows from the observation that L IST I D S COPE searches over a tree of policy scopes with height at most | (cid:83) ni =1 PA ∗ i | and I DENTIFY ( G , Y, S ) terminates in polynomial steps w.r.t.
Revised: This follows from the observation that L IST I D S COPE searches over a tree of policy scopes with height at most | (cid:83) ni =1 PA ∗ i | and I DENTIFY ( G , Y, S ) terminates in polynomial steps w.r.t.

Original: the size of diagram G .
Revised: the size of diagram G .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we demonstrate our framework on various imitation learning tasks, ranging from synthetic causal models to real-world datasets, including highway driving (Krajewski et al., 2018) and images (LeCun, 1998).
Revised: In this section, we demonstrate our framework on various imitation learning tasks, ranging from synthetic causal models to real-world datasets, including highway driving (Krajewski et al., 2018) and images (LeCun, 1998).

Original: We find that our approach is able to incorporate parametric knowledge about the reward function and achieve effective imitating policies across different causal diagrams.
Revised: We find that our approach is able to incorporate parametric knowledge about the reward function and achieve effective imitating policies across different causal diagrams.

Original: For all experiments, we evaluate our proposed Causal-IRL based on the canonical equation formulation in Eq.
Revised: For all experiments, we evaluate our proposed Causal-IRL based on the canonical equation formulation in Eq.

Original: As a baseline, we also include: (1) standard BC mimicking the expert’s nominal behavior policy; (2) standard IRL utilizing all observed covariates preceding every X i ∈
Revised: As a baseline, we also include: (1) standard BC mimicking the expert’s nominal behavior policy; (2) standard IRL utilizing all observed covariates preceding every X i ∈

Original: X while being blind to causal relationships in the underlying model; and (3) Causal-BC (Zhang et al., 2020; Kumor et al., 2021) that learn an imitating policy with the sequential π -backdoor criterion.
Revised: X while being blind to causal relationships in the underlying model; and (3) Causal-BC (Zhang et al., 2020; Kumor et al., 2021) that learn an imitating policy with the sequential π -backdoor criterion.

Original: We refer readers to Appendix D for additional experiments and more discussions on the experimental setup.
Revised: We refer readers to Appendix D for additional experiments and more discussions on the experimental setup.

Original: Backdoor Consider an SCM instance compatible with Fig. 1c including binary observed variables Z 1 , X 1 , Z 2 , X 2 , Y ∈ { 0 , 1 } .
Revised: Backdoor Consider an SCM instance compatible with Fig. 1c including binary observed variables Z 1 , X 1 , Z 2 , X 2 , Y ∈ { 0 , 1 } .

Original: Causal-BC utilizes a sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} ; while Causal-IRL utilizes the scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} satisfying the minimal sequential π -backdoor.
Revised: Causal-BC utilizes a sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} ; while Causal-IRL utilizes the scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} satisfying the minimal sequential π -backdoor.

Original: Simulation results, shown in Fig.
Revised: Simulation results, shown in Fig.

Original: 3a, reveal that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting additional parametric knowledge about the expected reward
Revised: 3a, reveal that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting additional parametric knowledge about the expected reward

Original: E
Revised: E

Original: [ Y | X 1 , X 2 , Z 2 ] ; Causal-BC is able to achieve the expert’s performance.
Revised: [ Y | X 1 , X 2 , Z 2 ] ; Causal-BC is able to achieve the expert’s performance.

Original: Unsurprisingly, neither BC nor IRL is able to obtain an imitating policy.
Revised: Unsurprisingly, neither BC nor IRL is able to obtain an imitating policy.

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (a) Backdoor
Revised: Expert (a) Backdoor

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (b) Highway Driving
Revised: Expert (b) Highway Driving

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (c) MNIST Digits
Revised: Expert (c) MNIST Digits

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (d) Infinite MDPUC
Revised: Expert (d) Infinite MDPUC

Original: Highway Driving We consider a learning scenario where the agent learns a driving policy from the observed trajectories of a human expert.
Revised: Highway Driving We consider a learning scenario where the agent learns a driving policy from the observed trajectories of a human expert.

Original: Causal diagram of this example is provided in Fig.(Appendix D) where X 1 is the accelerations of the ego vehicle at the previous step; Z 1 is both longitudinal and lateral historical accelerations of the ego vehicle two steps ago; X 2 is the velocity of the ego vehicle; Z 2 is the velocity of the preceding vehicle; W indicates the information from surrounding vehicles.
Revised: Causal diagram of this example is provided in Fig.(Appendix D) where X 1 is the accelerations of the ego vehicle at the previous step; Z 1 is both longitudinal and lateral historical accelerations of the ego vehicle two steps ago; X 2 is the velocity of the ego vehicle; Z 2 is the velocity of the preceding vehicle; W indicates the information from surrounding vehicles.

Original: Values of X 1 , X 2 , Z 1 , Z 2 are drawn from a real-world driving dataset HighD
Revised: Values of X 1 , X 2 , Z 1 , Z 2 are drawn from a real-world driving dataset HighD

Original: Krajewski et al.
Revised: Krajewski et al.

Original: The reward Y is decided by a non-linear function f Y ( X 2 , Z 2 , U Y ) .
Revised: The reward Y is decided by a non-linear function f Y ( X 2 , Z 2 , U Y ) .

Original: Both
Revised: Both

Original: Causal-IRL and Causal-BC utilize the scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} .
Revised: Causal-IRL and Causal-BC utilize the scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} .

Original: Causal-IRL also exploits the additional knowledge that the expected reward
Revised: Causal-IRL also exploits the additional knowledge that the expected reward

Original: E
Revised: E

Original: [ Y | X 1 , X 2 , Z 2 ] is a monotone function via reward augmentation (Li et al., 2017).
Revised: [ Y | X 1 , X 2 , Z 2 ] is a monotone function via reward augmentation (Li et al., 2017).

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 3b.
Revised: 3b.

Original: We found that
Revised: We found that

Original: Causal-IRL performs the best among all strategies.
Revised: Causal-IRL performs the best among all strategies.

Original: Causal-BC is able to achieve the expert’s performance.
Revised: Causal-BC is able to achieve the expert’s performance.

Original: BC and IRL perform the worst among all and fail to obtain an imitating policy.
Revised: BC and IRL perform the worst among all and fail to obtain an imitating policy.

Original: MNIST Digits Consider again the frontdoor diagram in Fig.
Revised: MNIST Digits Consider again the frontdoor diagram in Fig.

Original: 2a.
Revised: 2a.

Original: To evaluate the performance of our proposed approach in high-dimensional domains, we now replace variable Z with sampled images drawn from MNIST digits dataset (LeCun, 1998).
Revised: To evaluate the performance of our proposed approach in high-dimensional domains, we now replace variable Z with sampled images drawn from MNIST digits dataset (LeCun, 1998).

Original: The reward Y is decided by a linear function taking Z and an unobserved confounder U X,Y as input.
Revised: The reward Y is decided by a linear function taking Z and an unobserved confounder U X,Y as input.

Original: The Causal-IRL formulates the imitation problem as a two-person zero-sum game through the frontdoor adjustment described in Eq.
Revised: The Causal-IRL formulates the imitation problem as a two-person zero-sum game through the frontdoor adjustment described in Eq.

Original: (10), which can be solved by the MW algorithm (Freund & Schapire, 1999; Syed & Schapire, 2008).
Revised: (10), which can be solved by the MW algorithm (Freund & Schapire, 1999; Syed & Schapire, 2008).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3c, simulation results reveal that Causal-IRL outperforms Causal-BC and BC ; while IRL performs the worst among all the algorithms.
Revised: 3c, simulation results reveal that Causal-IRL outperforms Causal-BC and BC ; while IRL performs the worst among all the algorithms.

Original: Infinite MDPUC To demonstrate our proposed framework in the sequential decision-making setting with an infinite horizon, we consider a generalized Markov decision process incorporating unobserved confounders (Ruan & Di, 2022), called the MDPUC (Zhang & Bareinboim, 2022).
Revised: Infinite MDPUC To demonstrate our proposed framework in the sequential decision-making setting with an infinite horizon, we consider a generalized Markov decision process incorporating unobserved confounders (Ruan & Di, 2022), called the MDPUC (Zhang & Bareinboim, 2022).

Original: This sequential model simulates real-world driving dynamics.
Revised: This sequential model simulates real-world driving dynamics.

Original: By exploiting the Markov property over time steps, we are able to decompose the causal diagram over the infinite horizon into a collection of sub-graphs, one for each time step i = 1 , 2 , . . . .
Revised: By exploiting the Markov property over time steps, we are able to decompose the causal diagram over the infinite horizon into a collection of sub-graphs, one for each time step i = 1 , 2 , . . . .

Original: Fig.
Revised: Fig.

Original: 1d shows the causal diagram spanning time steps i = 1 , 2 , 3 .
Revised: 1d shows the causal diagram spanning time steps i = 1 , 2 , 3 .

Original: As a comparison, BC and IRL still utilize the stationary policy {⟨ X i , { Z i }⟩} .
Revised: As a comparison, BC and IRL still utilize the stationary policy {⟨ X i , { Z i }⟩} .

Original: By applying
Revised: By applying

Original: Thm. 

Revised: Thm.

Original: at each time step, we obtain a π -backdoor admissible policy scope {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} for Causal-IRL and Causal-BC .
Revised: at each time step, we obtain a π -backdoor admissible policy scope {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} for Causal-IRL and Causal-BC .

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 3d.
Revised: 3d.

Original: One could see by inspection that Causal-IRL performs the best and achieves the expert’s performance.
Revised: One could see by inspection that Causal-IRL performs the best and achieves the expert’s performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper investigates imitation learning via inverse reinforcement learning (IRL) in the semantical framework of structural causal models.
Revised: This paper investigates imitation learning via inverse reinforcement learning (IRL) in the semantical framework of structural causal models.

Original: The goal is to find an effective imitating policy that performs at least as well as the expert’s behavior policy from combinations of demonstration data, qualitative knowledge the data-generating mechanisms represented as a causal diagram, and quantitative knowledge about the reward function.
Revised: The goal is to find an effective imitating policy that performs at least as well as the expert’s behavior policy from combinations of demonstration data, qualitative knowledge the data-generating mechanisms represented as a causal diagram, and quantitative knowledge about the reward function.

Original: We provide a graphical criterion (Thm. 1) based on the sequential backdoor, which allows one to obtain an imitating policy by solving a canonical optimization equation of causal IRL.
Revised: We provide a graphical criterion (Thm. 1) based on the sequential backdoor, which allows one to obtain an imitating policy by solving a canonical optimization equation of causal IRL.

Original: Such a canonical formulation addresses the challenge of the presence of unobserved confounders (UCs), and is solvable by leveraging standard IRL algorithms (Props. 1 and 2).
Revised: Such a canonical formulation addresses the challenge of the presence of unobserved confounders (UCs), and is solvable by leveraging standard IRL algorithms (Props. 1 and 2).

Original: Finally, we move beyond the backdoor criterion and show that the canonical equation is achievable whenever expected rewards of policies are identifiable had the reward also been observed (Thms. 2 and 3).
Revised: Finally, we move beyond the backdoor criterion and show that the canonical equation is achievable whenever expected rewards of policies are identifiable had the reward also been observed (Thms. 2 and 3).

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: This paper investigates the theoretical framework of causal inverse RL from the natural trajectories of an expert demonstrator, even when the reward signal is unobserved.
Revised: This paper investigates the theoretical framework of causal inverse RL from the natural trajectories of an expert demonstrator, even when the reward signal is unobserved.

Original: Input covariates used by the expert to determine the original values of the action are unknown, introducing unobserved confounding bias in demonstration data.
Revised: Input covariates used by the expert to determine the original values of the action are unknown, introducing unobserved confounding bias in demonstration data.

Original: Our framework may apply to various fields in reality, including autonomous vehicle development, industrial automation, and chronic disease management.
Revised: Our framework may apply to various fields in reality, including autonomous vehicle development, industrial automation, and chronic disease management.

Original: A positive impact of this work is that we discuss the potential risk of training IRL policy from demonstrations with the presence of unobserved confounding (UC).
Revised: A positive impact of this work is that we discuss the potential risk of training IRL policy from demonstrations with the presence of unobserved confounding (UC).

Original: Our formulation of causal IRL is inherently robust against confounding bias.
Revised: Our formulation of causal IRL is inherently robust against confounding bias.

Original: For example, solving the causal IRL problem in Eq.
Revised: For example, solving the causal IRL problem in Eq.

Original: (2) requires the imitator to learn an effective policy that maximizes the reward in a worst-case causal model where the performance gap between the expert and imitator is the largest possible.
Revised: (2) requires the imitator to learn an effective policy that maximizes the reward in a worst-case causal model where the performance gap between the expert and imitator is the largest possible.

Original: More broadly, automated decision systems using causal inference methods prioritize safety and robustness during their decision-making processes.
Revised: More broadly, automated decision systems using causal inference methods prioritize safety and robustness during their decision-making processes.

Original: Such requirements are increasingly essential since black-box
Revised: Such requirements are increasingly essential since black-box

Original: AI systems are prevalent, and our understandings of their potential implications are still limited.
Revised: AI systems are prevalent, and our understandings of their potential implications are still limited.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The complete proof of all theoretical results presented in this paper, including Thms. 1 and 2, is provided in Appendix B. Details on the implementation of the proposed algorithms are included
Revised: The complete proof of all theoretical results presented in this paper, including Thms. 1 and 2, is provided in Appendix B. Details on the implementation of the proposed algorithms are included

Original: Appendix C. Finally, Appendix D provides a detailed description of the experimental setup.
Revised: Appendix C. Finally, Appendix D provides a detailed description of the experimental setup.

Original: Readers could find all appendices as part of the supplementary text after “References” section.
Revised: Readers could find all appendices as part of the supplementary text after “References” section.

Original: We provided references to all existing datasets used in experiments, including H IGH D (Krajewski et al., 2018) and
Revised: We provided references to all existing datasets used in experiments, including H IGH D (Krajewski et al., 2018) and

Original: MNIST (LeCun, 1998).
Revised: MNIST (LeCun, 1998).

Original: Other experiments are synthetic and do not introduce any new assets.
Revised: Other experiments are synthetic and do not introduce any new assets.

Original: A D ERIVATION OF IRL P OLICIES
Revised: A D ERIVATION OF IRL P OLICIES

Original: In this section, we provide detailed derivation of IRL policies in Introduction (Sec. 1).
Revised: In this section, we provide detailed derivation of IRL policies in Introduction (Sec. 1).

Original: Fig.
Revised: Fig.

Original: 1a Let F be a set of reward function F = { f Y ( x, z ) =
Revised: 1a Let F be a set of reward function F = { f Y ( x, z ) =

Original: αx + βz − γxz | 0 <
Revised: αx + βz − γxz | 0 <

Original: α < γ } .
Revised: α < γ } .

Original: Fix any reward function f Y ∈ R .
Revised: Fix any reward function f Y ∈ R .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: That is, once a reward function f Y is fixed, the IRL policy is an optimal policy
Revised: That is, once a reward function f Y is fixed, the IRL policy is an optimal policy

Original: X ← π ∗ ( Z ) maximizing the reward f Y .
Revised: X ← π ∗ ( Z ) maximizing the reward f Y .

Original: Note that for any policy X ← π ( Z ) ,
Revised: Note that for any policy X ← π ( Z ) ,

Original: [Equation]
Revised: [Equation]

Original: For Z = 0 , since α > 0 , the optimal action π ∗ ( Z = 0) is thus given by
Revised: For Z = 0 , since α > 0 , the optimal action π ∗ ( Z = 0) is thus given by

Original: [Equation]
Revised: [Equation]

Original: Similarly, for Z = 1 , the optimal action π ∗ ( Z = 1) is given by
Revised: Similarly, for Z = 1 , the optimal action π ∗ ( Z = 1) is given by

Original: [Equation]
Revised: [Equation]

Original: The above solution follows from α < γ .
Revised: The above solution follows from α < γ .

Original: The above equations together imply that the optimal policy π ∗ : X ← ¬ Z .
Revised: The above equations together imply that the optimal policy π ∗ : X ← ¬ Z .

Original: Finally, since π ∗ is the solution of the outer minimization step for any reward function f Y ∈ F , it is verifiable that π ∗ must be the policy obtained by IRL algorithms.
Revised: Finally, since π ∗ is the solution of the outer minimization step for any reward function f Y ∈ F , it is verifiable that π ∗ must be the policy obtained by IRL algorithms.

Original: [Equation]
Revised: [Equation]

Original: This means that any policy π ∗ mapping from X to Z could be a solution for the minimax problem min π max f Y
Revised: This means that any policy π ∗ mapping from X to Z could be a solution for the minimax problem min π max f Y

Original: [Equation]
Revised: [Equation]

Original: F E [ f Y
Revised: F E [ f Y

Original: [Equation]
Revised: [Equation]

Original: X, Z )]
Revised: X, Z )]

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: π )] .
Revised: π )] .

Original: For an arbitrary policy π ( X | Z ) ,
Revised: For an arbitrary policy π ( X | Z ) ,

Original: [Equation]
Revised: [Equation]

Original: Among quantities in the above equation, for any x, z ,
Revised: Among quantities in the above equation, for any x, z ,

Original: [Equation]
Revised: [Equation]

Original: The last step follows since U 1 , U 2 are both uniformly drawn over { 0 , 1 } and Z ← U 1 ⊕ U 2 .
Revised: The last step follows since U 1 , U 2 are both uniformly drawn over { 0 , 1 } and Z ← U 1 ⊕ U 2 .

Original: This implies that the expected reward of any policy π is equal to
Revised: This implies that the expected reward of any policy π is equal to

Original: [Equation]
Revised: [Equation]

Original: Therefore, we must have E [ Y | do ( π ∗ )]
Revised: Therefore, we must have E [ Y | do ( π ∗ )]

Original: = 0 .
Revised: = 0 .

Original: 5 where π ∗ is a solution obtained by IRL algorithms.
Revised: 5 where π ∗ is a solution obtained by IRL algorithms.

Original: B P ROOFS
Revised: B P ROOFS

Original: In this section, we provide proofs for the theoretical results presented in the paper.
Revised: In this section, we provide proofs for the theoretical results presented in the paper.

Original: Throughout this paper, detailed parametrizations of the underlying SCM M are assumed to be unknown to the agent.
Revised: Throughout this paper, detailed parametrizations of the underlying SCM M are assumed to be unknown to the agent.

Original: Instead, the agent has access to a causal diagram G associated with M , and observed trajectories of the expert’s demonstrations, summarized as the observational distribution P ( O ) .
Revised: Instead, the agent has access to a causal diagram G associated with M , and observed trajectories of the expert’s demonstrations, summarized as the observational distribution P ( O ) .

Original: These theoretical assumptions are required for the our proposed causal IRL algorithm, which delineate the technical limitations of the work.
Revised: These theoretical assumptions are required for the our proposed causal IRL algorithm, which delineate the technical limitations of the work.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a causal diagram G , if there exists a minimal π -backdoor admissible scope
Revised: Given a causal diagram G , if there exists a minimal π -backdoor admissible scope

Original: S = {⟨ X i , Z i ⟩} ni =1 in G , consider the following conditions:
Revised: S = {⟨ X i , Z i ⟩} ni =1 in G , consider the following conditions:

Original: Let effective actions X ∗ =
Revised: Let effective actions X ∗ =

Original: X ∩ An ( Y ) G S and effective covariates Z ∗ =
Revised: X ∩ An ( Y ) G S and effective covariates Z ∗ =

Original: (cid:83) X i ∈
Revised: (cid:83) X i ∈

Original: X ∗ Z i ; 2.
Revised: X ∗ Z i ; 2.

Original: For i = 1 , . . . , n + 1 , let X ∗ <
Revised: For i = 1 , . . . , n + 1 , let X ∗ <

Original: i = {∀ X j ∈ X ∗ | j < i } and Z ∗ <
Revised: i = {∀ X j ∈ X ∗ | j < i } and Z ∗ <

Original: i =
Revised: i =

Original: (cid:83) X j ∈ X ∗ <
Revised: (cid:83) X j ∈ X ∗ <

Original: i Z j .
Revised: i Z j .

Original: Then, for any policy π ∼ S , the expected reward E [ Y | do ( π )] is computable from P ( O , Y ) as:
Revised: Then, for any policy π ∼ S , the expected reward E [ Y | do ( π )] is computable from P ( O , Y ) as:

Original: [Equation]
Revised: [Equation]

Original: where the occupancy measure ρ π ( x ∗ , z ∗ )
Revised: where the occupancy measure ρ π ( x ∗ , z ∗ )

Original: = (cid:81)
Revised: = (cid:81)

Original: X i ∈
Revised: X i ∈

Original: X ∗ P (cid:0) z i | x ∗ <i , z ∗ <i (cid:1)
Revised: X ∗ P (cid:0) z i | x ∗ <i , z ∗ <i (cid:1)

Original: π i ( x i |
Revised: π i ( x i |

Original: z i ) .
Revised: z i ) .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, assume that X = X ∗ , i.e., all actions in X are active after policy π ∼ S is deployed.
Revised: Without loss of generality, assume that X = X ∗ , i.e., all actions in X are active after policy π ∼ S is deployed.

Original: Otherwise, we could always reduce the action set X to the active actions X ∗ since P ( Y | do ( π ))
Revised: Otherwise, we could always reduce the action set X to the active actions X ∗ since P ( Y | do ( π ))

Original: =
Revised: =

Original: P ( Y | do ( π X ∗ )) where π X ∗ is sequence of decision rules { π i | X
Revised: P ( Y | do ( π X ∗ )) where π X ∗ is sequence of decision rules { π i | X

Original: i ∈
Revised: i ∈

Original: X ∗ } with restriction to X ∗ .
Revised: X ∗ } with restriction to X ∗ .

Original: For any i = 1 , . . . , n , let X i =
Revised: For any i = 1 , . . . , n , let X i =

Original: { X 1 , . . .
Revised: { X 1 , . . .

Original: , X i − 1 } and X ≤ i =
Revised: , X i − 1 } and X ≤ i =

Original: X <i ∪ { X i } .
Revised: X <i ∪ { X i } .

Original: Similarly, we define
Revised: Similarly, we define

Original: Z <i = (cid:83) j<i Z j and Z ≤
Revised: Z <i = (cid:83) j<i Z j and Z ≤

Original: i
Revised: i

Original: =
Revised: =

Original: Z <i ∪ Z i .
Revised: Z <i ∪ Z i .

Original: Let π ≥
Revised: Let π ≥

Original: i be a sequence of decision rules { π i , π i +1 , . .
Revised: i be a sequence of decision rules { π i , π i +1 , . .

Original: π n } and π >i = π ≥ i \ { π i } .
Revised: π n } and π >i = π ≥ i \ { π i } .

Original: It is sufficient to show that for any i = 1 , . . . , n ,
Revised: It is sufficient to show that for any i = 1 , . . . , n ,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: By basic probabilistic operations,
Revised: By basic probabilistic operations,

Original: [Equation]
Revised: [Equation]

Original: In the last step, P ( x <i , z ≤ i | do ( π ≥ i )) =
Revised: In the last step, P ( x <i , z ≤ i | do ( π ≥ i )) =

Original: P ( x <i , z ≤ i ) since all causal diagrams G ( i ) are directed acyclic graph; and X <i , Z ≤ i are non-descendants of X i , X i +1 , . . .
Revised: P ( x <i , z ≤ i ) since all causal diagrams G ( i ) are directed acyclic graph; and X <i , Z ≤ i are non-descendants of X i , X i +1 , . . .

Original: , X n .
Revised: , X n .

Original: Next we will show that
Revised: Next we will show that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is equivalent to show that
Revised: This is equivalent to show that

Original: [Equation]
Revised: [Equation]

Original: First, by definition of sequential π -backdoor,
Revised: First, by definition of sequential π -backdoor,

Original: [Equation]
Revised: [Equation]

Original: Suppose Eq.
Revised: Suppose Eq.

Original: (25) does not hold.
Revised: (25) does not hold.

Original: We must have a node W ∈ Z <i ∪{ X i } opening a collider path from
Revised: We must have a node W ∈ Z <i ∪{ X i } opening a collider path from

Original: X i to Y .
Revised: X i to Y .

Original: Assume that there exists a directed path from W to Y in G ( i ) X i .
Revised: Assume that there exists a directed path from W to Y in G ( i ) X i .

Original: Then one could construct backdoor path from X i to Y via node W , which contradicts Eq.
Revised: Then one could construct backdoor path from X i to Y via node W , which contradicts Eq.

Original: What remains is to show that there must exist a directed path from W to Y in G ( i ) X i .
Revised: What remains is to show that there must exist a directed path from W to Y in G ( i ) X i .

Original: Since S is minimal π -backdoor admissible, it is verifiable from (van der Zander et al., 2014, Lem. 3.4) that W ∈ An ( { Y } ) in G ( i ) .
Revised: Since S is minimal π -backdoor admissible, it is verifiable from (van der Zander et al., 2014, Lem. 3.4) that W ∈ An ( { Y } ) in G ( i ) .

Original: Consequently, the only case that W ̸∈ An ( { Y } ) in G ( i )
Revised: Consequently, the only case that W ̸∈ An ( { Y } ) in G ( i )

Original: X i is when
Revised: X i is when

Original: W ∈ An ( { X i } ) in G ( i ) .
Revised: W ∈ An ( { X i } ) in G ( i ) .

Original: Again, this allows us to construct a backdoor path between X i and Y via
Revised: Again, this allows us to construct a backdoor path between X i and Y via

Original: W in G ( i ) , which contradicts Eq.
Revised: W in G ( i ) , which contradicts Eq.

Original: Proposition 1.
Revised: Proposition 1.

Original: For a hypothesis class R = { r = w · ϕ | w ∈ S k } , the solution ν ∗ of the canonical equation in Eq.
Revised: For a hypothesis class R = { r = w · ϕ | w ∈ S k } , the solution ν ∗ of the canonical equation in Eq.

Original: (4) is obtainable by solving the following minimax problem:
Revised: (4) is obtainable by solving the following minimax problem:

Original: [Equation]
Revised: [Equation]

Original: where G is a k × n matrix given by G ( i, j )
Revised: where G is a k × n matrix given by G ( i, j )

Original: = (cid:80) x ∗ , z ∗ ϕ
Revised: = (cid:80) x ∗ , z ∗ ϕ

Original: ( i ) ( x ∗ , z ∗ ) ( ρ ( x ∗ , z ∗ ) − ρ π ( j ) ( x ∗ , z ∗ )) .
Revised: ( i ) ( x ∗ , z ∗ ) ( ρ ( x ∗ , z ∗ ) − ρ π ( j ) ( x ∗ , z ∗ )) .

Original: Proof.
Revised: Proof.

Original: Note that π is a mixed policy which is a probability distribution over deterministic policies compatible with scope S .
Revised: Note that π is a mixed policy which is a probability distribution over deterministic policies compatible with scope S .

Original: With a slight abuse of notation, we will treat π as a vector where π ( i ) is the probability assigned to the i -th deterministic policy π ( i ) .
Revised: With a slight abuse of notation, we will treat π as a vector where π ( i ) is the probability assigned to the i -th deterministic policy π ( i ) .

Original: The claimed statement follows immediately from the definition of the game matrix G .
Revised: The claimed statement follows immediately from the definition of the game matrix G .

Original: Proposition 2.
Revised: Proposition 2.

Original: For a hypothesis class R = { r : D X ∗ ×
Revised: For a hypothesis class R = { r : D X ∗ ×

Original: D Z ∗ (cid:55)→ R } regularized by ψ , the solution ν ∗ of the penalized canonical equation in Eq.
Revised: D Z ∗ (cid:55)→ R } regularized by ψ , the solution ν ∗ of the penalized canonical equation in Eq.

Original: (6) is obtainable by solving the following problem:
Revised: (6) is obtainable by solving the following problem:

Original: [Equation]
Revised: [Equation]

Original: where ψ ∗ be a conjugate function of ψ and is given by ψ ∗ = max r ∈ R X ×
Revised: where ψ ∗ be a conjugate function of ψ and is given by ψ ∗ = max r ∈ R X ×

Original: Z a ⊤ r − ψ ( r ) .
Revised: Z a ⊤ r − ψ ( r ) .

Original: Proof.
Revised: Proof.

Original: Let the vector α = ρ − ρ π .
Revised: Let the vector α = ρ − ρ π .

Original: The claimed statement follows from the definition of the conjugate function ψ ∗ .
Revised: The claimed statement follows from the definition of the conjugate function ψ ∗ .

Original: Theorem 2.
Revised: Theorem 2.

Original: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if for any policy π ∼ S ,
Revised: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if for any policy π ∼ S ,

Original: the expected reward E
Revised: the expected reward E

Original: [ Y | do ( π )] is computable from P ( O , Y ) as
Revised: [ Y | do ( π )] is computable from P ( O , Y ) as

Original: [Equation]
Revised: [Equation]

Original: where subsets X ∗ ⊆
Revised: where subsets X ∗ ⊆

Original: X , Z ∗ ⊆
Revised: X , Z ∗ ⊆

Original: O \ X ; and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is a function of the observational distribution P ( O ) and policy π .
Revised: O \ X ; and the imitator’s occupancy measure ρ π ( x ∗ , z ∗ ) is a function of the observational distribution P ( O ) and policy π .

Original: Proof.
Revised: Proof.

Original: the claimed statement follows from Lem.
Revised: the claimed statement follows from Lem.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if I DENTIFY ( G , Y, S ) ̸
Revised: Given a causal diagram G , a policy scope S is identifiable from P ( O , Y ) in G if and only if I DENTIFY ( G , Y, S ) ̸

Original: = “ F AIL ” .
Revised: = “ F AIL ” .

Original: Moreover, I DENTIFY ( G , Y, S ) returns an identification formula of the form in Eq.
Revised: Moreover, I DENTIFY ( G , Y, S ) returns an identification formula of the form in Eq.

Original: (8) where X ∗ =
Revised: (8) where X ∗ =

Original: Pa ( C Y ) ∩ X and Z ∗ =
Revised: Pa ( C Y ) ∩ X and Z ∗ =

Original: Pa ( C Y ) \ ( { Y } ∪ X ) ; and C Y is a c-component containing reward Y in subgraph G
Revised: Pa ( C Y ) \ ( { Y } ∪ X ) ; and C Y is a c-component containing reward Y in subgraph G

Original: [ An ( Z Y )] .
Revised: [ An ( Z Y )] .

Original: Proof.
Revised: Proof.

Original: We first review the identification algorithm for effects of policies with scope S in the causal diagram G (Tian, 2008).
Revised: We first review the identification algorithm for effects of policies with scope S in the causal diagram G (Tian, 2008).

Original: Let D
Revised: Let D

Original: =
Revised: =

Original: An ( { Y } ) \ X in the induced diagram G .
Revised: An ( { Y } ) \ X in the induced diagram G .

Original: For any policy π ∼ S ,
Revised: For any policy π ∼ S ,

Original: P ( Y | do ( π )) could be written as, following (Tian, 2008, Eq. 15):
Revised: P ( Y | do ( π )) could be written as, following (Tian, 2008, Eq. 15):

Original: [Equation]
Revised: [Equation]

Original: It has been shown that effects P ( Y | do ( π )) of all policies π ∼ S are identifiable if and only if
Revised: It has been shown that effects P ( Y | do ( π )) of all policies π ∼ S are identifiable if and only if

Original: Q [ D ] is identifiable from P ( O , Y ) in G (Tian, 2008; Correa & Bareinboim, 2019).
Revised: Q [ D ] is identifiable from P ( O , Y ) in G (Tian, 2008; Correa & Bareinboim, 2019).

Original: Let D 1 , . . .
Revised: Let D 1 , . . .

Original: , D m denote c-components in G [ D ] ; and let D Y denote the c-component in G [ D ] containing Y and let
Revised: , D m denote c-components in G [ D ] ; and let D Y denote the c-component in G [ D ] containing Y and let

Original: D 1 , . . .
Revised: D 1 , . . .

Original: , D m be other c-components.
Revised: , D m be other c-components.

Original: Q [ D ] could be written as ((Tian, 2002, Lemma 11)):
Revised: Q [ D ] could be written as ((Tian, 2002, Lemma 11)):

Original: [Equation]
Revised: [Equation]

Original: Similarly, let S 1 , . .
Revised: Similarly, let S 1 , . .

Original: , S n denote c-components in the diagram G .
Revised: , S n denote c-components in the diagram G .

Original: The algorithm then attempts to identify each c-factor Q
Revised: The algorithm then attempts to identify each c-factor Q

Original: [ D i ] from Q [ S j ] where S j is a c-component in G containing variables in D i .
Revised: [ D i ] from Q [ S j ] where S j is a c-component in G containing variables in D i .

Original: This procedure could be done by applying the I DENTIFY algorithm introduced in (Tian, 2002), which was later shown to be complete (Huang & Valtorta, 2006; Shpitser, 2008).
Revised: This procedure could be done by applying the I DENTIFY algorithm introduced in (Tian, 2002), which was later shown to be complete (Huang & Valtorta, 2006; Shpitser, 2008).

Original: Details of algorithm I DENTIFY are described in Alg. 2.
Revised: Details of algorithm I DENTIFY are described in Alg. 2.

Original: Algorithm 2:
Revised: Algorithm 2:

Original: DENTIFY
Revised: DENTIFY

Original: 1: Input: c-components C ⊆ T , a causal diagram G 2: Let A
Revised: 1: Input: c-components C ⊆ T , a causal diagram G 2: Let A

Original: =
Revised: =

Original: An ( C ) G [ T ] .
Revised: An ( C ) G [ T ] .

Original: if A = T then Output Q
Revised: if A = T then Output Q

Original: [ C ] = (cid:80) t \ c Q [ T ] .
Revised: [ C ] = (cid:80) t \ c Q [ T ] .

Original: 4: end if 5: if A = C then Output “F AIL ”.
Revised: 4: end if 5: if A = C then Output “F AIL ”.

Original: 6: end if 7: if C ⊂ A ⊂ T then 8: Assume that in G [ A ] , C is contained in a c-
Revised: 6: end if 7: if C ⊂ A ⊂ T then 8: Assume that in G [ A ] , C is contained in a c-

Original: component T ′ .
Revised: component T ′ .

Original: Compute Q [ T ′ ] from Q [ A ] = (cid:80) t \ a Q [ T ] by (Tian, 2002, Lemma 11).
Revised: Compute Q [ T ′ ] from Q [ A ] = (cid:80) t \ a Q [ T ] by (Tian, 2002, Lemma 11).

Original: Output I DENTIFY
Revised: Output I DENTIFY

Original: ( C , T ′ , G ) .
Revised: ( C , T ′ , G ) .

Original: 11: end if
Revised: 11: end if

Original: Since all variables in D \ { Y } are non-descendant of Y in G , it is verifiable that the identification formula for each Q [ D i ] is a function of distribution P ( O ) , not including reward Y .
Revised: Since all variables in D \ { Y } are non-descendant of Y in G , it is verifiable that the identification formula for each Q [ D i ] is a function of distribution P ( O ) , not including reward Y .

Original: Let S Y denote a c-component in G that contains D Y .
Revised: Let S Y denote a c-component in G that contains D Y .

Original: Therefore, it is sufficient to show that c-factor Q [ D Y ] is identifiable from Q [ S Y ] as follows:
Revised: Therefore, it is sufficient to show that c-factor Q [ D Y ] is identifiable from Q [ S Y ] as follows:

Original: [Equation]
Revised: [Equation]

Original: where S is a certain subset in O \ Pa ( D Y ) ; and g ( P ( O )) is a function consisting of a series of arithmetic operation on the observational distribution P ( O ) .
Revised: where S is a certain subset in O \ Pa ( D Y ) ; and g ( P ( O )) is a function consisting of a series of arithmetic operation on the observational distribution P ( O ) .

Original: We next show that I DENTIFY ( D Y , S Y , G ) returns an identification formula of the form Eq.
Revised: We next show that I DENTIFY ( D Y , S Y , G ) returns an identification formula of the form Eq.

Original: If S Y =
Revised: If S Y =

Original: An ( D Y ) G [ S Y ]
Revised: An ( D Y ) G [ S Y ]

Original: (Step 3), the algorithm returns an expression Q [ D Y ]
Revised: (Step 3), the algorithm returns an expression Q [ D Y ]

Original: = (cid:80) s Y \ d Y Q
Revised: = (cid:80) s Y \ d Y Q

Original: [ S Y ] .
Revised: [ S Y ] .

Original: (Tian, 2002, Lemma 7) shows that Q [ S Y ] is given by, following a topological ordering ≺ in G :
Revised: (Tian, 2002, Lemma 7) shows that Q [ S Y ] is given by, following a topological ordering ≺ in G :

Original: [Equation]
Revised: [Equation]

Original: where pa + =
Revised: where pa + =

Original: Pa ( C V ) \ { V } ; and C V is a c-component in subgraph G [
Revised: Pa ( C V ) \ { V } ; and C V is a c-component in subgraph G [

Original: [Equation]
Revised: [Equation]

Original: V ′ ∈ V | V ′ ≺ V } ] containing V .
Revised: V ′ ∈ V | V ′ ≺ V } ] containing V .

Original: Summing over variables in S Y \ D Y we obtain Eq.
Revised: Summing over variables in S Y \ D Y we obtain Eq.

Original: We next consider cases where the condition in Step 3 is not satisfied; and I DENTIFY ( D Y , S Y , G ) has to enter the recursive step in Step 10.
Revised: We next consider cases where the condition in Step 3 is not satisfied; and I DENTIFY ( D Y , S Y , G ) has to enter the recursive step in Step 10.

Original: We will utilize the following claim.
Revised: We will utilize the following claim.

Original: Claim 1.
Revised: Claim 1.

Original: At each recursive call in I DENTIFY
Revised: At each recursive call in I DENTIFY

Original: ( D Y , S Y , G ) (Step 10), Q [ T ′ ] could be written as:
Revised: ( D Y , S Y , G ) (Step 10), Q [ T ′ ] could be written as:

Original: [Equation]
Revised: [Equation]

Original: 31) where S is a certain subset in O \
Revised: 31) where S is a certain subset in O \

Original: Pa ( D Y ) ; and g ( P ( O )) function consisting of a series of arithmetic operation on P ( O ) .
Revised: Pa ( D Y ) ; and g ( P ( O )) function consisting of a series of arithmetic operation on P ( O ) .

Original: We note that S and g could be different for each different T ′ .
Revised: We note that S and g could be different for each different T ′ .

Original: The above claim allows us to derive the identification formula of the form Eq.
Revised: The above claim allows us to derive the identification formula of the form Eq.

Original: Since if Q [ D Y ] is identifiable, it must enter Step 3 after some recursive steps.
Revised: Since if Q [ D Y ] is identifiable, it must enter Step 3 after some recursive steps.

Original: I DENTIFY algorithm returns a formula
Revised: I DENTIFY algorithm returns a formula

Original: Q [ D Y ] =
Revised: Q [ D Y ] =

Original: (cid:80) t \ c Q [ T ] where A =
Revised: (cid:80) t \ c Q [ T ] where A =

Original: An ( D Y ) G [ T ] .
Revised: An ( D Y ) G [ T ] .

Original: Since Q [ T ] takes the form of Eq.
Revised: Since Q [ T ] takes the form of Eq.

Original: (31), we obtain the claimed statement Eq.
Revised: (31), we obtain the claimed statement Eq.

Original: We will prove Claim 1 by induction on the number of recursive call m in I DENTIFY ( C , T ′ , G ) .
Revised: We will prove Claim 1 by induction on the number of recursive call m in I DENTIFY ( C , T ′ , G ) .

Original: Base case m = 1 .
Revised: Base case m = 1 .

Original: This follows immediately from (Tian, 2002, Lems.
Revised: This follows immediately from (Tian, 2002, Lems.

Original: 10 and 11).
Revised: 10 and 11).

Original: At Step 9,
Revised: At Step 9,

Original: Q
Revised: Q

Original: [ A ] = (cid:80) s Y \ a Q [ S Y ] and Q [ S Y ] is given by Eq.
Revised: [ A ] = (cid:80) s Y \ a Q [ S Y ] and Q [ S Y ] is given by Eq.

Original: This implies Q [ S ] could be written in the form of:
Revised: This implies Q [ S ] could be written in the form of:

Original: [Equation]
Revised: [Equation]

Original: For any variable V ∈ A , let A ≺ V denote variables in A that precedes V following the topological ordering ≺ in G .
Revised: For any variable V ∈ A , let A ≺ V denote variables in A that precedes V following the topological ordering ≺ in G .

Original: (Tian, 2002, Lemma 11) shows that
Revised: (Tian, 2002, Lemma 11) shows that

Original: [Equation]
Revised: [Equation]

Original: Since all variables in A \{ Y } precedes Y according to the topological ordering ≺ , we have Y ̸∈ A ≺ V for every V ∈
Revised: Since all variables in A \{ Y } precedes Y according to the topological ordering ≺ , we have Y ̸∈ A ≺ V for every V ∈

Original: A \ { Y } .
Revised: A \ { Y } .

Original: This implies Q [ T ′ ] could be written as
Revised: This implies Q [ T ′ ] could be written as

Original: [Equation]
Revised: [Equation]

Original: The last step follows from (Tian, 2002, Lemma 10).
Revised: The last step follows from (Tian, 2002, Lemma 10).

Original: Replacing Q [ A ] with Eq.
Revised: Replacing Q [ A ] with Eq.

Original: (32) we obtain
Revised: (32) we obtain

Original: Q [
Revised: Q [

Original: [Equation]
Revised: [Equation]

Original: where g ′′ is a series of arithmetic operations of P ( O ) .
Revised: where g ′′ is a series of arithmetic operations of P ( O ) .

Original: Eq.
Revised: Eq.

Original: (38) holds since g ( P ( O )) is not a function of reward signal Y .
Revised: (38) holds since g ( P ( O )) is not a function of reward signal Y .

Original: This proves Eq.
Revised: This proves Eq.

Original: (31) for the base case m = 1 .
Revised: (31) for the base case m = 1 .

Original: Induction case m = k
Revised: Induction case m = k

Original: + 1 .
Revised: + 1 .

Original: Suppose the result holds for m ≤ k where k ≥ 1 and consider the case m = k
Revised: Suppose the result holds for m ≤ k where k ≥ 1 and consider the case m = k

Original: + 1 .
Revised: + 1 .

Original: At Step 9, Q [ A ] = (cid:80) t \ a Q [ T ] .
Revised: At Step 9, Q [ A ] = (cid:80) t \ a Q [ T ] .

Original: By the induction assumption, Q [ T ] could be written as Eq.
Revised: By the induction assumption, Q [ T ] could be written as Eq.

Original: This implies that Q [ S ] could be written in the form of Eq.
Revised: This implies that Q [ S ] could be written in the form of Eq.

Original: Again, following the derivation in the base case m = 1 , we prove that Eq.
Revised: Again, following the derivation in the base case m = 1 , we prove that Eq.

Original: (31) for the induction case m = k
Revised: (31) for the induction case m = k

Original: + 1 .
Revised: + 1 .

Original: Let g i
Revised: Let g i

Original: ( P ( O )) denote the identification formula of Q
Revised: ( P ( O )) denote the identification formula of Q

Original: [ D i ] for every c-component D i in G
Revised: [ D i ] for every c-component D i in G

Original: [ D ] .
Revised: [ D ] .

Original: Eqs.
Revised: Eqs.

Original: (27) to (29) together imply that for any policy π ∼ S ,
Revised: (27) to (29) together imply that for any policy π ∼ S ,

Original: [Equation]
Revised: [Equation]

Original: Simplifying the above equation gives
Revised: Simplifying the above equation gives

Original: [Equation]
Revised: [Equation]

Original: where S ′ is a certain subset of X ∗ ∪ Z ∗ ; and ρ π is a function of policy π and the observational distribution P
Revised: where S ′ is a certain subset of X ∗ ∪ Z ∗ ; and ρ π is a function of policy π and the observational distribution P

Original: ( O ) .
Revised: ( O ) .

Original: Note that it is possible that T =
Revised: Note that it is possible that T =

Original: ( X ∗ ∪ Z ∗ ) \ S ′ ̸
Revised: ( X ∗ ∪ Z ∗ ) \ S ′ ̸

Original: = ∅ .
Revised: = ∅ .

Original: In that case, one could fix
Revised: In that case, one could fix

Original: T to an arbitrary constant t by multiplying the indicator function 1 { T = t } .
Revised: T to an arbitrary constant t by multiplying the indicator function 1 { T = t } .

Original: Lemma 2.
Revised: Lemma 2.

Original: Given a causal diagram G , for policy scopes S ′ ⊆ S , S ′ is identifiable from distribution
Revised: Given a causal diagram G , for policy scopes S ′ ⊆ S , S ′ is identifiable from distribution

Original: P ( O , Y ) in G if S is identifiable from P ( O , Y ) in G .
Revised: P ( O , Y ) in G if S is identifiable from P ( O , Y ) in G .

Original: Proof.
Revised: Proof.

Original: If a scope S is identifiable, then the expected rewards E [ Y | do ( π )] of all policies π with scope S are identifiable from P ( O , Y ) in G .
Revised: If a scope S is identifiable, then the expected rewards E [ Y | do ( π )] of all policies π with scope S are identifiable from P ( O , Y ) in G .

Original: This subsumes all policies associated with the subscope
Revised: This subsumes all policies associated with the subscope

Original: S ′ ⊆ S .
Revised: S ′ ⊆ S .

Original: Therefore, S ′ is identifiable from P ( O , Y ) in G .
Revised: Therefore, S ′ is identifiable from P ( O , Y ) in G .

Original: Theorem 3.
Revised: Theorem 3.

Original: For a causal diagram G and a reward Y , L IST I D S COPE ( G , Y, ∅ , (cid:83) ni =1 PA ∗ i ) enumerates a subset S ∗ ⊆ S
Revised: For a causal diagram G and a reward Y , L IST I D S COPE ( G , Y, ∅ , (cid:83) ni =1 PA ∗ i ) enumerates a subset S ∗ ⊆ S

Original: so that for any π ∼ S , there is π ∗ ∼ S ∗ where E [ Y | do ( π )]
Revised: so that for any π ∼ S , there is π ∗ ∼ S ∗ where E [ Y | do ( π )]

Original: =
Revised: =

Original: E [ Y | do ( π ∗ )] .
Revised: E [ Y | do ( π ∗ )] .

Original: Proof.
Revised: Proof.

Original: The recursive calls at Steps 8 and 9 guarantee that L IST I D S COPE generates every identifiable scope S in S ∗ exactly once where
Revised: The recursive calls at Steps 8 and 9 guarantee that L IST I D S COPE generates every identifiable scope S in S ∗ exactly once where

Original: [Equation]
Revised: [Equation]

Original: The pruning criteria in Steps 3 and 6 only remove branches containing exclusively only identifiable (or non-identifiable) scopes (Lem. 2).
Revised: The pruning criteria in Steps 3 and 6 only remove branches containing exclusively only identifiable (or non-identifiable) scopes (Lem. 2).

Original: In that case, it is sufficient to discard all such subscopes (non-identifiable) or only return the root scope S r of the corresponding branch (identifiable).
Revised: In that case, it is sufficient to discard all such subscopes (non-identifiable) or only return the root scope S r of the corresponding branch (identifiable).

Original: What remains is to show the following claim:
Revised: What remains is to show the following claim:

Original: Claim 2.
Revised: Claim 2.

Original: For any scope S ∈ S , there exists S ∗ ∈ S ∗ such that S ∗ perfectly mimics S : that is, for any π ∼ S , one could find π ∗ ∼ S ∗ so that E [ Y | do ( π )]
Revised: For any scope S ∈ S , there exists S ∗ ∈ S ∗ such that S ∗ perfectly mimics S : that is, for any π ∼ S , one could find π ∗ ∼ S ∗ so that E [ Y | do ( π )]

Original: =
Revised: =

Original: E [ Y | do ( π ∗ )] .
Revised: E [ Y | do ( π ∗ )] .

Original: Input : G , Expert demonstrations τ E 1: Apply Thm. 1 (or Thm. 2) to obtain formulas for the expert’s ρ ( x ∗ , z ∗ ) and imitator’s occupancy measure ρ π j ( x ∗ , z ∗ ) 2: Let µ ( i )
Revised: Input : G , Expert demonstrations τ E 1: Apply Thm. 1 (or Thm. 2) to obtain formulas for the expert’s ρ ( x ∗ , z ∗ ) and imitator’s occupancy measure ρ π j ( x ∗ , z ∗ ) 2: Let µ ( i )

Original: = (cid:80) x ∗ , z ∗ ϕ
Revised: = (cid:80) x ∗ , z ∗ ϕ

Original: ( i ) ( x ∗ , z ∗ ) ρ ( x ∗ , z ∗ ) and let µ π ( i ) =
Revised: ( i ) ( x ∗ , z ∗ ) ρ ( x ∗ , z ∗ ) and let µ π ( i ) =

Original: (cid:80) x ∗ , z ∗ ϕ
Revised: (cid:80) x ∗ , z ∗ ϕ

Original: ( i ) ( x ∗ , z ∗ ) ρ π
Revised: ( i ) ( x ∗ , z ∗ ) ρ π

Original: ( x ∗ , z ∗ ) 3: Let (cid:101) G
Revised: ( x ∗ , z ∗ ) 3: Let (cid:101) G

Original: ( i, µ π ) =
Revised: ( i, µ π ) =

Original: ((ˆ µ ( i ) − µ π ( i ))
Revised: ((ˆ µ ( i ) − µ π ( i ))

Original: − 2)
Revised: − 2)

Original: /4: Let β = (cid:18) 1 + (cid:113) 2ln
Revised: /4: Let β = (cid:18) 1 + (cid:113) 2ln

Original: ( k ) J (cid:19) −5: Initialize w
Revised: ( k ) J (cid:19) −5: Initialize w

Original: ( i ) =
Revised: ( i ) =

Original: 1 for i
Revised: 1 for i

Original: .., k 6: for iteration j = 0 , 1 ,
Revised: .., k 6: for iteration j = 0 , 1 ,

Original: J do 7:
Revised: J do 7:

Original: Set w j ( i ) =
Revised: Set w j ( i ) =

Original: w j ( i ) (cid:80) i w j ( i ) for i = 1 , ..., k 8:
Revised: w j ( i ) (cid:80) i w j ( i ) for i = 1 , ..., k 8:

Original: Compute the policy ˆ π j by arg min π w ⊤ (cid:101) Gπ , where w :=
Revised: Compute the policy ˆ π j by arg min π w ⊤ (cid:101) Gπ , where w :=

Original: w j 9:
Revised: w j 9:

Original: Compute ˆ µ j = µ ˆ π j 10: w
Revised: Compute ˆ µ j = µ ˆ π j 10: w

Original: 1 ( i
Revised: 1 ( i

Original: =
Revised: =

Original: ( i
Revised: ( i

Original: (ln(
Revised: (ln(

Original: ) · (cid:101) G ( i, µ j )) for i = 1 , ..., k 11: end for 12: Output: probability
Revised: ) · (cid:101) G ( i, µ j )) for i = 1 , ..., k 11: end for 12: Output: probability

Original: J choosing ˆ π j ,
Revised: J choosing ˆ π j ,

Original: , J } j = 0 , 1 , 2 , . . .
Revised: , J } j = 0 , 1 , 2 , . . .

Original: do 3:
Revised: do 3:

Original: Collect trajectories from distributions ρ ( x ∗ , z ∗ ) and ρ π j ( x ∗ , z ∗ ) 4:
Revised: Collect trajectories from distributions ρ ( x ∗ , z ∗ ) and ρ π j ( x ∗ , z ∗ ) 4:

Original: Update the parameters w of discriminator D j with gradient ˆ E [ ∇ w log( D j ( x ∗ , z ∗ ))]
Revised: Update the parameters w of discriminator D j with gradient ˆ E [ ∇ w log( D j ( x ∗ , z ∗ ))]

Original: + ˆ E π j [ ∇ w log(1 − D j
Revised: + ˆ E π j [ ∇ w log(1 − D j

Original: ( x ∗ , z ∗ ))] 5:
Revised: ( x ∗ , z ∗ ))] 5:

Original: Update the policy π j = arg min π E π [log(1 − D ( x ∗ , z ∗ ))] with policy optimization for DTR 6: end for
Revised: Update the policy π j = arg min π E π [log(1 − D ( x ∗ , z ∗ ))] with policy optimization for DTR 6: end for

Original: Fix any S ∈ S \ S ∗ .
Revised: Fix any S ∈ S \ S ∗ .

Original: Let X S =
Revised: Let X S =

Original: An ( Y ) G S ∩ X and let Z S
Revised: An ( Y ) G S ∩ X and let Z S

Original: =
Revised: =

Original: An ( Y ) G S \ X .
Revised: An ( Y ) G S \ X .

Original: We construct a scope S ∗ = {⟨ X i , Z ∗ i ⟩} ni =1 where Z ∗ i = PA ∗ i ∩ (cid:0)(cid:83) X ∈ X S pa ( X ) G S (cid:1) .
Revised: We construct a scope S ∗ = {⟨ X i , Z ∗ i ⟩} ni =1 where Z ∗ i = PA ∗ i ∩ (cid:0)(cid:83) X ∈ X S pa ( X ) G S (cid:1) .

Original: Similarly, define X S ∗ =
Revised: Similarly, define X S ∗ =

Original: An ( Y ) G S∗
Revised: An ( Y ) G S∗

Original: ∩ X and Z S ∗ =
Revised: ∩ X and Z S ∗ =

Original: An ( Y ) G S∗ \ X .
Revised: An ( Y ) G S∗ \ X .

Original: We will next show that X S
Revised: We will next show that X S

Original: =
Revised: =

Original: X S ∗ and Z S =
Revised: X S ∗ and Z S =

Original: Z S ∗ .
Revised: Z S ∗ .

Original: Suppose that X S ⊂ X S ∗ .
Revised: Suppose that X S ⊂ X S ∗ .

Original: Let X be an action in the set difference X S ∗ \ X S .
Revised: Let X be an action in the set difference X S ∗ \ X S .

Original: Then by the construction of scope S ∗ , there must exist a directed path from X to Y in G S ∗ via a node in Pa ( X ) G S for some X ∈ X S .
Revised: Then by the construction of scope S ∗ , there must exist a directed path from X to Y in G S ∗ via a node in Pa ( X ) G S for some X ∈ X S .

Original: However, such a path must also exist in the diagram G S .
Revised: However, such a path must also exist in the diagram G S .

Original: That is, X ∈ X S , which is a contradiction.
Revised: That is, X ∈ X S , which is a contradiction.

Original: Since G S ∗ does not add any new ancestor of Y , we also have Z S =
Revised: Since G S ∗ does not add any new ancestor of Y , we also have Z S =

Original: Z S ∗ .
Revised: Z S ∗ .

Original: It follows immediately from the factorization in (Tian, 2008, Eq. 15) that S ∗ perfectly mimics S .
Revised: It follows immediately from the factorization in (Tian, 2008, Eq. 15) that S ∗ perfectly mimics S .

Original: Moreover, since S is identifiable, the identification condition in (Tian, 2008, Thm. 1) (which is later shown to be complete in (Correa & Bareinboim, 2019)) implies that Q [ Z S ] must be identifiable from
Revised: Moreover, since S is identifiable, the identification condition in (Tian, 2008, Thm. 1) (which is later shown to be complete in (Correa & Bareinboim, 2019)) implies that Q [ Z S ] must be identifiable from

Original: P ( O , Y ) in G .
Revised: P ( O , Y ) in G .

Original: Since Z S =
Revised: Since Z S =

Original: Z S ∗ , applying the identification condition in (Tian, 2008, Thm. 1) again shows that S ∗ is identifiable from P ( O , Y ) in G .
Revised: Z S ∗ , applying the identification condition in (Tian, 2008, Thm. 1) again shows that S ∗ is identifiable from P ( O , Y ) in G .

Original: That is, S ∗ ∈ S ∗ , which completes the proof.
Revised: That is, S ∗ ∈ S ∗ , which completes the proof.

Original: C C AUSAL MWAL AND C AUSAL GAIL
Revised: C C AUSAL MWAL AND C AUSAL GAIL

Original: In this section, we provide details of causal MWAL and causal GAIL, including pseudo-code and practical considerations in their implementations.
Revised: In this section, we provide details of causal MWAL and causal GAIL, including pseudo-code and practical considerations in their implementations.

Original: Causal MWAL Similar to (Syed & Schapire, 2008), we are interested in learning a mixed policy π compatible with the scope S .
Revised: Causal MWAL Similar to (Syed & Schapire, 2008), we are interested in learning a mixed policy π compatible with the scope S .

Original: Particularly, a mixed policy π ∗ is a probability distribution over deterministic policies compatible with scope S .
Revised: Particularly, a mixed policy π ∗ is a probability distribution over deterministic policies compatible with scope S .

Original: Details of our Causal MWAL are described in Alg. 3.
Revised: Details of our Causal MWAL are described in Alg. 3.

Original: Step 8 amounts to finding the optimal policy in an SCM with a known expected E [ Y | x ∗ , z ∗ ] .
Revised: Step 8 amounts to finding the optimal policy in an SCM with a known expected E [ Y | x ∗ , z ∗ ] .

Original: There are a huge array of techniques available for this, such as dynamic programming or policy gradient.
Revised: There are a huge array of techniques available for this, such as dynamic programming or policy gradient.

Original: Step 9 is the same as computing the feature expectations of a given policy.
Revised: Step 9 is the same as computing the feature expectations of a given policy.

Original: These can be computed exactly by solving k systems of linear equations, or they can be approximated using iterative techniques.
Revised: These can be computed exactly by solving k systems of linear equations, or they can be approximated using iterative techniques.

Original: The algorithm is essentially the MW algorithm (Freund & Schapire, 1999), applied to a game matrix (cid:101) G defined in Step 3.
Revised: The algorithm is essentially the MW algorithm (Freund & Schapire, 1999), applied to a game matrix (cid:101) G defined in Step 3.

Original: Compared to the original game matrix G defined in
Revised: Compared to the original game matrix G defined in

Original: Prop. 1, (cid:101) G utilizes the estimates ˆ µ of the expert’s feature expectations µ from the observational data, rather than requiring that they be computed exactly.
Revised: Prop. 1, (cid:101) G utilizes the estimates ˆ µ of the expert’s feature expectations µ from the observational data, rather than requiring that they be computed exactly.

Original: Following the strategy in (Syed & Schapire, 2008), we estimate the feature expectations µ using its empirical mean from demonstrations.
Revised: Following the strategy in (Syed & Schapire, 2008), we estimate the feature expectations µ using its empirical mean from demonstrations.

Original: Causal GAIL When the reward is non-linear, given expert demonstrations and the underlying graph G , we can formalize the optimization problem based on the canonical equation Eq.
Revised: Causal GAIL When the reward is non-linear, given expert demonstrations and the underlying graph G , we can formalize the optimization problem based on the canonical equation Eq.

Original: (4) and
Revised: (4) and

Original: Prop. 2.
Revised: Prop. 2.

Original: More specifically, when we utilize a regularizer ψ ( r ) similar to (Ho & Ermon, 2016), the convex conjugate function ψ ∗ in Eq.
Revised: More specifically, when we utilize a regularizer ψ ( r ) similar to (Ho & Ermon, 2016), the convex conjugate function ψ ∗ in Eq.

Original: (7) is given by:
Revised: (7) is given by:

Original: [Equation]
Revised: [Equation]

Original: where function D ∈ Ω X ∗ ×
Revised: where function D ∈ Ω X ∗ ×

Original: Ω Z ∗ (cid:55)→ (0 , 1) is a discriminator classifier (e.g, a neural network).
Revised: Ω Z ∗ (cid:55)→ (0 , 1) is a discriminator classifier (e.g, a neural network).

Original: The above optimization problem is in the form of two neural networks competing against each other in a zero-sum game.
Revised: The above optimization problem is in the form of two neural networks competing against each other in a zero-sum game.

Original: Details of our proposed Causal GAIL is summarized in Alg. 4.
Revised: Details of our proposed Causal GAIL is summarized in Alg. 4.

Original: Step 4 optimizes parameters of the discriminator D using standard back-propagation.
Revised: Step 4 optimizes parameters of the discriminator D using standard back-propagation.

Original: Step 5 computes a policy that minimizes a fixed reward log(1 − D ) .
Revised: Step 5 computes a policy that minimizes a fixed reward log(1 − D ) .

Original: For discrete domains, transition probabilities P ( z i | x ∗ <
Revised: For discrete domains, transition probabilities P ( z i | x ∗ <

Original: i , z ∗ <i ) are estimated using empirical means; no value function approximation is used.
Revised: i , z ∗ <i ) are estimated using empirical means; no value function approximation is used.

Original: The optimal policy is then computed via standard dynamic programming (Howard, 1960; Puterman, 1994).
Revised: The optimal policy is then computed via standard dynamic programming (Howard, 1960; Puterman, 1994).

Original: For continuous domains, the optimal policy π ∗ is computed using policy gradient (Sutton et al., 1999).
Revised: For continuous domains, the optimal policy π ∗ is computed using policy gradient (Sutton et al., 1999).

Original: We note that effective actor-critic algorithms exist for optimizing policies in SCMs via value function approximation (Dawid & Didelez, 2010; Murphy, 2003; Zhang & van der Schaar, 2020).
Revised: We note that effective actor-critic algorithms exist for optimizing policies in SCMs via value function approximation (Dawid & Didelez, 2010; Murphy, 2003; Zhang & van der Schaar, 2020).

Original: D D ETAILS ON THE E XPERIMENTAL S ETUP
Revised: D D ETAILS ON THE E XPERIMENTAL S ETUP

Original: In this section, we describe details of the experimental setup and processing of datasets.
Revised: In this section, we describe details of the experimental setup and processing of datasets.

Original: We also provide in Appendix D.1 additional experiments.
Revised: We also provide in Appendix D.1 additional experiments.

Original: For all experiments, we evaluate 4 separate imitation learning strategies summarized as follows:
Revised: For all experiments, we evaluate 4 separate imitation learning strategies summarized as follows:

Original: BC : standard behavior cloning approach that learns a series of policy mappings by utilizing all observed covariates preceding every action X i ∈ X . 2.
Revised: BC : standard behavior cloning approach that learns a series of policy mappings by utilizing all observed covariates preceding every action X i ∈ X . 2.

Original: IRL : standard inverse reinforcement learning approach utilizes all the available covariates preceding every action X i ∈
Revised: IRL : standard inverse reinforcement learning approach utilizes all the available covariates preceding every action X i ∈

Original: X up to each time step.
Revised: X up to each time step.

Original: We will apply the MWAL algorithm (Syed & Schapire, 2008) when the expected reward E
Revised: We will apply the MWAL algorithm (Syed & Schapire, 2008) when the expected reward E

Original: [ Y | O ] is a linear combination of feature functions.
Revised: [ Y | O ] is a linear combination of feature functions.

Original: Otherwise, the GAIL algorithm (Ho & Ermon, 2016) is used to obtain an imitating policy due to the expressive power of neural networks.
Revised: Otherwise, the GAIL algorithm (Ho & Ermon, 2016) is used to obtain an imitating policy due to the expressive power of neural networks.

Original: Causal-BC : the causal behavior cloning approach Zhang et al.
Revised: Causal-BC : the causal behavior cloning approach Zhang et al.

Original: (2020); Kumor et al.
Revised: (2020); Kumor et al.

Original: (2021) first selects a set of covariates Z i for every action X i ∈
Revised: (2021) first selects a set of covariates Z i for every action X i ∈

Original: X following the sequential π -backdoor criterion (Def. 2).
Revised: X following the sequential π -backdoor criterion (Def. 2).

Original: It then learns an imitating policy π with scope S
Revised: It then learns an imitating policy π with scope S

Original: = {⟨ X i , Z i ⟩} ni =1 using standard BC algorithms.
Revised: = {⟨ X i , Z i ⟩} ni =1 using standard BC algorithms.

Original: Causal-IRL :
Revised: Causal-IRL :

Original: Our proposed causal inverse reinforcement learning approach obtains an imitating policy by solving the canonical causal IRL equation in Eq.
Revised: Our proposed causal inverse reinforcement learning approach obtains an imitating policy by solving the canonical causal IRL equation in Eq.

Original: We will utilize the Causal MWAL algorithm (Prop. 1) when the expected reward E [ Y | X ∗ , Z ∗ ] is a linear combination of feature functions.
Revised: We will utilize the Causal MWAL algorithm (Prop. 1) when the expected reward E [ Y | X ∗ , Z ∗ ] is a linear combination of feature functions.

Original: Otherwise, we resort to the Causal GAIL algorithm (Prop. 2) due to the expressive power of neural networks.
Revised: Otherwise, we resort to the Causal GAIL algorithm (Prop. 2) due to the expressive power of neural networks.

Original: For discrete domains (Experiments 1, 5, 6 and 7), we estimate transition distributions using empirical means and solve for an optimal policy in the RL step using dynamic programming.
Revised: For discrete domains (Experiments 1, 5, 6 and 7), we estimate transition distributions using empirical means and solve for an optimal policy in the RL step using dynamic programming.

Original: For an environment with continuous states and actions space (Experiments 2, 3, 4 and 8), we solve for the optimal policy using stochastic policy gradient (Sutton et al., 1999).
Revised: For an environment with continuous states and actions space (Experiments 2, 3, 4 and 8), we solve for the optimal policy using stochastic policy gradient (Sutton et al., 1999).

Original: Expected rewards of policies are estimated by a
Revised: Expected rewards of policies are estimated by a

Original: Monte-Carlo method computing empirical means over the agent’s trajectories.
Revised: Monte-Carlo method computing empirical means over the agent’s trajectories.

Original: Effective function
Revised: Effective function

Original: [Table]
Revised: [Table]

Original: approximation methods also exist to evaluate the value function of policies in high-dimensional domains (Tsitsiklis & Van Roy, 1997; Murphy, 2005).
Revised: approximation methods also exist to evaluate the value function of policies in high-dimensional domains (Tsitsiklis & Van Roy, 1997; Murphy, 2005).

Original: Each experiment is repeated for 100 times; we measure the expected reward E
Revised: Each experiment is repeated for 100 times; we measure the expected reward E

Original: [ Y | do ( π )] of all algorithms averaging over 100 repetitions.
Revised: [ Y | do ( π )] of all algorithms averaging over 100 repetitions.

Original: All experiments were performed on Intel Cascade Lake processors with 30 vCPUs and 120 GB memory in Ubuntu 18.04, implemented in Python.
Revised: All experiments were performed on Intel Cascade Lake processors with 30 vCPUs and 120 GB memory in Ubuntu 18.04, implemented in Python.

Original: We will release the source code with the camera-ready version of the paper if the manuscript is accepted.
Revised: We will release the source code with the camera-ready version of the paper if the manuscript is accepted.

Original: Experiment 1: Backdoor We study the problem of imitation learning in an SCM compatible with the causal diagram in Fig. 1c.
Revised: Experiment 1: Backdoor We study the problem of imitation learning in an SCM compatible with the causal diagram in Fig. 1c.

Original: X 1 , X 2 , Z 1 , Z 2 are binary variables in { 0 , 1 } .
Revised: X 1 , X 2 , Z 1 , Z 2 are binary variables in { 0 , 1 } .

Original: The reward signal Y is determined by a non-linear function.
Revised: The reward signal Y is determined by a non-linear function.

Original: Detailed parametrization of this SCM is provided as follows:
Revised: Detailed parametrization of this SCM is provided as follows:

Original: [Equation]
Revised: [Equation]

Original: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , Z 1 ⟩ , ⟨ X 2 , { Z 1 , X 1 , Z 2 }⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , Z 1 ⟩ , ⟨ X 2 , { Z 1 , X 1 , Z 2 }⟩} , which we denote by π bc and π irl respectively.

Original: Causal-BC utilizes the sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} and learns an imitating policy π c-bc .
Revised: Causal-BC utilizes the sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 2 }⟩} and learns an imitating policy π c-bc .

Original: Our proposed Causal-IRL learner obtains an imitating policy π c-irl associated with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} , which satisfies the minimal sequential π -backdoor in Def. 3.
Revised: Our proposed Causal-IRL learner obtains an imitating policy π c-irl associated with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} , which satisfies the minimal sequential π -backdoor in Def. 3.

Original: Since the expected reward E [ Y | X 1 , X 2 , Z 2 ] is nonlinear, we solve for π c-irl using the Causal GAIL algorithm described in Prop. 2.
Revised: Since the expected reward E [ Y | X 1 , X 2 , Z 2 ] is nonlinear, we solve for π c-irl using the Causal GAIL algorithm described in Prop. 2.

Original: Reward augmentation (RA) is performed to incorporates the parametric knowledge that E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to values of X 1 , X 2 (Li et al., 2017).
Revised: Reward augmentation (RA) is performed to incorporates the parametric knowledge that E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to values of X 1 , X 2 (Li et al., 2017).

Original: This is done by adding an additional regularization function in Eq.
Revised: This is done by adding an additional regularization function in Eq.

Original: (7) to encourage the Causal-IRL agent to take higher values of actions X 1 , X 2 .
Revised: (7) to encourage the Causal-IRL agent to take higher values of actions X 1 , X 2 .

Original: Simulation results are shown in Fig. 3a.
Revised: Simulation results are shown in Fig. 3a.

Original: Values of expected rewards for all imitation strategies are provided in Table 1.
Revised: Values of expected rewards for all imitation strategies are provided in Table 1.

Original: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting additional parametric knowledge about the expected reward E [ Y | X 1 , X 2 , Z 2 ] ; Causal-BC can learn a policy that mimics the expert’s performance.
Revised: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting additional parametric knowledge about the expected reward E [ Y | X 1 , X 2 , Z 2 ] ; Causal-BC can learn a policy that mimics the expert’s performance.

Original: As expected, BC and IRL fail to obtain an imitating policy performing as well as the expert’s policy.
Revised: As expected, BC and IRL fail to obtain an imitating policy performing as well as the expert’s policy.

Original: Experiment 2: Highway Driving This experiment evaluates our proposed Causal-IRL on a real-world HighD dataset Krajewski et al.
Revised: Experiment 2: Highway Driving This experiment evaluates our proposed Causal-IRL on a real-world HighD dataset Krajewski et al.

Original: (2018) which consists of humans’ natural driving trajectories on highway.
Revised: (2018) which consists of humans’ natural driving trajectories on highway.

Original: We study the problem of imitation learning in an SCM compatible with the causal diagram in Fig. 4; where reward signal Y is determined by a non-linear function.
Revised: We study the problem of imitation learning in an SCM compatible with the causal diagram in Fig. 4; where reward signal Y is determined by a non-linear function.

Original: Detailed construction of this SCM is described as follows
Revised: Detailed construction of this SCM is described as follows

Original: Z 1 is both longitudinal and lateral historical accelerations of the ego vehicle two steps ago.
Revised: Z 1 is both longitudinal and lateral historical accelerations of the ego vehicle two steps ago.

Original: X 1 is the accelerations of the ego vehicle at the previous step.
Revised: X 1 is the accelerations of the ego vehicle at the previous step.

Original: Z 2 is the velocity of the preceding vehicle.
Revised: Z 2 is the velocity of the preceding vehicle.

Original: X 2 is the velocity of the ego vehicle.
Revised: X 2 is the velocity of the ego vehicle.

Original: We bootstrap samples of Z 1 , X 1 , Z 2 , X 2 from the HighD dataset. 6.
Revised: We bootstrap samples of Z 1 , X 1 , Z 2 , X 2 from the HighD dataset. 6.

Original: L was constructed such that X 2 satisfies the relation L
Revised: L was constructed such that X 2 satisfies the relation L

Original: = I { X 2 − Z 2 > − 0 .
Revised: = I { X 2 − Z 2 > − 0 .

Original: This leads to a distribution P ( L = 1) = 0 .
Revised: This leads to a distribution P ( L = 1) = 0 .

Original: Values
Revised: Values

Original: of
Revised: of

Original: Y
Revised: Y

Original: , W are decided
Revised: , W are decided

Original: by the
Revised: by the

Original: following causal mechanisms
Revised: following causal mechanisms

Original: [Equation]
Revised: [Equation]

Original: We also perform the standard causal discovery algorithm (Verma & Pearl, 1988; Spirtes et al., 2000) to the HighD data over variables Z 1 ,
Revised: We also perform the standard causal discovery algorithm (Verma & Pearl, 1988; Spirtes et al., 2000) to the HighD data over variables Z 1 ,

Original: X 1 , Z 2 , X 2 .
Revised: X 1 , Z 2 , X 2 .

Original: The analysis reveals the independence relationship ( Z 2 ⊥⊥ X 1 , Z 1 ) .
Revised: The analysis reveals the independence relationship ( Z 2 ⊥⊥ X 1 , Z 1 ) .

Original: In practice, it is common that the leading vehicle’s velocity will not be affected by the follower’s past accelerations (Treiber et al., 2000; Higgs et al., 2011).
Revised: In practice, it is common that the leading vehicle’s velocity will not be affected by the follower’s past accelerations (Treiber et al., 2000; Higgs et al., 2011).

Original: We do not impose any constraints among variables Z 1 , X 1 , X 2 belonging to the ego vehicle.
Revised: We do not impose any constraints among variables Z 1 , X 1 , X 2 belonging to the ego vehicle.

Original: That is, the above construction procedure is compatible with the causal diagram in Fig.
Revised: That is, the above construction procedure is compatible with the causal diagram in Fig.

Original: X
Revised: X

Original: Z
Revised: Z

Original: X
Revised: X

Original: Z
Revised: Z

Original: L
Revised: L

Original: W
Revised: W

Original: Y
Revised: Y

Original: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 1 , X 1 , W, Z 2 }⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 1 , X 1 , W, Z 2 }⟩} , which we denote by π bc and π irl respectively.

Original: On the other hand, Causal-BC and
Revised: On the other hand, Causal-BC and

Original: Causal-IRL utilizes scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} which satisfies the minimal sequential π -backdoor (Def. 3).
Revised: Causal-IRL utilizes scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} which satisfies the minimal sequential π -backdoor (Def. 3).

Original: Causal-BC learns a policy π c-bc via behavior cloning.
Revised: Causal-BC learns a policy π c-bc via behavior cloning.

Original: Meanwhile, for any policy π with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} , the expected reward E [ Y |
Revised: Meanwhile, for any policy π with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , { Z 2 }⟩} , the expected reward E [ Y |

Original: [Equation]
Revised: [Equation]

Original: ( π )] decomposes as, following Thm. 1,
Revised: ( π )] decomposes as, following Thm. 1,

Original: [Equation]
Revised: [Equation]

Original: Using the above identification formula, Causal-IRL translates the imitation problem into the canonical equation of Eq.
Revised: Using the above identification formula, Causal-IRL translates the imitation problem into the canonical equation of Eq.

Original: (4) and obtains an imitating policy
Revised: (4) and obtains an imitating policy

Original: π c-irl via the Causal GAIL
Revised: π c-irl via the Causal GAIL

Original: algorithm (Prop. 2 and Alg. 4).
Revised: algorithm (Prop. 2 and Alg. 4).

Original: We evaluate Causal-IRL in the HightD dataset provided with additional parametric knowledge about the reward signal.
Revised: We evaluate Causal-IRL in the HightD dataset provided with additional parametric knowledge about the reward signal.

Original: Therefore, Causal-IRL is able to exploit the fact that the expected reward E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to X 2 , Z 2 .
Revised: Therefore, Causal-IRL is able to exploit the fact that the expected reward E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to X 2 , Z 2 .

Original: Reward augmentation (Li et al., 2017) is performed to account for such parametric knowledge in the Causal GAIL algorithm.
Revised: Reward augmentation (Li et al., 2017) is performed to account for such parametric knowledge in the Causal GAIL algorithm.

Original: Particularly, we add an additional regularization function in the optimization problem of Eq.
Revised: Particularly, we add an additional regularization function in the optimization problem of Eq.

Original: (7) to encourage the agent to take higher values of action X 2 and achieve higher values of state Z 2 .
Revised: (7) to encourage the agent to take higher values of action X 2 and achieve higher values of state Z 2 .

Original: We show simulation results in Fig.
Revised: We show simulation results in Fig.

Original: 3b.
Revised: 3b.

Original: Details of expected rewards for all imitation strategies are described in Table 1.
Revised: Details of expected rewards for all imitation strategies are described in Table 1.

Original: We found that Causal-BC learner is able to achieve the expert’s performance.
Revised: We found that Causal-BC learner is able to achieve the expert’s performance.

Original: Causal-IRL is able to dominate the expert and other imitation strategies.
Revised: Causal-IRL is able to dominate the expert and other imitation strategies.

Original: As expected, BC and
Revised: As expected, BC and

Original: IRL perform the worst among all algorithms and fail to obtain an effective imitating policy matching the expert’s behavior policy.
Revised: IRL perform the worst among all algorithms and fail to obtain an effective imitating policy matching the expert’s behavior policy.

Original: Experiment 3: MNIST Digits
Revised: Experiment 3: MNIST Digits

Original: This experiment evaluates our proposed Causal-IRL on the
Revised: This experiment evaluates our proposed Causal-IRL on the

Original: MNIST digit dataset (LeCun, 1998), which consists of images of handwritten digits.
Revised: MNIST digit dataset (LeCun, 1998), which consists of images of handwritten digits.

Original: Again, we study the problem of imitation learning in an SCM compatible with the frontdoor diagram in Fig.
Revised: Again, we study the problem of imitation learning in an SCM compatible with the frontdoor diagram in Fig.

Original: 2a.
Revised: 2a.

Original: However, values of mediator Z are now replaced with an image bootstrapped from MNIST dataset containing digit 0 or 1 .
Revised: However, values of mediator Z are now replaced with an image bootstrapped from MNIST dataset containing digit 0 or 1 .

Original: More specifically, the data-generating mechanisms of this SCM are described as follows:
Revised: More specifically, the data-generating mechanisms of this SCM are described as follows:

Original: [Equation]
Revised: [Equation]

Original: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.

Original: Since there exists no sequential π -backdoor admissible scope in Fig.
Revised: Since there exists no sequential π -backdoor admissible scope in Fig.

Original: 2a, Causal-BC fails and resorts to a policy obtained by standard BC , i.e., π c-bc = π bc .
Revised: 2a, Causal-BC fails and resorts to a policy obtained by standard BC , i.e., π c-bc = π bc .

Original: Finally, Causal-IRL obtains an imitating policy π c-irl with scope {⟨ X, ∅⟩} .
Revised: Finally, Causal-IRL obtains an imitating policy π c-irl with scope {⟨ X, ∅⟩} .

Original: It formulates the canonical equation in Eq.
Revised: It formulates the canonical equation in Eq.

Original: (10) as a two-person zero-sum game, which can be solved by the Causal MWAL algorithm described in Prop. 1.
Revised: (10) as a two-person zero-sum game, which can be solved by the Causal MWAL algorithm described in Prop. 1.

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 3c.
Revised: 3c.

Original: Details of expected rewards for all imitation strategies are shown in Table 1.
Revised: Details of expected rewards for all imitation strategies are shown in Table 1.

Original: We found that Causal-IRL outperforms Causal-BC and BC ; while IRL performs the worst among all algorithms.
Revised: We found that Causal-IRL outperforms Causal-BC and BC ; while IRL performs the worst among all algorithms.

Original: Experiment 4: Infinite MDPUC This experiment evaluates our proposed Causal-IRL on an
Revised: Experiment 4: Infinite MDPUC This experiment evaluates our proposed Causal-IRL on an

Original: MDPUC model with the infinite horizon (Ruan & Di, 2022), which is a Markov decision process explicitly incorporating the existence of unobserved confounders (Zhang & Bareinboim, 2016; 2022).
Revised: MDPUC model with the infinite horizon (Ruan & Di, 2022), which is a Markov decision process explicitly incorporating the existence of unobserved confounders (Zhang & Bareinboim, 2016; 2022).

Original: Fig.
Revised: Fig.

Original: 1d shows a simplified causal diagram of this MDPUC model for i = 1 , 2 , 3 .
Revised: 1d shows a simplified causal diagram of this MDPUC model for i = 1 , 2 , 3 .

Original: A more detailed causal diagram was described in (Ruan & Di, 2022).
Revised: A more detailed causal diagram was described in (Ruan & Di, 2022).

Original: More specifically, at the time step i , the expert takes an action X
Revised: More specifically, at the time step i , the expert takes an action X

Original: i based on the current state Z i , U π and past action X i − 1 .
Revised: i based on the current state Z i , U π and past action X i − 1 .

Original: Covariates U
Revised: Covariates U

Original: ( Z,Y ) i , and U π are unobserved to the imitator, thus summarized as exogenous variables.
Revised: ( Z,Y ) i , and U π are unobserved to the imitator, thus summarized as exogenous variables.

Original: At every time step i , only information that the imitator could utilize is the past observed states and actions { Z 0 , X 0 , Z 1 , X 1 , . . .
Revised: At every time step i , only information that the imitator could utilize is the past observed states and actions { Z 0 , X 0 , Z 1 , X 1 , . . .

Original: , Z i − 1 , X i − 1 } .
Revised: , Z i − 1 , X i − 1 } .

Original: Detailed semantics of the SCM is described as follows.
Revised: Detailed semantics of the SCM is described as follows.

Original: U ( Z,Y )
Revised: U ( Z,Y )

Original: i is the road conditions.
Revised: i is the road conditions.

Original: U π denotes the level of expert driving skills.
Revised: U π denotes the level of expert driving skills.

Original: U ( A ) i is the action of the leading vehicle, which cannot be observed by the imitator or the expert.
Revised: U ( A ) i is the action of the leading vehicle, which cannot be observed by the imitator or the expert.

Original: Z i contains the velocities of the follower vehicle and the leading vehicle, and the gap between them, which is affected by U ( A ) i − 1 , Z i − 1 , X i − 

Revised: Z i contains the velocities of the follower vehicle and the leading vehicle, and the gap between them, which is affected by U ( A ) i − 1 , Z i − 1 , X i −

Original: and U ( Z,Y ) i .
Revised: and U ( Z,Y ) i .

Original: X i is the acceleration or deceleration of the follower vehicle.
Revised: X i is the acceleration or deceleration of the follower vehicle.

Original: Y i is the reward, which takes multiple goals into consideration, e.g., efficiency, comfort, and safety.
Revised: Y i is the reward, which takes multiple goals into consideration, e.g., efficiency, comfort, and safety.

Original: When the agent is involved in a collision, the environment will be stopped, and the agent will receive a negative penalty.
Revised: When the agent is involved in a collision, the environment will be stopped, and the agent will receive a negative penalty.

Original: Both BC and IRL learn a policy associated with the scope S
Revised: Both BC and IRL learn a policy associated with the scope S

Original: = {⟨ X i , Z i ⟩} ∞ i =1 .
Revised: = {⟨ X i , Z i ⟩} ∞ i =1 .

Original: On the other hand,
Revised: On the other hand,

Original: Causal-BC and Causal-IRL utilize scope S ∗ = {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} ∞ i =1 which satisfies the minimal sequential π -backdoor (Def. 3).
Revised: Causal-BC and Causal-IRL utilize scope S ∗ = {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} ∞ i =1 which satisfies the minimal sequential π -backdoor (Def. 3).

Original: Causal-BC learns a stationary policy π c-bc ∼ S ∗ via standard behavioral cloning.
Revised: Causal-BC learns a stationary policy π c-bc ∼ S ∗ via standard behavioral cloning.

Original: Meanwhile, Causal-IRL exploits the Markov property up to time step i , which is,
Revised: Meanwhile, Causal-IRL exploits the Markov property up to time step i , which is,

Original: [Equation]
Revised: [Equation]

Original: By doing so, Causal-IRL translates the imitation problem into the canonical equation of Eq.
Revised: By doing so, Causal-IRL translates the imitation problem into the canonical equation of Eq.

Original: (4) and obtains an imitating policy
Revised: (4) and obtains an imitating policy

Original: π c-irl via the Causal GAIL
Revised: π c-irl via the Causal GAIL

Original: algorithm (Prop. 2 and Alg. 4).
Revised: algorithm (Prop. 2 and Alg. 4).

Original: Fig.
Revised: Fig.

Original: 3d shows the results, where Causal-IRL outperforms Causal-BC , IRL and BC .
Revised: 3d shows the results, where Causal-IRL outperforms Causal-BC , IRL and BC .

Original: D. A DDITIONAL E XPERIMENTS
Revised: D. A DDITIONAL E XPERIMENTS

Original: We also evaluate our imitation approach on various synthetic SCM instances with other forms of reward functions.
Revised: We also evaluate our imitation approach on various synthetic SCM instances with other forms of reward functions.

Original: Overall, we found that simulation results match our findings in the main manuscript.
Revised: Overall, we found that simulation results match our findings in the main manuscript.

Original: Our causal IRL approach consistently dominates state-of-art imitation learning methods across various causal diagrams.
Revised: Our causal IRL approach consistently dominates state-of-art imitation learning methods across various causal diagrams.

Original: Furthermore, it is able to incorporate parametric knowledge about the reward function and achieve effective imitating policies from high-dimensional demonstration data.
Revised: Furthermore, it is able to incorporate parametric knowledge about the reward function and achieve effective imitating policies from high-dimensional demonstration data.

Original: Experiment 5:
Revised: Experiment 5:

Original: Frontdoor We study the problem of imitation learning in an SCM compatible with the frontdoor diagram in Fig.
Revised: Frontdoor We study the problem of imitation learning in an SCM compatible with the frontdoor diagram in Fig.

Original: 2a.
Revised: 2a.

Original: X, Y, Z are binary variables in { 0 , 1 } .
Revised: X, Y, Z are binary variables in { 0 , 1 } .

Original: The reward signal Y is determined by a non-linear function.
Revised: The reward signal Y is determined by a non-linear function.

Original: Detailed parametrization of this SCM is provided as follows:
Revised: Detailed parametrization of this SCM is provided as follows:

Original: [Equation]
Revised: [Equation]

Original: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.

Original: Since there exists no sequential π -backdoor admissible scope in Fig.
Revised: Since there exists no sequential π -backdoor admissible scope in Fig.

Original: 2a, Causal-BC
Revised: 2a, Causal-BC

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (a) Experiment
Revised: Expert (a) Experiment

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (b) Experiment
Revised: Expert (b) Experiment

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (c) Experiment
Revised: Expert (c) Experiment

Original: B C
Revised: B C

Original: I R L C a u s a l B CC a u s a l I R L 0.
Revised: I R L C a u s a l B CC a u s a l I R L 0.

Original: Expert (d) Experiment fails and resorts to a policy obtained by standard BC , i.e., π c-bc = π bc .
Revised: Expert (d) Experiment fails and resorts to a policy obtained by standard BC , i.e., π c-bc = π bc .

Original: Finally, Causal-IRL obtains an imitating policy π c-irl with {⟨ X, ∅⟩} by solving the canonical equation in Eq.
Revised: Finally, Causal-IRL obtains an imitating policy π c-irl with {⟨ X, ∅⟩} by solving the canonical equation in Eq.

Original: It performs reward augmentation (Li et al., 2017) to incorporate the parametric knowledge that E [ Y | X, Z ] is a monotone function with regard to values of X, Z .
Revised: It performs reward augmentation (Li et al., 2017) to incorporate the parametric knowledge that E [ Y | X, Z ] is a monotone function with regard to values of X, Z .

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 5a.
Revised: 5a.

Original: Details of expected rewards for all imitation strategies are provided in Table 1.
Revised: Details of expected rewards for all imitation strategies are provided in Table 1.

Original: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy by exploiting additional parametric knowledge about the expected reward
Revised: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy by exploiting additional parametric knowledge about the expected reward

Original: E
Revised: E

Original: [ Y | X, Z ] ; while other strategies fail to match the suboptimal expert.
Revised: [ Y | X, Z ] ; while other strategies fail to match the suboptimal expert.

Original: Experiment 6: Backdoor with Linear Reward We now consider an SCM instance compatible with the causal diagram in Fig.
Revised: Experiment 6: Backdoor with Linear Reward We now consider an SCM instance compatible with the causal diagram in Fig.

Original: 6 where reward signal Y is determined by a linear function;
Revised: 6 where reward signal Y is determined by a linear function;

Original: X 1 , X 2 , Z 1 , Z 2 are binary variables in { 0 , 1 } .
Revised: X 1 , X 2 , Z 1 , Z 2 are binary variables in { 0 , 1 } .

Original: Detailed parametrization of this SCM is given by:
Revised: Detailed parametrization of this SCM is given by:

Original: [Equation]
Revised: [Equation]

Original: Figure
Revised: Figure

Original: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 1 , X 1 }⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , { Z 1 , X 1 }⟩} , which we denote by π bc and π irl respectively.

Original: Causal-BC utilizes the sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , ∅⟩} and learns an imitating policy π c-bc .
Revised: Causal-BC utilizes the sequential π -backdoor admissible scope {⟨ X 1 , { Z 1 }⟩ , ⟨ X 2 , ∅⟩} and learns an imitating policy π c-bc .

Original: Our proposed Causal-IRL learner obtains an imitating policy π c-irl associated with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , ∅⟩} , which satisfies the minimal sequential π -backdoor in
Revised: Our proposed Causal-IRL learner obtains an imitating policy π c-irl associated with scope {⟨ X 1 , ∅⟩ , ⟨ X 2 , ∅⟩} , which satisfies the minimal sequential π -backdoor in

Original: Def. 3.
Revised: Def. 3.

Original: Since the expected reward E [ Y | X 1 , X 2 ] is a linear function, we solve for π c-irl using the Causal MWAL algorithm described in Prop. 1.
Revised: Since the expected reward E [ Y | X 1 , X 2 ] is a linear function, we solve for π c-irl using the Causal MWAL algorithm described in Prop. 1.

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 5c.
Revised: 5c.

Original: Expected rewards for all imitation strategies are provided in Table 1.
Revised: Expected rewards for all imitation strategies are provided in Table 1.

Original: Our analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting the linearity of the expected reward E [ Y | X 1 , X 2 ] ; Causal-BC is able to learn a policy that mimics the expert’s performance; while BC and IRL perform the worst among all.
Revised: Our analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting the linearity of the expected reward E [ Y | X 1 , X 2 ] ; Causal-BC is able to learn a policy that mimics the expert’s performance; while BC and IRL perform the worst among all.

Original: Experiment 7: Frontdoor with Linear Reward Consider again the frontdoor diagram in Fig.
Revised: Experiment 7: Frontdoor with Linear Reward Consider again the frontdoor diagram in Fig.

Original: 2a with binary variables X, Y, Z ∈ { 0 , 1 } .
Revised: 2a with binary variables X, Y, Z ∈ { 0 , 1 } .

Original: The reward signal Y is now determined by a linear function.
Revised: The reward signal Y is now determined by a linear function.

Original: Detailed parametrization of the underlying SCM is provided as follows:
Revised: Detailed parametrization of the underlying SCM is provided as follows:

Original: [Equation]
Revised: [Equation]

Original: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.
Revised: Both BC and IRL learn a policy associated with the scope {⟨ X, ∅⟩} , which we denote by π bc and π irl respectively.

Original: Causal-BC fails to find a π -backdoor admissible scope and resorts to a standard
Revised: Causal-BC fails to find a π -backdoor admissible scope and resorts to a standard

Original: BC policy π c-bc = π bc .
Revised: BC policy π c-bc = π bc .

Original: Causal-IRL obtains an imitating policy π c-irl with scope {⟨ X, ∅⟩} .
Revised: Causal-IRL obtains an imitating policy π c-irl with scope {⟨ X, ∅⟩} .

Original: It formulates the canonical equation in Eq.
Revised: It formulates the canonical equation in Eq.

Original: (10) as a two-person zero-sum game, which can be solved by the Causal MWAL algorithm described in Prop. 1.
Revised: (10) as a two-person zero-sum game, which can be solved by the Causal MWAL algorithm described in Prop. 1.

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 5c.
Revised: 5c.

Original: Expected rewards for all imitation strategies are provided in Table 1.
Revised: Expected rewards for all imitation strategies are provided in Table 1.

Original: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting the linearity of the expected reward E [ Y | X, Z ] ; while IRL performs the worst among all strategies.
Revised: The analysis reveals that Causal-IRL consistently outperforms the expert’s policy and other imitation strategies by exploiting the linearity of the expected reward E [ Y | X, Z ] ; while IRL performs the worst among all strategies.

Original: Experiment 8: HighD without Reward Augmentation
Revised: Experiment 8: HighD without Reward Augmentation

Original: Finally, we evaluate Causal-IRL in the HightD dataset without utilizing any additional parametric knowledge about the reward signal.
Revised: Finally, we evaluate Causal-IRL in the HightD dataset without utilizing any additional parametric knowledge about the reward signal.

Original: For this experiment, the simulation setup and training process remain the same as in Experiment 2.
Revised: For this experiment, the simulation setup and training process remain the same as in Experiment 2.

Original: The only difference here is that Causal-IRL now does not exploit the fact that the expected reward E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to X 2 , Z 2 .
Revised: The only difference here is that Causal-IRL now does not exploit the fact that the expected reward E [ Y | X 1 , X 2 , Z 2 ] is a monotone function with regard to X 2 , Z 2 .

Original: We show in Fig.
Revised: We show in Fig.

Original: 5d simulation results for this experiment.
Revised: 5d simulation results for this experiment.

Original: Table 1 provides details of expected rewards for all imitation strategies.
Revised: Table 1 provides details of expected rewards for all imitation strategies.

Original: We could see by inspection that the performance of BC , IRL ,
Revised: We could see by inspection that the performance of BC , IRL ,

Original: Causal-BC and the expert remains the same.
Revised: Causal-BC and the expert remains the same.

Original: The performance of Causal-IRL decreases since it is now unable to exploit the monotonicity of the expected reward
Revised: The performance of Causal-IRL decreases since it is now unable to exploit the monotonicity of the expected reward

Original: E [ Y | X 1 , X 2 , Z ] .
Revised: E [ Y | X 1 , X 2 , Z ] .

Original: However, it is still able to achieve the expert’s performance and obtain an effective imitating policy.
Revised: However, it is still able to achieve the expert’s performance and obtain an effective imitating policy.

Original: E R ELATED W ORK AND C ONTRIBUTIONS
Revised: E R ELATED W ORK AND C ONTRIBUTIONS

Original: Imitation Learning (IL) paradigm investigates the problem of an imitator learning how to behave in an environment with an unknown reward function by observing demonstrations from a human expert (Argall et al., 2009; Billard et al., 2008; Hussein et al., 2017; Osa et al., 2018).
Revised: Imitation Learning (IL) paradigm investigates the problem of an imitator learning how to behave in an environment with an unknown reward function by observing demonstrations from a human expert (Argall et al., 2009; Billard et al., 2008; Hussein et al., 2017; Osa et al., 2018).

Original: There are two major learning modalities that implements IL – behavioral cloning (BC)
Revised: There are two major learning modalities that implements IL – behavioral cloning (BC)

Original: (Widrow, 1964; Pomerleau, 1989; Muller et al., 2006; M ¨ ulling et al., 2013; Mahler & Goldberg, 2017) and inverse reinforcement learning (IRL)
Revised: (Widrow, 1964; Pomerleau, 1989; Muller et al., 2006; M ¨ ulling et al., 2013; Mahler & Goldberg, 2017) and inverse reinforcement learning (IRL)

Original: Ng et al.
Revised: Ng et al.

Original: ; Ziebart et al.
Revised: ; Ziebart et al.

Original: ; Ho & Ermon (2016); Fu et al.
Revised: ; Ho & Ermon (2016); Fu et al.

Original: BC methods directly mimic the expert’s behavior policy by learning a mapping from observed states to the expert’s action via supervised learning.
Revised: BC methods directly mimic the expert’s behavior policy by learning a mapping from observed states to the expert’s action via supervised learning.

Original: Alternatively, IRL methods first learn a potential reward function under which the expert’s behavior policy is optimal.
Revised: Alternatively, IRL methods first learn a potential reward function under which the expert’s behavior policy is optimal.

Original: The imitator then obtains a policy by employing standard RL methods to maximize the learned reward function.
Revised: The imitator then obtains a policy by employing standard RL methods to maximize the learned reward function.

Original: Under some common assumptions, both BC and IRL are able to obtain policies that achieve the expert’s performance (Kumor et al., 2021; Swamy et al., 2021).
Revised: Under some common assumptions, both BC and IRL are able to obtain policies that achieve the expert’s performance (Kumor et al., 2021; Swamy et al., 2021).

Original: Moreover, when additional parametric knowledge about the reward function is provided, IRL may produce a policy that outperforms the expert’s in the underlying environment (Syed & Schapire, 2008;
Revised: Moreover, when additional parametric knowledge about the reward function is provided, IRL may produce a policy that outperforms the expert’s in the underlying environment (Syed & Schapire, 2008;

Original: Li et al., 2017; Yu et al., 2020).
Revised: Li et al., 2017; Yu et al., 2020).

Original: Despite the performance guarantees provided by existing imitation methods, both BC and IRL rely on the assumption that the expert’s input observations match those available to the imitator.
Revised: Despite the performance guarantees provided by existing imitation methods, both BC and IRL rely on the assumption that the expert’s input observations match those available to the imitator.

Original: While this assumption may certainly hold in some settings, it is quite stringent in others, where agents may differ in their sensory capabilities, have distinct goals in mind, and evolve their reasoning (e.g., “software”) and attention independently.
Revised: While this assumption may certainly hold in some settings, it is quite stringent in others, where agents may differ in their sensory capabilities, have distinct goals in mind, and evolve their reasoning (e.g., “software”) and attention independently.

Original: When the expert and imitator’s input covariates disagree, the expert’s demonstration could be contaminated with the presence of unobserved confounders (UCs).
Revised: When the expert and imitator’s input covariates disagree, the expert’s demonstration could be contaminated with the presence of unobserved confounders (UCs).

Original: In this case, naively applying standard imitation methods may not necessarily lead to satisfactory performance, even when the expert him or herself behaves optimally (Zhang et al., 2020).
Revised: In this case, naively applying standard imitation methods may not necessarily lead to satisfactory performance, even when the expert him or herself behaves optimally (Zhang et al., 2020).

Original: Causal Inference (CI) addresses the challenge of unobserved confounding bias in the observational data by exploiting causal assumptions about the data-generating mechanisms (commonly through causal graphs and potential outcomes).
Revised: Causal Inference (CI) addresses the challenge of unobserved confounding bias in the observational data by exploiting causal assumptions about the data-generating mechanisms (commonly through causal graphs and potential outcomes).

Original: Several criteria and algorithms have been developed (Pearl, 2000; Spirtes et al., 2000; Bareinboim & Pearl, 2016).
Revised: Several criteria and algorithms have been developed (Pearl, 2000; Spirtes et al., 2000; Bareinboim & Pearl, 2016).

Original: More recently, there exists an emerging line of research under the rubric of causal imitation learning that augments the imitation paradigm to account for environments consisting of arbitrary causal mechanisms and the aforementioned mismatch between expert and imitator’s sensory capabilities (de Haan et al., 2019; Zhang et al., 2020;
Revised: More recently, there exists an emerging line of research under the rubric of causal imitation learning that augments the imitation paradigm to account for environments consisting of arbitrary causal mechanisms and the aforementioned mismatch between expert and imitator’s sensory capabilities (de Haan et al., 2019; Zhang et al., 2020;

Original: Etesami & Geiger, 2020; Kumor et al., 2021).
Revised: Etesami & Geiger, 2020; Kumor et al., 2021).

Original: Closest to our work, Zhang et al.
Revised: Closest to our work, Zhang et al.

Original: (2020); Kumor et al.
Revised: (2020); Kumor et al.

Original: (2021) derive graphical criteria that completely characterize when and how BC could lead to successful imitation even when the agents perceive reality differently.
Revised: (2021) derive graphical criteria that completely characterize when and how BC could lead to successful imitation even when the agents perceive reality differently.

Original: However, most of these causal imitation methods focus on the BC setting, where the performance of the imitator is limited by the expert’s policy.
Revised: However, most of these causal imitation methods focus on the BC setting, where the performance of the imitator is limited by the expert’s policy.

Original: (de Haan et al., 2019) studies the problem of “causal confusion” by assuming that there does not exist any UC.
Revised: (de Haan et al., 2019) studies the problem of “causal confusion” by assuming that there does not exist any UC.

Original: There are several other works that investigate this problem based on the framework of (or variations of)
Revised: There are several other works that investigate this problem based on the framework of (or variations of)

Original: Markov decision processes (MDP) (Puterman, 1994), including contextual MDPs (Tennenholtz et al., 2021), MDPs with temporally correlated noises (Swamy et al., 2022), and MDPs with the presence of UCs (Ruan & Di, 2022).
Revised: Markov decision processes (MDP) (Puterman, 1994), including contextual MDPs (Tennenholtz et al., 2021), MDPs with temporally correlated noises (Swamy et al., 2022), and MDPs with the presence of UCs (Ruan & Di, 2022).

Original: Still, it is unclear how to perform
Revised: Still, it is unclear how to perform

Original: IRL-type training beyond MDP settings, especially when expert policies are non-stationary, the underlying environment is non-Markovian, and UCs are generally present.
Revised: IRL-type training beyond MDP settings, especially when expert policies are non-stationary, the underlying environment is non-Markovian, and UCs are generally present.

Original: Contributions
Revised: Contributions

Original: This paper investigates the settings where the expert’s demonstrations are contaminated with biases induced by UCs.
Revised: This paper investigates the settings where the expert’s demonstrations are contaminated with biases induced by UCs.

Original: When UCs are present, it is possible to construct a simple causal model with binary variables such that existing IRL methods cannot learn an effective imitating policy from demonstrations alone, regardless of how advanced the feature extraction procedures are and what the sample size is.
Revised: When UCs are present, it is possible to construct a simple causal model with binary variables such that existing IRL methods cannot learn an effective imitating policy from demonstrations alone, regardless of how advanced the feature extraction procedures are and what the sample size is.

Original: In a nutshell, our contribution is to empower imitation learning to be robust against confounding bias through knowledge augmentation via a causal lens.
Revised: In a nutshell, our contribution is to empower imitation learning to be robust against confounding bias through knowledge augmentation via a causal lens.

Original: Accordingly, we propose the first causal IRL method that could perform at least as well as the expert while being robust against UCs by exploiting causal relationships encoded in the underlying environment.
Revised: Accordingly, we propose the first causal IRL method that could perform at least as well as the expert while being robust against UCs by exploiting causal relationships encoded in the underlying environment.

Original: The learned
Revised: The learned

Original: IRL policy may significantly outperform the expert’s policy by leveraging additional parametric knowledge about the reward signal.
Revised: IRL policy may significantly outperform the expert’s policy by leveraging additional parametric knowledge about the reward signal.

Original: Moreover, our method reduces the causal IRL to the canonical form, which allows us to apply existing IRL methods to find an effective imitating policy, despite the presence of unobserved confounding bias.
Revised: Moreover, our method reduces the causal IRL to the canonical form, which allows us to apply existing IRL methods to find an effective imitating policy, despite the presence of unobserved confounding bias.

Original: Finally, we would like to clarify that there are many challenges in reinforcement learning, including estimation in the high-dimensional domain.
Revised: Finally, we would like to clarify that there are many challenges in reinforcement learning, including estimation in the high-dimensional domain.

Original: In this paper, we investigate a probably undermined but crucial one, which is, when the expert’s demonstrations are contaminated with biases induced by
Revised: In this paper, we investigate a probably undermined but crucial one, which is, when the expert’s demonstrations are contaminated with biases induced by

Original: UCs, potentially leading to suboptimal or even undesired imitating behaviors.
Revised: UCs, potentially leading to suboptimal or even undesired imitating behaviors.

Original: It is orthogonal to many challenges that state-of-art IRL methods concern, e.g., high-dimensionality, and we believe that it is as equally urgent to address as other existing challenges in IRL.
Revised: It is orthogonal to many challenges that state-of-art IRL methods concern, e.g., high-dimensionality, and we believe that it is as equally urgent to address as other existing challenges in IRL.

Original: Our goal in this paper is to enhance the existing IRL methods to be robust against confounding bias through knowledge augmentation, provided with a causal perspective about the environment.
Revised: Our goal in this paper is to enhance the existing IRL methods to be robust against confounding bias through knowledge augmentation, provided with a causal perspective about the environment.

Original: This is where we believe that the communities of causality and imitation learning could and should converge.
Revised: This is where we believe that the communities of causality and imitation learning could and should converge.

Original: F F REQUENTLY A SKED Q UESTIONS
Revised: F F REQUENTLY A SKED Q UESTIONS

Original: Q1.
Revised: Q1.

Original: Are causal assumptions necessary for causal imitation learning?
Revised: Are causal assumptions necessary for causal imitation learning?

Original: Answer.
Revised: Answer.

Original: Yes, causal assumptions are necessary for practical imitation learning when some input covariates of the expert’s policy are latent to the imitator, and unobserved confounders (UCs) are present in the expert’s demonstrations.
Revised: Yes, causal assumptions are necessary for practical imitation learning when some input covariates of the expert’s policy are latent to the imitator, and unobserved confounders (UCs) are present in the expert’s demonstrations.

Original: The departing point of our work is the realization that an imitating policy that performs as well as the expert is generally underdetermined by the observational data alone.
Revised: The departing point of our work is the realization that an imitating policy that performs as well as the expert is generally underdetermined by the observational data alone.

Original: Indeed, this observation is not new, and was first made in (Zhang et al.
Revised: Indeed, this observation is not new, and was first made in (Zhang et al.

Original: , 2020).For concreteness, consider models M 1 , M 2 , unknown to researchers, where in M 1 , X ← U , Y ← X ; in M 2 , X ← U , Y ← X ⊕ U ; in
Revised: , 2020).For concreteness, consider models M 1 , M 2 , unknown to researchers, where in M 1 , X ← U , Y ← X ; in M 2 , X ← U , Y ← X ⊕ U ; in

Original: M i , i = 1 , 2 , P ( U = 0) =
Revised: M i , i = 1 , 2 , P ( U = 0) =

Original: P ( U = 1) = 0 .
Revised: P ( U = 1) = 0 .

Original: We assume that Y, U are unobserved; Y is the reward.
Revised: We assume that Y, U are unobserved; Y is the reward.

Original: In both M 1 and M 2 , the observational distribution P ( X = 0) =
Revised: In both M 1 and M 2 , the observational distribution P ( X = 0) =

Original: P ( X = 1) = 0 .
Revised: P ( X = 1) = 0 .

Original: In M 1 , P ( y ) is imitable with policy π ( x )
Revised: In M 1 , P ( y ) is imitable with policy π ( x )

Original: =
Revised: =

Original: P ( x ) ; while in M 2 , P ( Y = 

Revised: P ( x ) ; while in M 2 , P ( Y =

Original: | do ( π ))
Revised: | do ( π ))

Original: = 0 .for
Revised: = 0 .for

Original: any π ( x ) , which is far from P ( Y = 0)
Revised: any π ( x ) , which is far from P ( Y = 0)

Original: = 1 .
Revised: = 1 .

Original: This example shows that when UCs are present, imitation learning from observations alone is generally impossible.
Revised: This example shows that when UCs are present, imitation learning from observations alone is generally impossible.

Original: A common approach to addressing the challenges of UCs is to explore causal relationships among variables.
Revised: A common approach to addressing the challenges of UCs is to explore causal relationships among variables.

Original: Such relationships could be compactly represented in the form of a causal diagram (Pearl, 2000).
Revised: Such relationships could be compactly represented in the form of a causal diagram (Pearl, 2000).

Original: Consider again the previous example.
Revised: Consider again the previous example.

Original: Note that the bi-directed arrow X ↔ Y exists in G 2 but not G 1 .
Revised: Note that the bi-directed arrow X ↔ Y exists in G 2 but not G 1 .

Original: One could distinguish SCMs M 1 , M 2 using their corresponding causal diagrams G 1 , G 2 , and obtain an effective imitating policy when it is possible (i.e., M 1 ).
Revised: One could distinguish SCMs M 1 , M 2 using their corresponding causal diagrams G 1 , G 2 , and obtain an effective imitating policy when it is possible (i.e., M 1 ).

Original: Finally, we would like to point out that most standard imitation learning methods, including both BC and IRL, focus on the Markov decision process (MDP) model.
Revised: Finally, we would like to point out that most standard imitation learning methods, including both BC and IRL, focus on the Markov decision process (MDP) model.

Original: It (perhaps implicitly) assumes the structural constraint of unconfoundedness , i.e., UCs do not exist in the expert’s demonstrations.
Revised: It (perhaps implicitly) assumes the structural constraint of unconfoundedness , i.e., UCs do not exist in the expert’s demonstrations.

Original: In other words, existing BC and IRL methods are developed based on a common set of causal assumptions, which we could compactly represent using a canonical causal diagram (e.g., see (Zhang & Bareinboim, 2022)).
Revised: In other words, existing BC and IRL methods are developed based on a common set of causal assumptions, which we could compactly represent using a canonical causal diagram (e.g., see (Zhang & Bareinboim, 2022)).

Original: This canonical MDP graph does not include the presence of UCs, thus not representative of the more generalized setting studied in this paper.
Revised: This canonical MDP graph does not include the presence of UCs, thus not representative of the more generalized setting studied in this paper.

Original: Q2.
Revised: Q2.

Original: Is the requirement of causal diagram a limitation?
Revised: Is the requirement of causal diagram a limitation?

Original: Answer.
Revised: Answer.

Original: We note that requiring a causal diagram as input is not a limitation of our work.
Revised: We note that requiring a causal diagram as input is not a limitation of our work.

Original: Instead, as explained in Q1, causal assumptions are necessary for practical IRL that aims to provide guarantees and understand the conditions algorithms may succeed in many real-world applications.
Revised: Instead, as explained in Q1, causal assumptions are necessary for practical IRL that aims to provide guarantees and understand the conditions algorithms may succeed in many real-world applications.

Original: Our reasoning and arguments are summarized below:
Revised: Our reasoning and arguments are summarized below:

Original: Unobserved confounders generally exist in demonstration data when the sensory capabilities of the imitator and the expert differ, e.g., HighD (Krajewski et al., 2018).
Revised: Unobserved confounders generally exist in demonstration data when the sensory capabilities of the imitator and the expert differ, e.g., HighD (Krajewski et al., 2018).

Original: Without any causal assumption, existing imitation learning algorithms, including BC and IRL, are unable to learn an effective policy from the demonstration data alone that performs as well as the expert (Etesami & Geiger, 2020;
Revised: Without any causal assumption, existing imitation learning algorithms, including BC and IRL, are unable to learn an effective policy from the demonstration data alone that performs as well as the expert (Etesami & Geiger, 2020;

Original: Kumor et al., 2021; Zhang et al., 2020).
Revised: Kumor et al., 2021; Zhang et al., 2020).

Original: That is, when UCs are present, causal assumptions about the environment are necessary to achieve reasonable imitation performance via IRL.
Revised: That is, when UCs are present, causal assumptions about the environment are necessary to achieve reasonable imitation performance via IRL.

Original: Existing IRL methods assume the underlying environment to be an MDP model, which satisfies the Markov property and precludes the analysis of UCs.
Revised: Existing IRL methods assume the underlying environment to be an MDP model, which satisfies the Markov property and precludes the analysis of UCs.

Original: A causal diagram (Pearl, 2000) represents an arbitrary collection of causal relationships in the environment.
Revised: A causal diagram (Pearl, 2000) represents an arbitrary collection of causal relationships in the environment.

Original: This flexibility allows one to encode the existence of UCs explicitly.
Revised: This flexibility allows one to encode the existence of UCs explicitly.

Original: To sum up, this paper extends existing IRL methods, via simple augmentation, to the generalized setting where UCs generally exist, and causal relationships are encoded in the form of a causal diagram.
Revised: To sum up, this paper extends existing IRL methods, via simple augmentation, to the generalized setting where UCs generally exist, and causal relationships are encoded in the form of a causal diagram.

Original: In a very general sense, the structural assumptions required to perform causal inferences are inevitable, as shown in (Bareinboim et al., 2022, Theorem 1).
Revised: In a very general sense, the structural assumptions required to perform causal inferences are inevitable, as shown in (Bareinboim et al., 2022, Theorem 1).

Original: We then note that in some compelling real-world applications, one may have a qualitative understanding of the causal relationships of the environment, e.g., the ad-placement engine (Bottou et al., 2013).
Revised: We then note that in some compelling real-world applications, one may have a qualitative understanding of the causal relationships of the environment, e.g., the ad-placement engine (Bottou et al., 2013).

Original: Also, there are quite general and systematic methods under the rubrics of “causal discovery” capable of learning a causal diagram (or its equivalence class) from observational data (Spirtes et al., 2000).
Revised: Also, there are quite general and systematic methods under the rubrics of “causal discovery” capable of learning a causal diagram (or its equivalence class) from observational data (Spirtes et al., 2000).

Original: Our methods could be combined with causal discovery algorithms to learn an imitating policy after the graph is obtained.
Revised: Our methods could be combined with causal discovery algorithms to learn an imitating policy after the graph is obtained.

Original: This means that the dependence on domain experts could be minimized.
Revised: This means that the dependence on domain experts could be minimized.

Original: Q3.
Revised: Q3.

Original: The learned policy requires a separate decision rule at each time step.
Revised: The learned policy requires a separate decision rule at each time step.

Original: How does it differ from stationary policies in MDPs?
Revised: How does it differ from stationary policies in MDPs?

Original: Answer.
Revised: Answer.

Original: A policy that remains invariant at each step is said to be a stationary policy.
Revised: A policy that remains invariant at each step is said to be a stationary policy.

Original: It is a special case of non-stationary policies where decision rules are different at every time step (Puterman, 1994; Fix et al., 1993).
Revised: It is a special case of non-stationary policies where decision rules are different at every time step (Puterman, 1994; Fix et al., 1993).

Original: Stationary policies are optimal in MDP models that satisfy the Markov property.
Revised: Stationary policies are optimal in MDP models that satisfy the Markov property.