Original: Stationary policies are optimal in MDP models that satisfy the Markov property.
Revised: Stationary policies are optimal in MDP models that satisfy the Markov property.

Original: However, its optimality guarantee no longer holds in non-Markovian environments, where the transition distributions differ over every stage of action.
Revised: However, its optimality guarantee no longer holds in non-Markovian environments, where the transition distributions differ over every stage of action.

Original: This paper focuses on settings where Markovianity cannot be ascertained.
Revised: This paper focuses on settings where Markovianity cannot be ascertained.

Original: Generally, policies considered in this paper are also referred to as dynamic treatment regimes (DTR) (Murphy et al., 2001) in the causal inference literature.
Revised: Generally, policies considered in this paper are also referred to as dynamic treatment regimes (DTR) (Murphy et al., 2001) in the causal inference literature.

Original: DTRs are a popular model, and have been widely applied for medical decision-making.
Revised: DTRs are a popular model, and have been widely applied for medical decision-making.

Original: Unlike standard MDP settings,
Revised: Unlike standard MDP settings,

Original: DTR does not assume that the underlying environment satisfies the Markov property and the transition functions are generally different between different stages of intervention (action).
Revised: DTR does not assume that the underlying environment satisfies the Markov property and the transition functions are generally different between different stages of intervention (action).

Original: Consequently, the optimal DTR policy is generally non-stationary, requiring a separate policy for every action stage.
Revised: Consequently, the optimal DTR policy is generally non-stationary, requiring a separate policy for every action stage.

Original: We would like to highlight that the most challenging aspect of our investigation is confounding biases in the expert’s demonstrations induced by unobserved confounders (UCs).
Revised: We would like to highlight that the most challenging aspect of our investigation is confounding biases in the expert’s demonstrations induced by unobserved confounders (UCs).

Original: UCs are pervasive in practical scenarios since not all information available to the decision-maker is recorded and available to the AI system, including the aforementioned medical treatment.
Revised: UCs are pervasive in practical scenarios since not all information available to the decision-maker is recorded and available to the AI system, including the aforementioned medical treatment.

Original: Proposing a causal framework to imitate non-stationary policies for non-Markovian is more general, and can be naturally adapted to stationary policies in a Markovian environment.
Revised: Proposing a causal framework to imitate non-stationary policies for non-Markovian is more general, and can be naturally adapted to stationary policies in a Markovian environment.

Original: Q4.
Revised: Q4.

Original: What is the difference between GAIL and Causal GAIL?
Revised: What is the difference between GAIL and Causal GAIL?

Original: Answer.
Revised: Answer.

Original: GAIL was original developed in the Markov decision process (MDP)
Revised: GAIL was original developed in the Markov decision process (MDP)

Original: (Ho &
Revised: (Ho &

Original: Ermon, 2016), which encodes the assumption of unconfoundedness , i.e., unobserved confounders do not exist in the expert’s demonstrations.
Revised: Ermon, 2016), which encodes the assumption of unconfoundedness , i.e., unobserved confounders do not exist in the expert’s demonstrations.

Original: On the other hand, Causal GAIL does not rely on such an assumption and is robust to the bias introduced by the presence of unobserved confounding.
Revised: On the other hand, Causal GAIL does not rely on such an assumption and is robust to the bias introduced by the presence of unobserved confounding.

Original: Details of Causal GAIL are provided in Alg.
Revised: Details of Causal GAIL are provided in Alg.

Original: 4 (Appendix C).
Revised: 4 (Appendix C).

Original: During the initialization (Step 1), it first applies the identification procedure Thms.
Revised: During the initialization (Step 1), it first applies the identification procedure Thms.

Original: 1 and 

Revised: 1 and

Original: to reduce the causal imitation problem to its corresponding canonical equation (Eq.
Revised: to reduce the causal imitation problem to its corresponding canonical equation (Eq.

Original: Starting there, the learning procedure is essentially the GAIL algorithm applied to non-stationary DTR policies (Murphy et al., 2001).
Revised: Starting there, the learning procedure is essentially the GAIL algorithm applied to non-stationary DTR policies (Murphy et al., 2001).

Original: Step 4 updates the reward function D , parameterized as a neural network discriminator, using the standard gradient descent.
Revised: Step 4 updates the reward function D , parameterized as a neural network discriminator, using the standard gradient descent.

Original: Step 5 computes an optimal policy in an
Revised: Step 5 computes an optimal policy in an

Original: SCM given a fixed reward function.
Revised: SCM given a fixed reward function.

Original: There are a huge array of techniques available for this.
Revised: There are a huge array of techniques available for this.

Original: As for discrete domains, the transition distributions are computed using empirical means, and we then solve for an optimal policy using standard dynamic programming (Bellman, 1966).
Revised: As for discrete domains, the transition distributions are computed using empirical means, and we then solve for an optimal policy using standard dynamic programming (Bellman, 1966).

Original: For an environment with continuous states and actions space, we solve for the optimal policy using stochastic policy gradient (Sutton et al., 1999).
Revised: For an environment with continuous states and actions space, we solve for the optimal policy using stochastic policy gradient (Sutton et al., 1999).

Original: Expected rewards of policies are estimated by a Monte-Carlo method computing empirical means over the agent’s trajectories.
Revised: Expected rewards of policies are estimated by a Monte-Carlo method computing empirical means over the agent’s trajectories.

Original: Effective function approximation methods also exist to evaluate the value function of policies in high-dimensional domains (Tsitsiklis & Van Roy, 1997; Murphy, 2005).
Revised: Effective function approximation methods also exist to evaluate the value function of policies in high-dimensional domains (Tsitsiklis & Van Roy, 1997; Murphy, 2005).

Original: Q5.
Revised: Q5.

Original: Could the proposed method be scaled up to high-dimensional, complex environments?
Revised: Could the proposed method be scaled up to high-dimensional, complex environments?

Original: Answer.
Revised: Answer.

Original: Yes, our causal-IRL method naturally scales to general sequential decisionmaking settings with higher dimensional domains.
Revised: Yes, our causal-IRL method naturally scales to general sequential decisionmaking settings with higher dimensional domains.

Original: A key observation in our proposed approach is the reduction to the canonical equation of causal IRL in Eq.
Revised: A key observation in our proposed approach is the reduction to the canonical equation of causal IRL in Eq.

Original: Once the canonical equation is obtained, the learner could then solves for an effective imitating policy using the standard IRL method, including MWAL and GAIL.
Revised: Once the canonical equation is obtained, the learner could then solves for an effective imitating policy using the standard IRL method, including MWAL and GAIL.

Original: Details of the implementation are described in Sec.
Revised: Details of the implementation are described in Sec.

Original: 2.2 and Appendix C.
Revised: 2.2 and Appendix C.

Original: This paper introduces novel algorithms (Thms.and 2) to reduce the causal imitation learning problem to its corresponding canonical form (when it is possible).
Revised: This paper introduces novel algorithms (Thms.and 2) to reduce the causal imitation learning problem to its corresponding canonical form (when it is possible).

Original: The reduction procedure takes at most a polynomial-number of steps with regard to the size of the causal diagram.
Revised: The reduction procedure takes at most a polynomial-number of steps with regard to the size of the causal diagram.

Original: This allows us to benefit from the state-of-art computational framework for IRL in MDP models while ascertaining the validity of the procedure through proper causal tools.
Revised: This allows us to benefit from the state-of-art computational framework for IRL in MDP models while ascertaining the validity of the procedure through proper causal tools.

Original: For instance, we demonstrate the validity of our algorithm in Experiment 3 using the MNIST digits (LeCun, 1998).
Revised: For instance, we demonstrate the validity of our algorithm in Experiment 3 using the MNIST digits (LeCun, 1998).

Original: Here, we would also like to clarify that there are many challenges in IRL.
Revised: Here, we would also like to clarify that there are many challenges in IRL.

Original: Scalability to the high-dimensional domain is one of them.
Revised: Scalability to the high-dimensional domain is one of them.

Original: However, in this paper, we investigate another challenge, a probably undermined but crucial one, which is, the settings where the expert’s demonstrations are contaminated with biases induced by unobserved confounders (UCs).
Revised: However, in this paper, we investigate another challenge, a probably undermined but crucial one, which is, the settings where the expert’s demonstrations are contaminated with biases induced by unobserved confounders (UCs).

Original: It is orthogonal to the challenge of high-dimensionality that state-of-art IRL methods concern, and we strongly believe that it is as equally urgent to address as other existing challenges in
Revised: It is orthogonal to the challenge of high-dimensionality that state-of-art IRL methods concern, and we strongly believe that it is as equally urgent to address as other existing challenges in

Original: IRL.
Revised: IRL.

Original: Indeed, when UCs are present, it is possible to construct a simple model with binary variables such that existing IRL methods cannot learn an effective imitating policy from demonstrations alone, regardless of how advanced the feature extraction procedures are and what the sample size is.
Revised: Indeed, when UCs are present, it is possible to construct a simple model with binary variables such that existing IRL methods cannot learn an effective imitating policy from demonstrations alone, regardless of how advanced the feature extraction procedures are and what the sample size is.

Original: Our goal in this paper is to enhance the existing IRL methods to be robust against confounding bias through knowledge augmentation, provided with a causal perspective about the environment.
Revised: Our goal in this paper is to enhance the existing IRL methods to be robust against confounding bias through knowledge augmentation, provided with a causal perspective about the environment.

Original: This is where we believe that the communities of causality and imitation learning could and should converge.
Revised: This is where we believe that the communities of causality and imitation learning could and should converge.

Original: Thus, it is not our purpose to beat existing IRL methods in another benchmark with high-dimensional domains that match the assumptions they expect (i.e., UCs are assumed away).
Revised: Thus, it is not our purpose to beat existing IRL methods in another benchmark with high-dimensional domains that match the assumptions they expect (i.e., UCs are assumed away).

Original: Q6.
Revised: Q6.

Original: Could the proposed method be scaled to long-sequence decision problems with infinite horizons?
Revised: Could the proposed method be scaled to long-sequence decision problems with infinite horizons?

Original: Answer.
Revised: Answer.

Original: Yes, the proposed methods could be generalized to sequential decision problems with infinite horizons by incorporating common assumptions of temporal models.
Revised: Yes, the proposed methods could be generalized to sequential decision problems with infinite horizons by incorporating common assumptions of temporal models.

Original: In
Revised: In

Original: Experiment 4, we evaluate the proposed algorithm in a generalized Markov decision process incorporating unobserved confounders, called the MDPUC (Ruan & Di, 2022; Zhang &
Revised: Experiment 4, we evaluate the proposed algorithm in a generalized Markov decision process incorporating unobserved confounders, called the MDPUC (Ruan & Di, 2022; Zhang &

Original: Bareinboim, 2022).
Revised: Bareinboim, 2022).

Original: The graph underlying this experiment properly simulates the real-world driving decision process.
Revised: The graph underlying this experiment properly simulates the real-world driving decision process.

Original: By exploiting the Markov property over time, we can decompose the causal diagram over the infinite horizon into a collection of sub-graphs, one for each pair of the action X i and the reward Y i .
Revised: By exploiting the Markov property over time, we can decompose the causal diagram over the infinite horizon into a collection of sub-graphs, one for each pair of the action X i and the reward Y i .

Original: Fig.
Revised: Fig.

Original: 1d shows the causal diagram spanning over time step i = 1 , 2 , 3 .
Revised: 1d shows the causal diagram spanning over time step i = 1 , 2 , 3 .

Original: As a comparison, BC and IRL still utilize the standard 1-step information scope {⟨ X i , { Z i }⟩} .
Revised: As a comparison, BC and IRL still utilize the standard 1-step information scope {⟨ X i , { Z i }⟩} .

Original: By applying Thm.
Revised: By applying Thm.

Original: 1 at each time step, we obtain a π -backdoor admissible policy scope {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} for Causal-IRL and Causal-BC .
Revised: 1 at each time step, we obtain a π -backdoor admissible policy scope {⟨ X i , { Z i , X i − 1 , Z i − 1 }⟩} for Causal-IRL and Causal-BC .

Original: Simulation results are shown in Fig.
Revised: Simulation results are shown in Fig.

Original: 3d.
Revised: 3d.

Original: One could see by inspection that Causal-IRL performs the best among all algorithms and achieve the experts’ performance.
Revised: One could see by inspection that Causal-IRL performs the best among all algorithms and achieve the experts’ performance.

Original: Q7.
Revised: Q7.

Original: How is Causal
Revised: How is Causal

Original: IRL
Revised: IRL

Original: able
Revised: able

Original: to
Revised: to

Original: outperform
Revised: outperform

Original: Causal
Revised: Causal

Original: BC
Revised: BC

Original: and the expert
Revised: and the expert

Original: Answer.
Revised: Answer.

Original: When the imitator has no prior knowledge about the reward function, methods like GAIL model the reward function as a general discriminator (i.e., a neural network), and are able to converge to the equilibrium point where the imitator exactly matches the expert’s distributions.
Revised: When the imitator has no prior knowledge about the reward function, methods like GAIL model the reward function as a general discriminator (i.e., a neural network), and are able to converge to the equilibrium point where the imitator exactly matches the expert’s distributions.

Original: In other words, causal IRL generally coincides with causal BC without additional parametric knowledge about the latent reward signal.
Revised: In other words, causal IRL generally coincides with causal BC without additional parametric knowledge about the latent reward signal.

Original: In general, causal IRL consistently dominates causal BC since it can exploit parametric knowledge about the reward function.
Revised: In general, causal IRL consistently dominates causal BC since it can exploit parametric knowledge about the reward function.

Original: This observation is not new and was noted in the literature, e.g., see (Syed & Schapire, 2008; Li et al., 2017).
Revised: This observation is not new and was noted in the literature, e.g., see (Syed & Schapire, 2008; Li et al., 2017).

Original: Effective methods exist to encode prior knowledge about the reward signal when training GAIL-type imitators, which are referred to as reward augmentation (Li et al., 2017).
Revised: Effective methods exist to encode prior knowledge about the reward signal when training GAIL-type imitators, which are referred to as reward augmentation (Li et al., 2017).

Original: For instance, in Experiment 2, we train causal-IRL imitator using reward augmentation that encourages the agent to accelerate when the velocity of the preceding vehicle is high.
Revised: For instance, in Experiment 2, we train causal-IRL imitator using reward augmentation that encourages the agent to accelerate when the velocity of the preceding vehicle is high.

Original: Simulation results in Fig.
Revised: Simulation results in Fig.

Original: 3b show that causal-IRL can consistently outperform the expert and causal BC by exploiting the additional parametric knowledge about reward.
Revised: 3b show that causal-IRL can consistently outperform the expert and causal BC by exploiting the additional parametric knowledge about reward.

Original: In Experiment 8 (highway driving), we trained another causal-IRL imitator without additional knowledge about the reward.
Revised: In Experiment 8 (highway driving), we trained another causal-IRL imitator without additional knowledge about the reward.

Original: Simulation results in
Revised: Simulation results in

Original: Fig.
Revised: Fig.

Original: 5d show that the performance of causal-IRL and causal-BC coincide, exactly matching the expert’s performance.
Revised: 5d show that the performance of causal-IRL and causal-BC coincide, exactly matching the expert’s performance.

Original: Q8.
Revised: Q8.

Original: Could the proposed method be applicable to settings with multiple reward signals?
Revised: Could the proposed method be applicable to settings with multiple reward signals?

Original: Answer.
Revised: Answer.

Original: Yes, following the conventions of the dynamic treatment regime (DTR) literature (Murphy et al., 2001), the reward signal is represented as a single random variable.
Revised: Yes, following the conventions of the dynamic treatment regime (DTR) literature (Murphy et al., 2001), the reward signal is represented as a single random variable.

Original: In a nutshell, policy optimization methods for DTR extend immediately to multiple reward signals, one per every stage of action (Chakraborty & Moodie, 2013).
Revised: In a nutshell, policy optimization methods for DTR extend immediately to multiple reward signals, one per every stage of action (Chakraborty & Moodie, 2013).

Original: Theoretically, Thms.and 2 hold when the reward Y is a set of variables.
Revised: Theoretically, Thms.and 2 hold when the reward Y is a set of variables.

Original: The reward function r ( x ∗ , z ∗ ) is defined as the cumulative reward (cid:80) Y ∈ Y E
Revised: The reward function r ( x ∗ , z ∗ ) is defined as the cumulative reward (cid:80) Y ∈ Y E

Original: [ Y | x ∗ , z ∗ ] .
Revised: [ Y | x ∗ , z ∗ ] .

Original: For concreteness, we demonstrate our algorithm in Experiment 4 to optimize the cumulative reward in a sequential decisionmaking problem with an infinite horizon.
Revised: For concreteness, we demonstrate our algorithm in Experiment 4 to optimize the cumulative reward in a sequential decisionmaking problem with an infinite horizon.

Original: Q9.
Revised: Q9.

Original: What is the projection algorithm, and why do we need to perform such projections?
Revised: What is the projection algorithm, and why do we need to perform such projections?

Original: Answer.
Revised: Answer.

Original: The projection algorithm (Tian, 2002, Sec. 4.5) marginalizes unobserved endogenous variables in a causal diagram and allows us to focus on the causal relationships among observed endogenous variables.
Revised: The projection algorithm (Tian, 2002, Sec. 4.5) marginalizes unobserved endogenous variables in a causal diagram and allows us to focus on the causal relationships among observed endogenous variables.

Original: It is particularly useful for causal identification since the identifiability of causal effects remains invariant across the projection operation.
Revised: It is particularly useful for causal identification since the identifiability of causal effects remains invariant across the projection operation.

Original: Details fo the projection algorithm is provided in Alg.
Revised: Details fo the projection algorithm is provided in Alg.

Original: It transforms an arbitrary causal diagram G with observed endogenous variables O into a simplified causal diagram H such that unobserved endogenous variables V \ O are omitted.
Revised: It transforms an arbitrary causal diagram G with observed endogenous variables O into a simplified causal diagram H such that unobserved endogenous variables V \ O are omitted.

Original: For instance, consider a causal diagram G with directed arrows
Revised: For instance, consider a causal diagram G with directed arrows

Original: X
Revised: X

Original: → Z → Y and a bi-directed arrow X ↔ Y ; variable Z is unobserved so the observational distribution
Revised: → Z → Y and a bi-directed arrow X ↔ Y ; variable Z is unobserved so the observational distribution

Original: P ( X, Y ) .
Revised: P ( X, Y ) .

Original: In this case, one could simplify the diagram by projecting out node Z , i.e.,
Revised: In this case, one could simplify the diagram by projecting out node Z , i.e.,

Original: H =
Revised: H =

Original: P ROJECT ( G ) .
Revised: P ROJECT ( G ) .

Original: The resulting diagram H is given by X → Y and X ↔ Y .
Revised: The resulting diagram H is given by X → Y and X ↔ Y .

Original: It can be shown that the interventional distribution P ( y | do ( x )) is not identifiable in the projected diagram H , thus is not identifiable from P ( X, Y ) in the original diagram G .
Revised: It can be shown that the interventional distribution P ( y | do ( x )) is not identifiable in the projected diagram H , thus is not identifiable from P ( X, Y ) in the original diagram G .

Original: Due to these nice properties of the projection algorithm, in Sec. 3, we consistently assume the causal diagram G to be the outcome of the projection algorithm, only consisting of endogenous nodes O , Y .
Revised: Due to these nice properties of the projection algorithm, in Sec. 3, we consistently assume the causal diagram G to be the outcome of the projection algorithm, only consisting of endogenous nodes O , Y .

Original: The identifiablity results in Thm. 2 and Lem.
Revised: The identifiablity results in Thm. 2 and Lem.

Original: 1 hold without loss of generality.
Revised: 1 hold without loss of generality.

Original: Q10.
Revised: Q10.

Original: Why does the proposed method mainly focus on ν ∗ ≤ 0 ?
Revised: Why does the proposed method mainly focus on ν ∗ ≤ 0 ?

Original: Answer.
Revised: Answer.

Original: The condition ν ∗ ≤ 0 indicates that the expert’s policy might be sub-optimal and the Causal IRL algorithm is able to find a policy that improves over the expert.
Revised: The condition ν ∗ ≤ 0 indicates that the expert’s policy might be sub-optimal and the Causal IRL algorithm is able to find a policy that improves over the expert.

Original: Many real-world applications exist where the expert generating the demonstrations is possibly sub-optimal and could be improved.
Revised: Many real-world applications exist where the expert generating the demonstrations is possibly sub-optimal and could be improved.

Original: For instance, consider the development of autonomous vehicles, where the reinforcement signal is never fully known, and imitation learning methods are widely used.
Revised: For instance, consider the development of autonomous vehicles, where the reinforcement signal is never fully known, and imitation learning methods are widely used.

Original: To collect demonstration data, it often requires a human demonstrator, possibly a driving expert, to drive the vehicle around the target environment (e.g., a city, a highway) and record the natural driving trajectories.
Revised: To collect demonstration data, it often requires a human demonstrator, possibly a driving expert, to drive the vehicle around the target environment (e.g., a city, a highway) and record the natural driving trajectories.

Original: No matter how clean their driving record is, the human demonstrator is naturally error-prone and cannot guarantee optimal driving behaviors at all times (Atchley et al., 2011).
Revised: No matter how clean their driving record is, the human demonstrator is naturally error-prone and cannot guarantee optimal driving behaviors at all times (Atchley et al., 2011).

Original: Indeed, in many complicated learning tasks, it might be difficult for the demonstrator to be consistently optimal, even for
Revised: Indeed, in many complicated learning tasks, it might be difficult for the demonstrator to be consistently optimal, even for

Original: 1: Input: A causal diagram G .
Revised: 1: Input: A causal diagram G .

Original: 2: Output: A causal diagram H where all endogenous variables are observed, i.e., V = O .
Revised: 2: Output: A causal diagram H where all endogenous variables are observed, i.e., V = O .

Original: 3: Let O , L be, respectively, observed endogenous variables and latent endogenous variables in G .
Revised: 3: Let O , L be, respectively, observed endogenous variables and latent endogenous variables in G .

Original: 4: Let H be a causal diagram constructed as follows.
Revised: 4: Let H be a causal diagram constructed as follows.

Original: 5: for each observed V ∈ O in G do 6:
Revised: 5: for each observed V ∈ O in G do 6:

Original: Add an observed node V in H .
Revised: Add an observed node V in H .

Original: 7: end for 8: for each pair S, E ∈ O in G s.t.
Revised: 7: end for 8: for each pair S, E ∈ O in G s.t.

Original: S ̸
Revised: S ̸

Original: = E do 9: if there exists a directed path S → E in G then 10:
Revised: = E do 9: if there exists a directed path S → E in G then 10:

Original: Add an edge S → E in H .
Revised: Add an edge S → E in H .

Original: 11: else if there exists a path S → V 1 → · · · → V n → E in G s.t.
Revised: 11: else if there exists a path S → V 1 → · · · → V n → E in G s.t.

Original: V 1 , · · · V n ∈ L then 12:
Revised: V 1 , · · · V n ∈ L then 12:

Original: Add an edge S → E in H .
Revised: Add an edge S → E in H .

Original: 13: else if there exists a bidirected edge S ↔ E in G then 14:
Revised: 13: else if there exists a bidirected edge S ↔ E in G then 14:

Original: Add a bidirected edge S ↔ E in H .
Revised: Add a bidirected edge S ↔ E in H .

Original: 15: else if there exists a path S ← V l, 1 ← · · · ← V l,n ↔ V r,m → · · · → V r, 1 → E in G s.t.
Revised: 15: else if there exists a path S ← V l, 1 ← · · · ← V l,n ↔ V r,m → · · · → V r, 1 → E in G s.t.

Original: V l, 1 , · · · , V l,n , V r, 1 , · · · , V r,m ∈ L then 16:
Revised: V l, 1 , · · · , V l,n , V r, 1 , · · · , V r,m ∈ L then 16:

Original: Add a bidirected edge S ↔ E in H .
Revised: Add a bidirected edge S ↔ E in H .

Original: 17: else if there exists a path S ← V l, 1 ← · · · ← V l,n ← V c
Revised: 17: else if there exists a path S ← V l, 1 ← · · · ← V l,n ← V c

Original: → V r,m → · · · → V r, 1 → E in G s.t.
Revised: → V r,m → · · · → V r, 1 → E in G s.t.

Original: V l, 1 , · · · , V l,n , V c , V r, 1 , · · · , V r,m ∈ L then 18:
Revised: V l, 1 , · · · , V l,n , V c , V r, 1 , · · · , V r,m ∈ L then 18:

Original: Add a bidirected edge S ↔ E in H .
Revised: Add a bidirected edge S ↔ E in H .

Original: 19: end if 20: end for 21: Return H .
Revised: 19: end if 20: end for 21: Return H .

Original: relatively veteran experts, e.g., playing complex video games or stock trading (Newell et al., 1972; Nikolaidis et al., 2017).
Revised: relatively veteran experts, e.g., playing complex video games or stock trading (Newell et al., 1972; Nikolaidis et al., 2017).

Original: Behavioral cloning methods mimic the demonstrator’s policy and thus never outperform the sub-optimal expert.
Revised: Behavioral cloning methods mimic the demonstrator’s policy and thus never outperform the sub-optimal expert.

Original: On the other hand, by exploiting additional parametric knowledge about the latent reward signal, inverse RL methods are able to obtain an effective policy that consistently dominates the expert’s policy.
Revised: On the other hand, by exploiting additional parametric knowledge about the latent reward signal, inverse RL methods are able to obtain an effective policy that consistently dominates the expert’s policy.

Original: This observation is not new, and was first proposed in (Syed & Schapire, 2008), who also formalized the use of performance gap ν ∗ .
Revised: This observation is not new, and was first proposed in (Syed & Schapire, 2008), who also formalized the use of performance gap ν ∗ .

Original: (Zhang et al., 2020) studied the behavioral cloning from the combination of confounded observations and causal constraints about the environment.
Revised: (Zhang et al., 2020) studied the behavioral cloning from the combination of confounded observations and causal constraints about the environment.

Original: In this work, we take inspiration from inverse RL approaches and develop non-trivial imitation learning methods that are robust to both unobserved confounding and a sub-optimal expert.
Revised: In this work, we take inspiration from inverse RL approaches and develop non-trivial imitation learning methods that are robust to both unobserved confounding and a sub-optimal expert.

Original: Q11. How about ν ∗ > 0 ?
Revised: Q11. How about ν ∗ > 0 ?

Original: Answer.
Revised: Answer.

Original: When the performance gap ν ∗ > 0 , there exists at least one causal model such that the imitator can never achieve the expert’s performance.
Revised: When the performance gap ν ∗ > 0 , there exists at least one causal model such that the imitator can never achieve the expert’s performance.

Original: For instance, consider a bow graph consisting of a directed arrow X → Y and a bi-directed arrow X ↔ Y .
Revised: For instance, consider a bow graph consisting of a directed arrow X → Y and a bi-directed arrow X ↔ Y .

Original: We could construct a causal model M compatible with this causal constraint such that no intervention do ( x ) on action X could outperform the expert’s performance
Revised: We could construct a causal model M compatible with this causal constraint such that no intervention do ( x ) on action X could outperform the expert’s performance

Original: E
Revised: E

Original: [ Y ] .
Revised: [ Y ] .

Original: Let an exogenous variable U be uniformly drawn over a binary domain { 0 , 1 } .
Revised: Let an exogenous variable U be uniformly drawn over a binary domain { 0 , 1 } .

Original: Let the expert’s policy be X ← U and the reward function Y ←U ⊕ X  . 
Revised: Let the expert’s policy be X ← U and the reward function Y ← X ⊕ ¬ U . Fig.

Revised: 7 below describes the causal diagram associated with SCM M .

Revised: [Figure]

Original: Evaluating the expert’s reward  gives
Revised: Evaluating the expert’s reward in SCM M gives

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for any intervention do ( x ) , the imitator’s reward is given by
Revised: On the other hand, for any intervention do ( x ) , the imitator’s reward is given by

Original: [Equation]
Revised: [Equation]

Original: The last step holds since U is a uniform binary variable.
Revised: The last step holds since U is a uniform binary variable.

Original: In this environment, for any action, the imitator’s reward E
Revised: In this environment, for any action, the imitator’s reward E

Original: [ Y | do ( x )]
Revised: [ Y | do ( x )]

Original: = 0 .
Revised: = 0 .

Original: 5 is far from the expert’s performance E
Revised: 5 is far from the expert’s performance E

Original: [ Y ] .
Revised: [ Y ] .

Original: This means that in the bow graph, the performance gap is at least ν ∗ >
Revised: This means that in the bow graph, the performance gap is at least ν ∗ >

Original: E
Revised: E

Original: [ Y ] − E [ Y | do ( x )]
Revised: [ Y ] − E [ Y | do ( x )]

Original: = 0 .
Revised: = 0 .

Original: Such settings are referred to as “non-imitable” in (Zhang et al., 2020, Lemma 1). 
Revised: Such settings are referred to as “non-imitable” in (Zhang et al., 2020, Lemma 1). Q12.

Revised: How does causality help inverse RL to get better performance than the expert?

Original: Answer.
Revised: Answer.

Original: By utilizing causal relationships embedded in the environment, the learner is able to address the bias/distribution drift due to the presence of unobserved confounding in demonstration data.
Revised: By utilizing causal relationships embedded in the environment, the learner is able to address the bias/distribution drift due to the presence of unobserved confounding in demonstration data.

Original: Once the adjustment formula is obtained (i.e., Thms. 1 and 2), IRL algorithms could produce a policy that substantially outperforms the expert by exploiting parametric knowledge about the reward, e.g., linearity (Syed & Schapire, 2008) or through reward augmentation (Li et al., 2017).
Revised: Once the adjustment formula is obtained (i.e., Thms. 1 and 2), IRL algorithms could produce a policy that substantially outperforms the expert by exploiting parametric knowledge about the reward, e.g., linearity (Syed & Schapire, 2008) or through reward augmentation (Li et al., 2017).

Original: In this work, one of our main contributions is to extend existing IRL approaches to more generalized settings where the expert’s demonstrations are imperfect, and the phenomenon of unobserved confounding exists (i.e., Markovianity cannot be assumed).
Revised: In this work, one of our main contributions is to extend existing IRL approaches to more generalized settings where the expert’s demonstrations are imperfect, and the phenomenon of unobserved confounding exists (i.e., Markovianity cannot be assumed).

Original: To further ground our contributions, we summarize the current literature on imitation learning as follows, in order to highlight that our work fills a critical literature gap where unobserved confounders exist in the sequential imitating processes and the expert is sub-optimal:
Revised: To further ground our contributions, we summarize the current literature on imitation learning as follows, in order to highlight that our work fills a critical literature gap where unobserved confounders exist in the sequential imitating processes and the expert is sub-optimal:

Original: Optimal Expert
Revised: Optimal Expert

Original: Sub-optimal Expert

Revised: [Equation]

Original: Unconfounded
Revised: Unconfounded

Original: Behavior Cloning (BC):
Revised: Behavior Cloning (BC):

Original: Widrow (1964); Pomerleau (1989);
Revised: Widrow (1964); Pomerleau (1989);

Original: Muller et al.
Revised: Muller et al.

Original: ; M¨ulling et al.
Revised: M ¨ ulling et al.

Original: ; Mahler & Goldberg (2017)
Revised: ; Mahler & Goldberg (2017)

Original: Invserse Reinforcement Learning (IRL):
Revised: Invserse Reinforcement Learning (IRL):

Original: Ng et al.
Revised: Ng et al.

Original: ; Ziebart et al.
Revised: ; Ziebart et al.

Original: Ho & Ermon (2016); Fu et al.
Revised: Ho & Ermon (2016); Fu et al.

Original: Confounded
Revised: Confounded

Original: Causal BC:
Revised: Causal BC:

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020); Kumor et al.
Revised: (2020); Kumor et al.

Original: Causal IRL: our work
Revised: Causal IRL: our work

Original: To sum up, behavior cloning (BC) is able to achieve satisfactory performance when the demonstration data is unconfounded, and the expert is (near) optimal.
Revised: To sum up, behavior cloning (BC) is able to achieve satisfactory performance when the demonstration data is unconfounded, and the expert is (near) optimal.

Original: Inverse RL (IRL) can outperform the suboptimal expert by exploiting additional parametric knowledge about the reward signal.
Revised: Inverse RL (IRL) can outperform the suboptimal expert by exploiting additional parametric knowledge about the reward signal.

Original: However, both BC and IRL are fragile to data drift due to the presence of unobserved confounders.
Revised: However, both BC and IRL are fragile to data drift due to the presence of unobserved confounders.

Original: Causal BC (Zhang et al., 2020; Kumor et al., 2021) produces a confounding-robust policy but is still limited by the performance of a suboptimal expert.
Revised: Causal BC (Zhang et al., 2020; Kumor et al., 2021) produces a confounding-robust policy but is still limited by the performance of a suboptimal expert.

Original: Finally, this paper empowers IRL methods with the causal inference theory so that the IRL imitator could produce a policy robust to both unobserved confounding and the suboptimal expert. 
Revised: Finally, this paper empowers IRL methods with the causal inference theory so that the IRL imitator could produce a policy robust to both unobserved confounding and the suboptimal expert. Q13. What is the problem formulation of causal IRL? Answer.

Revised: Details of the problem formulation have been described in Sec. 2.

Revised: Formally, the problem of causal IRL can be summarized as follows:

Revised: – Input: (1) the expert’s demonstrations, summarized as the observational distribution P ( O ) , and (2) a causal diagram G associated with the underlying SCM.

Revised: – Output: a policy π ∗ =

Revised: { π ∗ 1 ( X 1 | Z 1 ) , . . .

Revised: , π ∗ n ( X n | Z n ) } determining actions X .

Revised: – Objective: to learn an imitating policy π ∗ that achieves the expert’s performance, i.e., E [ Y | do ( π ∗ )] ≥ E

Revised: [ Y ] .

Revised: Q14.

Revised: Could the authors provide a table of notations?

Revised: Answer.

Revised: Yes, we provide in Table 2 a summary of important notations used in this paper.

Revised: [Table]

Revised: Preliminaries:

Revised: [Table]

Revised: Causal IRL:

Revised: [Table]

Revised: a submodel from M where f X replaced by π

Revised: [Table]

Revised: atomic intervention on X by constants x

Revised: [Table]

Revised: policy intervention on actions X following π

Revised: [Table]

Revised: occupancy measure of the imitator

Revised: [Table]

Revised: do

Revised: [Table]

Revised: ] imitator performance

Revised: Graph Notations :

Revised: G causal diagram associated with M

Revised: [Table]

Revised: S causal diagram associated with submodel M π pa , ch , de , an parents, children, descendants, and ancestors

Revised: Pa , Ch , De , An parents with the argument, children with the argument, descendants with the argument, and ancestors with the argument