Original: Causal Imitation Learning with Unobserved Contexts
Revised: Causal Imitation Learning with Unobserved Contexts

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider imitation learning problems where the expert has access to a perepisode context that is hidden from the learner, both in the demonstrations andat test-time.
Revised: We consider imitation learning problems where the expert has access to a perepisode context that is hidden from the learner, both in the demonstrations and attest-time.

Original: While the learner might not be able to accurately reproduce expertbehavior early on in an episode, by considering the entire history of states andactions, they might be able to eventually identify the context and act as the expertwould.
Revised: While the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions,they might be able to eventually identify the context and act as the expert would.

Original: We show that on-policy imitation learning algorithms (with or withoutaccess to a queryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods and are able to avoid thelatching behavior that plagues the latter.
Revised: We show that on-policy imitation learning algorithms (with or without access to aqueryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods.

Original: We conduct experiments in a toy banditdomain that show that there exist sharp phase transitions of whether off-policyapproaches are able to match expert performance asymptotically, in contrast to theuniformly good performance of on-policy approaches.
Revised: We conduct experiments in a toy banditdomain that show that there exist sharp phase transitions of whether off-policyapproaches are able to match expert performance asymptotically, in contrast to theuniformly good performance of on-policy approaches.

Original: We demonstrate that onseveral continuous control tasks, on-policy approaches are  able to use history toidentify the context while off-policy approaches are unable to do so.
Revised: We demonstrate that onseveral continuous control tasks, off-policy approaches are not able to effectivelyuse history to reduce their uncertainty over the context, in contrast to on-policymethods.

Original:  Introduction
Revised:  Introduction

Original: An unstated assumption in much of the work in imitation learning (IL) is that the learner and theexpert have access to the same state information.
Revised: An unstated assumption in much of the work in imitation learning (IL) is that the learner and theexpert have access to the same state information.

Original: With powerful enough statistical models, thisassumption places us in the realizable setting – i.e. the imitator can actually behave like the expert.
Revised: With powerful enough statistical models, thisassumption places the problem in the realizable setting – i.e. the imitator can actually behave likethe expert.

Original: In practice however, an expert might have more information than the learner does.
Revised: In practice, however, an expert might have more information than the learner does.

Original: For example,an expert policy might be trained in simulation with privileged access to state before being used tosupervise a learner policy thatonly has access to a subset of features. This recipe  has enjoyed successin domains from motion planning with obstacles [Choudhury et al., 2018], to autonomous driving[Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].
Revised: Forexample, an expert policy might be trained in simulation with privileged access to state before beingused to supervise a learner policy that has access to a subset of features, a recipe that has enjoyedsuccess in domains from motion planning with obstacles [Choudhury et al., 2018], to autonomousdriving [Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].

Original: Recent theoretical work has established “no-go” results for successfully imitating an expert that hasaccess to more information [Zhang et al., 2020, Kumor et al., 2021].
Revised: Recent theoretical work has established “no-go” results for successfully imitating an expert that hasaccess to more information [Zhang et al., 2020, Kumor et al., 2021].

Original: The core of their argumentsis that without seeing some feature that influences expert behavior but is not echoed elsewherein the state, the learner might not be able to properly ground the expert actions in the observedstate.
Revised: The core of their argumentsis that without seeing some feature that influences expert behavior but is not echoed elsewherein the state, the learner might not be able to properly ground the expert actions in the observedstate.

Original: In causal inference terms, this hidden information acts as an unobserved confounder whichprevents identification of the desired treatment effect (the expert action).
Revised: In causal inference terms, this hidden information acts as an unobserved confounder whichprevents identification of the desired treatment effect (the expert action).

Original: Despite these results,impressive empirical successes have been achieved even when the learner has a more impoverishedstate representation than the expert.
Revised: Despite these results,impressive empirical successes have been achieved even when the learner has a more impoverishedstate representation than the expert.

Original: In this work, we reconcile theory and practice by considering a broad class of problems where thelearner’s ability to mimic expert actions increases as more observations are revealed.
Revised: In this work, we reconcile theory and practice by considering a broad class of problems where thelearner’s ability to mimic expert actions increases as more observations are revealed.

Original: We study thelarge-horizon limit to tease out what is key to good performance.
Revised: We study thelarge-horizon limit to tease out what is key to good performance.

Original: We find that off-policy approaches(e.g.
Revised: We find that off-policy approaches(e.g.

Original: behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisionscan lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.
Revised: behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisionscan lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.

Original: Weshow that for some problem families, there exists a sharp phase transition in problem parameterswhere behavior cloning shifts between being consistent to having arbitrarily poor performance.
Revised: Weshow that for some problem families, there exists a sharp phase transition in problem parameterswhere behavior cloning shifts between being consistent to having arbitrarily poor performance.

Original: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Rosset al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal
Revised: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Rosset al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal

Original: Control [Bagnell, 2015])
Revised: Control [Bagnell, 2015])

Original: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent ofparameters) asymptotically consistent on these problems.
Revised: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent ofparameters) asymptotically consistent on these problems.

Original: We believe this strong separation betweenon- and off- policy approaches helps explain both the poor performance of behavioral cloning evenin regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,
Revised: This strong separation between on- and offpolicy approaches we believe explains empirical findings of poor performance of behavioral cloningeven in regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,

Original: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kuefler et al., 2017] and the successof on-policy methodsmentioned above.
Revised: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kuefler et al., 2017] and also explainsthe success of on-policy methods above.

Original: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,
Revised: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,

Original: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and
Revised: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and

Original: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expertperformance [Swamy et al., 2021].
Revised: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expertperformance [Swamy et al., 2021].

Original: Thus, for our experiments, we introduce partial observability toensure that we are focused on part of what makes imitation learning hard in practice.
Revised: Thus, for our experiments, we introduce partial observability toensure that our results are closer to the real world.

Original: We study in detail Contextual Markov Decision Process (CMDPs)
Revised: We study in detail Contextual Markov Decision Process (CMDP)

Original: [Hallak et al., 2015] that satisfyan asymptotic realizability condition.
Revised: [Hallak et al., 2015] that satisfyan identifiability condition.

Original: Intuitively, this means we can expect that proper utilizationof history to eventually enable accurate prediction of the context.
Revised: Intuitively, this means we can expect that proper utilization of historyeventually enables accurate prediction of the context.

Original: A key result we show is that foridentifiable CMDPs where the learner can recover from mistakes early on in an episode, on-policyimitation learning algorithms that operate in the space of histories are able to asymptotically matchtime-averaged expert value, while off-policy approaches struggle to do so.
Revised: A key result we show is that for identifiable CMDPs where the learner can recover from mistakes early on in an episode, on-policy imitationlearning algorithms that operate in the space of histories are able to asymptotically match expertvalue, while off-policy approaches struggle to do so.

Original: More concretely, our work makes three contributions:
Revised: More concretely, our work makes three contributions:

Original: We show that under appropriate identifiability and recoverability conditions, the contextdependent expert policy becomes asymptotically realizable , enabling on-policy imitation
Revised: We show that under appropriate identifiability and recoverability conditions, the context- dependent expert policy might become asymptotically realizable , enabling on-policy imita-

Original:  learning algorithms to match (or nearly match)time-averaged expert performance.
Revised: tion learning algorithms to match (or nearly match) expert performance.2.

Original: More generally, we show that when longer history allows the learner to get closer to realizing

Original: the expert policy, on-policy methods are able to take advantage of this property while offpolicy methods are stuck with the consequences of their mistakes early on in an episode.

Original: This manifests as off-policy methods producing policies that merely repeat previous actions. We conduct experiments in a simplified bandit domain which show that there exist sharp phase transitions in terms of when off-policy imitation learning algorithms match expert performance in contrast to the uniform value-equivalence of the policies produced by onpolicy approaches.
Revised: We conduct experiments in a simplified bandit domain which show that there exist sharpphase transitions in terms of when off-policy imitation learning algorithms equipped withhistory work in contrast to the uniform value-equivalence of the policies produced byon-policy approaches.3.

Original: Wealso conduct experiments on continuous control tasks that show that on-policy algorithms are able to take advantage of history to correctly identify the context ina way that off-policy methods are not.
Revised: We conduct experiments on continuous control tasks that show that on-policy algorithmsare able to take advantage of history to correctly identify the context in a way that off-policymethods are not.

Original: We begin with a discussion of related work.
Revised: We begin with a discussion of related work.

Original:  Related Work
Revised:  Related Work

Original: One of the fundamental challenges of imitation learning (or any sequential prediction task where thelearner consumes some function of its own prior predictions) is the likelihood of significant covariateshift between training-time data and test-time observations
Revised: One of the fundamental challenges of imitation learning (or any sequential prediction task where thelearner consumes some function of its own prior predictions) is the likelihood of significant covariateshift between training-time data and test-time observations

Original: [Ross and Bagnell, 2010].
Revised: [Ross and Bagnell, 2010].

Original: In short, thishappens because the learner might end up in states not seen in the demonstrations and is thus unsurehow to act.
Revised: In short, thishappens because the learner might end up in states not seen in the demonstrations and is thus unsurehow to act.

Original: Early work in this area includes that of Daumé et al.
Revised: Early work in this area includes that of Daumé et al.

Original: [2009] in the natural languageprocessing and that of Ross et al.
Revised: [2009] in the natural languageprocessing and that of Ross et al.

Original: [2010] inimitation learning and robotics, both of which come tothe preceding conclusion.
Revised: [2010] in robotics, both of which come to the preceding conclusion.

Original: Recent work by Spencer et al.
Revised: Recent work by Spencer et al.

Original: [2021] shows that there are actually morethan one regime of covariate shift.
Revised: [2021] shows that there are actually more than one regime of covariateshift.

Original: In the “easy” regime where the expert is realizable, off-policymethods like behavior cloning match expert performance when data and model capacity are largeenough.
Revised: In the “easy” regime where the expert is realizable, off-policy methods like behavior cloningmatch expert performance when data and model capacity are large enough.

Original: However, in harder regimes where the expert is non-realizable due to model misspecification,off-policy methods compound in error and one must rely on on-policy methods, either those thatrequire an interactive expert [Ross et al., 2010] or an interactive simulator [Ziebart et al., 2008, Swamy et al., 2021].
Revised: However, in harderregimes where the expert is non-realizable due to model misspecification (or effectively so due todata scarcity), off-policy methods compound in error and one must rely on on-policy methods, eitherthose that require an interactive expert [Ross et al., 2010] or an interactive simulator Ziebart et al.[2008], Swamy et al.

Original: One  instance of model misspecificationof practical interest is when the learner is denied stateinformation that the expert uses.
Revised: One practically important instance of model misspecification is when the learner is denied stateinformation that the expert uses.

Original: For instance, in self-driving, the human expert has richer contextabout the scene than the limited perception system of the car.
Revised: For instance, in self-driving, the human expert has richer contextabout the scene than the limited perception system of the learner.

Original: While the standard solution is to adda history of past states and actions to the model, practitioners have often noted that this leads to a
Revised: While the standard solution is toadd a history of past states and actions to the model, practitioners have often noted that accelerations.

Original: “latching effect” where the learner simply repeats the past action.

Original: For example, Muller et al.

Original: [2006]note such latching with steering actions, Kuefler et al.

Original: [2017], Bansal et al.

Original: [2018] note this withbraking actions, and Codevilla et al.

Original: [2019] with accelerations.

Original: Recent work by Ortega et al.
Revised: Recent work by Ortega et al.

Original: [2021]also points out this latching behavior which they term as “self-delusion.”
Revised: [2021] also points out this latching behavior which theyterm as “self-delusion”.

Original: Once one identifies the downstream effect of missing context as covariate shift, a natural questionis whether an extension of prior covariate-shift-robust imitation learning methods to the space ofhistories would be able to learn effectively in partial information settings.
Revised: Once one identifies the downstream effect of missing context as covariate shift, a natural questionis whether an extension of prior covariate-shift-robust imitation learning methods to the space ofhistories would be able to learn effectively in confounded settings.

Original: Prior work has answeredparts of this question.
Revised: Prior work has answered parts ofthis question.

Original: For example, Choudhury et al.
Revised: For example, Choudhury et al.

Original: [2018] proves that interactive imitation learningover the space of histories converges to the QMDP approximation of the expert’s policy [Littmanet al., 1995].
Revised: [2018] proves that interactive imitation learning overthe space of histories converges to the QMDP approximation of the expert’s policy [Littman et al.

Original: Recent work Ortega et al.
Revised: In fact, the proposed algorithm of Ortega et al.

Original: [2021]views these kind of partial information setting througha causal lens [Pearl et al., 2016], and provides an algorithm (counterfactual teaching) equivalent tothe interactive-expert FORWARD algorithm of Ross and Bagnell [2010] under log-loss.
Revised: [2021] (counterfactual teaching) is identical tothe earlier FORWARD algorithm of Ross and Bagnell [2010] under log-loss.

Original: We buildupon these analyses by providing conditions under which such approaches will converge to a policythat is equivalent in value to that of the expert in the presence of unobserved contexts.
Revised: Our main contributionprovides conditions under which such approaches will converge to a policy that is equivalent in valueto that of the expert in the presence of unobserved contexts.

Original: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,
Revised: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,

Original: Kumor et al., 2021], context that is updated throughout the episode and is only reflected for a singlestep prevents the learner from refining its estimates via considering history.
Revised: Kumor et al., 2021], context that is updated throughout the episode and is only reflected for a singlestep prevents the learner from refining its estimates via considering history.

Original: Our results apply tosettings beyond the CMDP where a similar asymptotic realizability property holds.
Revised: We would expect ourresults to generalize to settings beyond the CMDP where a similar asymptotic realizability propertyholds true.

Original: Tennenholtz et al.[2021] consider IL in contextual MDPs but give the learner access to the confounder at test time anddo not consider policies that operate over the space of histories.de Haan et al.
Revised: Tennenholtz et al. [2021] consider IL in contextual MDPs but give the learner access tothe confounder at test time and do not consider policies that operate over the space of histories.

Original: [2019] also considerimitation learning from a causal inference perspective but focus on issues of covariate shift [Spenceret al., 2021], rather than those of missing information.

Original: Consider a finite-horizon Contextual Markov Decision Process (CMDP) parameterized by⟨S , A , C , T , r, T ⟩ where S , A , C are the state, action, and context spaces, T : S × A × C → ∆( S ) 115 is the transition operator, r : S ×
Revised: Recently, Swamy et al.

Original: A × C → [ − 1 , 1] is the reward function, and T is the horizon. At 116 the beginning of each episode, a context is sampled from p ( c ) and  isheld fixed until the next reset. Intuitively, there exists a family of reward and transition functions that are indexed by the context foreach episode. We use h t ∈ H to denote a t -step history: ( s 1 , a 1 , . . .
Revised: [2021] proved that for a wide class of interactive expert imitation learningalgorithms, one should expect learner sub-optimality to grow as O ( ϵHT ) , where T is the horizon ofthe problem and H is a recoverability constant that upper bounds the number of steps it takes for theexpert to recover from an arbitrary learner mistake Leike et al.

Original: , s t ) .
Revised: [2016], Ross et al.

Original: We see trajectories generated by an expert policy π E : S × C → ∆( A ) and search over time-varying policies π : H→ ∆( A ) . 120 We use π 1 ...t to refer to  the sequence of policies that comprise this time-varying policy. We assumethat our policy class Π is convex and compact.
Revised: We arguethat for infinite horizon problems, one requires a finite H to argue value equivalence for the policythat an on-policy learner converges to – otherwise, a mistake early on can doom the learner for therest of the horizon.  Imitation Learning with Unobserved Contexts

Original: We begin with a toy example of the latching effect and how it leads to poor policy performance.
Revised: We begin with a toy example studied by Ortega et al. 3.1 Phase Transitions for Off-Policy Imitation Learning

Original: Problem 3.1 (Causal Bandit Problem, [Ortega et al., 2021]) .
Revised: Problem 3.1 (Causal Bandit Problem) .

Original: Consideran episodic MDP with Kactions (arms) and a single state.
Revised: Consider a K -armed bandit.

Original: At the beginning of each episode, a context c ∈ [ K ] is chosenuniformly at random and represents the correct arm for that episode.
Revised: At the beginning of each episode,a context c ∈ [ K ] is chosen uniformly at random and represents the correct arm for that episode.

Original: Pulling an arm leads to binaryfeedback: + if the arm was correct and − otherwise.
Revised: Pulling an arm leads to binary feedback: + if the arm was correct and − otherwise.

Original: This feedback is flipped with probabilityϵ obs ∈ [0 , 1] .
Revised: This feedbackis flipped with probability ϵ obs ∈ [0 , 1] .

Original: The learner observes expert demonstrations where at each timestep, the expert plays thecorrect arm with probability 1 − ϵ exp ∈ [0 , 1] and another arm uniformly at random otherwise.
Revised: The learner observes expert demonstrations where at eachtimestep, the expert plays the correct arm with probability 1 − ϵ exp ∈ [0 , 1] and another arm uniformlyat random otherwise.

Original: Thereward function is 1 for pulling the correct arm and 0 otherwise.
Revised: The reward function is 1 for pulling the correct arm and 0 otherwise.

Original: We emphasize that the learner doesnot observe the rewards, just noisy binary feedback as an observation after each pull.
Revised: Weemphasize that the learner does not observe the rewards, just noisy binary feedback as an observationafter each pull.

Original: As the learner does not observe the correct arm, we are in the partial information setting.
Revised: [Figure]

Original: However, asone might expect, by pulling all arms enough times and observing the noisy feedback, the learnercan narrow down which arm they should pull for the rest of the episode.
Revised: Intuitively, by tabulating the noisy observations, the learner should be able to narrow down what thecorrect arm is.

Original: Note that the learner staysin the same state after each action so they are free to perform this exploration without long-termconsequences. Observation 1: Off-Policy Methods Have Consistency Phase Transitions.
Revised: We focus on this problem as it is trivially recoverable – the learner is reset to the samestate after each action so they are free to explore without consequences.

Original: In Fig.
Revised: In Fig.

Original: 1, we plot,for a variety of settings of the problem parameters ( ϵ exp , ϵ obs ) whether the learner makes mistakeswith a significantly higher probability than the expertdoes for behavioral cloning (an off-policyalgorithm) and DAgger (an on-policy algorithm).
Revised: 1, we plot, for avariety of settings of the problem parameters ( ϵ exp , ϵ obs ) whether the learner makes mistakes with asignificantly higher probability than the expert for behavioral cloning (an off-policy algorithm) and DAgger (an on-policy algorithm which utilizes a queryable expert).

Original: We give all learners access to the full history ofinteractions (i.e. arms pulled and noisy feedback observed).
Revised: We give all learners access to thefull history of interactions (i.e. arms pulled and noisy feedback observed).

Original: With exactly ϵ obs = 0 .
Revised: With exactly ϵ obs = 0 .

Original: 5 , the learnergets no information from the observations, preventing any algorithm from learning properly.
Revised: 5 , the learner gets no information from observation, preventing any algorithmfrom learning properly.

Original: Thismeans that under this particular setting, the problem instance is not identifiable , a concept we developfurther below.
Revised: This means that under this particular setting, the problem instance is notidentifiable , a concept we formalize below.

Original: In contrast to the uniform consistency of DAgger whenever the problem is identifiable,

Original: 0.8 e exp 0.

Original: e o b s

Original: Consistency of π bc 0.

Original: 0.8 e exp 0.

Original: e o b s

Original: Consistency of π dagger    2000 t 0.

Original: P r o b . o f C o rr e c t A r m e obs = 0.3, e exp = 0. π dagger π bc 1 − e exp 1/ K we see sharp phase transitions in terms of where where BC is consistent – a small change to eitherproblem parameter can lead to a drastically different result.
Revised: In contrast to the uniform consistency of DAgger wheneverthe problem is identifiable, we see sharp phase transitions in terms of where where BC is consistent –a small change to either problem parameter can lead to a drastically different result.

Original: Observation 2: Off-Policy Methods Produce Latching Policies.

Original: At the first timestep, all learnerspick an arm uniformly at random as they have no information about the context.

Original: In the cases where

Original: BC is inconsistent, it continues to pick this arm ad infinitum as it treats its own past actions as thoughthey were the expert’s. This is what leads to the 1 K = 15 success rate seen in the rightmost part of Fig.1, even after 2000 timesteps of experience telling the learner that another arm should be pulled. Putdifferently, the BC learner collapses its uncertainty over the correct arm too quickly forthe negativefeedback it receives to push it to a different arm. Concerningly, this effect appears to be extremelysensitive to the parameters of the problem, rendering it difficult to predict and hedge against.
Revised: This means thatit can be rather hard to predict whether BC will work for a given problem, rendering it dangerous toapply in practice.

Original: Observation 3: Low Density Ratios Help Off-Policy Methods.

Original: Looking at the left side of Fig.1, a natural question might be why, as we increase ϵ exp , BC is consistent for a wider spectrum ofϵ obs values.

Original: Observe that as we increase ϵ exp , the expert has a higher chance of making a mistake,bringing its trajectory distribution closer to that of the uniform policy and lowering the maximumdensity ratio between learner and expert trajectory distributions.
Revised: As we increase ϵ exp , the expert has a higher chance of making a mistake, bringing its trajectorydistribution closer to that of the uniform policy and lowering the worst-case density ratio betweenlearner and expert trajectory distributions.

Original: As was established by Spencer et al.[2021], a low maximum density ratio is a sufficient condition for behavioral cloning to be consistent.
Revised: As was established by Spencer et al. [2021], a low densityratio is a sufficient condition for behavioral cloning to be consistent.

Original: This experiment appears to echo their theoretical results.
Revised: Our experiments echo thiscondition as a higher ϵ exp leads to greater flexibility in terms of choice of ϵ obs while maintainingconsistency.

Original: Putting together Fig.
Revised: The symmetry around the ϵ obs = 0 . 5 axis is due to the fact that the learner can stillfigure which arm is different from the rest on the basis of negated feedback.

Original: 1, on-policy approaches appear to be able to handle hidden context given accessto history while off-policy  can do so only in a way that depends heavily on problem parameters.A Bayesian Perspective on the Latching Effect
Revised: The main take-away from this experiment is that on-policy approaches appear to be able to handlehidden context given access to history, in contrast to off-policy approaches, which can do so but in away that depends heavily on problem parameters.

Original: To begin to explain this difference in behavior of on-policy and off-policy algorithms, we considerthe structural causal model (SCM) each of these classes of algorithms is implicitly assuming.
Revised: To begin to explain this difference in behavior ofon-policy and off-policy algorithms, we take a brief detour into the structural causal model (SCM)each of these classes of algorithms is implicitly assuming.

Original: 4.1 SCMs for Imitation Learning
Revised: 3.2 A Graphical Formalism for Imitation Learning

Revised: Consider a finite-horizon Contextual Markov Decision Process (CMDP) parameterized by⟨S , A , C , T , r, T ⟩ where S , A , C are the state, action, and context spaces, T : S × A × C → ∆( S ) 154 is the transition operator, r : S ×

Revised: A × C → [ − 1 , 1] is the reward function, and T is the horizon.

Revised: At 155 the beginning of each episode, a context is sampled from p ( c ) and is held fixed until the next reset.

Original: We observe that expert trajectories of length t are generated according to 
Revised: Intuitively, there exists a family of reward and transition functions that are indexed by the context foreach episode. We use h t ∈ H to denote a t -step history: ( s 1 , a 1 , . . . , s t ) . We see trajectories generated by an expert policy π E : S × C → ∆( A ) and search over time-varying policies π : H → ∆( A ) .

Revised: 159 We use π 1 ...t to refer to the sequence of policies that comprise this time-varying policy.

Revised: We assumethat our policy class Π is convex and compact.

Revised: Expert trajectories of length t are generated according to

Original: [Equation]
Revised: [Equation]

Original: 1) while learner trajectories are generated according to
Revised: while learner trajectories are generated according to

Original: [Equation]
Revised: [Equation]

Original: We use τ ∼ π E and τ ∼ π to denote T -step trajectories sampled according to the above distributions.
Revised: We use τ ∼ π E and τ ∼ π to denote T -step trajectories sampled according to the above distributions.

Original: We define our value and Q functions as usual: V π ( s ) = E [Equation]
Revised: The key difference between on-policy and off-policy imitation learning algorithms is the differencebetween the center and right of Fig. 2:

Original: ∼ [Equation] |
Revised: ) on policy assume the data they have seen thusfar is generated by executing learner π while policy assumes it is generated by executingthe expert policy π E .

Original: [Equation] = s [ [Equation]
Revised: The off-policy approximation is what leads to the latching behavior observedempirically: even if the first action was chosen at random, by treating it as though it was produced bythe expert π E (who sees the context c and picks actions that are correlated across time), the learner islikely to continue to repeat the past action.

Original: [Equation] s [Equation] performance be sssaaac
Revised: This is also what we observe empirically in the causal sssaaac

Original: T
Revised: T

Original: T π E π E π E π E π E(a) τ ∼ π E hhhaaac
Revised: T π E π E π E π E π E hhhaaac

Original: T
Revised: T

Original: T πππ(b) On-Policy hhhaaac
Revised: T πππhhhaaac

Original: T
Revised: T

Original: T π E π E ππ E (c) Off-Policy

Original: The key difference between on-policy and off-policy imitation learning algorithms is the differencebetween the center and right SCMs of Fig.

Original: On-policy algorithms assume the data they have seenthus far is generated by executing learner π (b) while off-policy algorithms assume it is generatedby executing the expert policy

Original: π E (c).

Original: The off-policy approximation is what leads to the latchingbehavior observed empirically: even if the first action was chosen at random, by treating it as though itwas produced by the expert π E (who sees the context c and therefore picks actions that are correlatedacross time), the learner is likely to continue to repeat the past action . This is also what we observeempirically in the causal bandit problem in the cases where behavioral cloning does not work: thelearner continues to play the first arm it chose, ignoring the feedback it gets that it is repeatedlymaking a mistake, and only matching expert performance on a 1 K of episodes. We note this places anoff-policy learner in a Catch-22 of sorts: they need to use history to narrow down the context but ifthey do, they can learn a naive latching policy that performs poorly at test-time. 4.2 Off Policy Methods Have an Incorrect Context Posterior
Revised: T π E π E ππ E bandit problem in the cases where behavioral cloning does not work: the learner continues to play thefirst arm it chose, ignoring the feedback it gets that it is repeatedly making a mistake.

Original: [Figure] Another way of seeing this point is by considering what the posterior over the confounder would beunder samples from each of these SCMs.
Revised: Another way of seeing this point is by considering what the posterior over the confounder would beunder samples from each of these SCMs.

Original: If the learner was able to accurately pin down the correctarm, they would be able to easily reproduce the expert policy.

Original: Thus, we can focus on correct-armidentification via Bayes Rule and assuming a uniform prior over contexts
Revised: Via Bayes Rule and assuming a uniform prior over contexts,we know that

Original: [Equation]
Revised: [Equation]

Original: Under the off-policy graphical model, p ( c, h t ) ∝ p ( τ ; π E ) : the probability of a history h t under theexpert’s distribution (Fig. 2, (a)).
Revised: Under the off-policy graphical model, p ( c, h t ) ∝ p ( τ ; π E ) – the probability of a history h t under theexpert’s distribution (the left SCM in Fig. 2).

Original: Expanding terms, we arrive atthe following expression
Revised: Expanding terms, we arrive at

Original: Proposition 4.1.

Original: The off-policy posterior over contexts is

Original: [Equation]
Revised: [Equation]

Original: Notice how under the expert’s generative model, the context c directly influences the actions soconditioning actions when attempting to predict c is correct.
Revised: Notice how under the expert’s generative model, the context c directly influences the actions so it isuseful to condition on them when attempting to predict c .

Original: We highlightin green the term thatencodes this dependence.
Revised: We highlight this term that encodes thisdependence in green.

Original: In contrast, in the on-policy graphical model (Fig. 2, center), treating theactions as interventions (as in Fig. 3) leads to  the correct posterior over contexts (if we assume thelearner has no privileged access to the context except via its influence on history).
Revised: In contrast, in the on-policy graphical model (Fig. 2, center), we could treatthe actions as interventions rather than as variables with a path to c and still end up with the sameposterior over contexts.

Original: Formally, Proposition 4.2. The on-policy posterior over contexts is
Revised: Mathematically,p

Original: [Equation]
Revised: [Equation]

Original: where the equality follows from the fact that
Revised: where the equality follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: [Equation] and the standard do () -calculus rules [Pearl et al., 2016].
Revised: (6) and the standard do () -calculus rules [Pearl et al., 2016].

Original: Intuitively, we are leveraging the fact thatthe learner’s actions have all their dependence on the context mediated through the history of states toignore them in our posterior calculation.
Revised: Intuitively, we are leveraging the fact thatthe learner’s actions have all their dependence on the context mediated through the history of states toignore them in our posterior calculation.

Original: Notice that this expression matches the off-policy posteriorexcept for the term in green: because the on-policy learner knows that the actions in the history werenot produced by the expert, it does not weight them by the expert’s probability of playing them in itsposterior update .
Revised: Notice that this expression matches the off-policy posteriorexcept for the term in green – because the on-policy learner knows that the actions in the history werenot produced by the expert, it does not weight them by the expert’s probability of playing them in it’sposterior update.

Original: Graphically, as far as the posterior over the context is concerned, we could be inthe following SCM.
Revised: Graphically, as far as the posterior over the context is concerned, we could be in thefollowing SCM:

Original: [Figure]
Revised: [Figure]

Original: By ignoring its own incorrect actions early on, the on-policy learner does not learn to merely repeatthe first thing it tried nor does it over-index on them and come to a false conclusion as to what thevalue of the context is.
Revised: By ignoring the incorrect actions early on, the on-policy learner does not learn to merely repeat thefirst thing it tried.

Original: The addition or removal of the green term can lead to markedly differentresults.
Revised: While this single-term might seem rather unimportant, it can lead to markedlydifferent results.

Original: For example, the learners we used to generate Fig. 1 had policies given by
Revised: In fact, the learners we used to generate Fig. 1 had policies given by

Original: [Equation]
Revised: [Equation]

Original: 7) and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that these policies are exactly equivalent except for the green term in the posterior of theoff-policy π bc , yet respond quite differently as the parameters of the problem are changed.
Revised: Note that these policies are exactly equivalent except for the green term in the posterior of the offpolicy π bc , yet have markedly different consistency properties as the parameters of the problem arechanged.

Original: Also notethat p on ( c | h t ) requires interaction with the environment to evaluate, while p off ( c | h t ) does not.
Revised: We now turn our attention to generalizing this argument beyond likelihood-based learners(i.e. those that match sufficient statistics of expert behavior – moments – rather than maintainingexplicit posteriors) and providing value equivalence guarantees for on-policy learners in the limit ofinfinite timesteps.

Original: A Moment-Matching Solution to the Latching Effect
Revised: 3.3 Asymptotic Realizability in Imitation Learning

Original: We now turn our attention to generalizing this argument beyond Bayesian learners, including tothose that match sufficient statistics of expert behavior – moments – rather than maintaining explicitposteriors, and providing value equivalence guarantees for history-equipped on-policy learners.
Revised: One might hope that with the use of history and the on-policy graphical model, one might be able toeventually match expert performance in contextual problems. This is an example of a more generalphenomenon we term asymptotic realizability (AR), in which the learner is able to perform as well asthe expert does with high probability after observing an arbitrary history of some length.

Original: 5.1 A Quick Review of Moment-Matching in Imitation Learning
Revised: We beginby defining our goal in AR problems: Definition 3.2 (Asymptotic Value Equivalence (AVE)) . We say that policy π is asymptoticallyvalue-equivalent (AVE) when the following condition holds true as T → ∞ :

Original: First, as in Swamy et al.
Revised: [Equation] Intuitively, this condition means that the learner performs as well as the expert does on average, givenenough time. In the finite horizon, we use average imitation gap (AIG) to refer to the followingquantity: [Equation]

Revised: A sufficient condition for AVE is that the AIG goes to 0 as T → ∞ .

Revised: We will now study the AIGproperties of on-policy and off-policy imitation learning algorithms.

Revised: While up to this point wehave focused on likelihood-based methods (e.g. behavioral cloning or history-based methods thatmaintain an explicit posterior and perform Bayesian filtering), our theory generalizes to the broaderclass of moment-matching algorithms, which includes, beyond the previously discussed approaches,

Revised: AdVIL/AdRIL/DAeQuIL [Swamy et al., 2021], GAIL

Revised: [Ho and Ermon, 2016], and MaxEnt IRL[Ziebart et al., 2008].

Revised: First, as in Swamy et al.

Original: [2021], we define F Q E as the set of on- Q moments , with f ∈ F Q E satisfying 220 type signature
Revised: [2021], we define F Q E as the set of on- Q moments , with f ∈ F Q E satisfying 221 type signature

Original: S ×
Revised: S ×

Original: A × C → [ − T, T ] .
Revised: A × C → [ − T, T ] .

Original: Intuitively, F Q E spans the set of possible expert Q -functions.
Revised: Intuitively, F Q E spans the set of possible expert Q -functions.

Original: 221 We require the actual expert Q -function to be contained: Q π E ∈ F Q E .
Revised: We require the actual expert Q -function to be contained: Q π E ∈ F Q E .

Original: We assume that F Q E is222 convex,  compact, closed under negation, and finite dimensional.
Revised: We assume that F Q E is convex, 223 compact, closed under negation, and finite dimensional.

Original: Second, define
Revised: We use

Original: F Q be the class of 223 off- Q moments (i.e.
Revised: F on [Equation] t )

Original: ∀ π ∈ Π , Q π ∈ F Q ) and satisfy the same assumptions as F Q E .
Revised: = { f/ [Equation]

Original: Third, let F r 224 denote the class ofreward moments and also satisfy the same function-class assumptions. All f ∈ F r 225 satisfy type signature
Revised: T − t ) : f ∈ F Q E } 224 to refer to a scaled version of the on- Q moments. We define two key quantities:

Original: S × A × C → [ − 1 , 1] and we assume that r ∈ F r . Swamy et al.
Revised: [Equation]

Original: [2021] prove that if one is able to approximately solve a two-player zero-sum momentmatching game between a policy player (that picks from Π ) and a discriminator (that picks from F Q E , F Q , or F r ), one has a bound on the performance difference between the learner and the expert. Depending on which class the discriminator selects from, one ends up with a different bound. For example, if one solves the game with the following payoff and f ∈ F r to an ϵ -approximate Nashequilibrium,
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: one has a guarantee that J ( π E ) − J ( π )≤ ϵT . Unfortunately, directly solving such a game isnotpossible in the hidden context setting as we do not see the contexts and therefore cannot evaluate themoment functions. 1 We turn our attention to adapting moment-matching to the contextual setting.
Revised: 11) where ϵ on ( t ) is the moment matching error at time t between actions selected by the expert and thelearner on states visited by the learner, H is a moment recoverability constant which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake Leike et al.

Original: For each moment class, we assume the existence of an observable moment class that operates overthe space of histories (i.e. H × A → R ) and has members that eventually produce outputs close to 238 that of their context-dependent counterparts.
Revised: [2016], Rosset al.

Original: In math, Assumption 5.1 (Asymptotic On- Q Moment Identifiability) . [Equation] where τ ∼ π 1 · π 2 denotes a trajectory drawn by following π 1 until timestep T − 1 and then switching 241 to π 2 .
Revised: [2010], Swamy et al.

Original: Analogous conditions can be defined for F r and F Q . We note that these conditions are 242 asymptotic in nature – we use δ ( t ) to refer to the finite-horizon expected difference between theobservable and context-dependent moments.
Revised: These can be seen as the natural history-space generalizations ofthe analogous concepts in Swamy et al.

Original: We use H to denote the moment recoverability constant ,which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake[Swamy et al., 2021]: [Equation] [Equation]
Revised: [2021] and govern the performance of interactive imitationlearning algorithms.

Original: For problems where the expert is able to effectively correct learner mistakes, this quantity can besignificantly smaller than the horizon.
Revised: A similar moment-matching error can be defined for off-policy algorithms, this time computed onexpert rollouts and as a function of learner Q moments.

Original: Define ˜ F on = { f/ 2 H : f ∈ ˜ F Q E } and ˜ F off  = { f/ 2 T  :  f ∈˜ F Q } to be scaled-down versions of the observable moments such that their range is [ − 1 , 1] .
Revised: Let F Q be the class of learner Q momentsand satisfy the same assumptions as F Q E . Analogously to above, we define F off ( t ) = { f/ 2( T − t ) : 236 f ∈ F Q } to refer to a scaled version of the off- Q moments.

Original: We cannow define our moment-matching errors: [Equation] [Equation]
Revised: We can now define our off-policy 237 moment matching error:

Original: [Equation]
Revised: [Equation]

Original: As argued by Swamy et al. [2021], ϵ on ( t ) governs the performance of on- Q algorithms like DAgger[Ross et al., 2010], ϵ off ( t ) the performance of off- Q algorithms like behavioral cloning [Pomerleau,1989], and ϵ rew ( t ) the performance of reward-matching algorithms like MaxEnt IRL [Ziebart et al.
Revised: We can also define a moment-matching error for online imitation learning algorithms (i.e. thosethat have access to the environment but not a queryable expert like GAIL [Ho and Ermon,

Original: and GAIL [Ho and Ermon, 2016].
Revised: 2016]or MaxEnt IRL

Original: With these definitions laid out, we can now prove how welleach class of algorithms handles unobserved contexts. 5.3 Asymptotic Realizability in Imitation Learning
Revised: [Ziebart et al., 2008]) in terms of reward moments that span the class of rewardfunctions. Let f ∈ F r satisfy type signature S

Original: For some partial information problems, the use of history coupled with on-policy feedback mightallow the learner to eventually match expert performance. This is an example of a more generalphenomenon we term asymptotic realizability (AR), in which the learner is able to perform as well asthe expert does with high probability after observing an arbitrary history of some length. We beginby defining the average imitation gap or AIG for short
Revised: × A × C

Original: Definition 5.2 (AIG( π , T)) .[Equation] [Equation]
Revised: → [ − 1 , 1] and assume that r ∈ F r .

Original: We now defineour performance target in AR problems. Definition 5.3 (Asymptotic Value Equivalence (AVE)) . We say that policy π is asymptoticallyvalue-equivalent (AVE) when the following condition holds true: [Equation]
Revised: We 242 define the reward moment-matching error as

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this condition means that the learner performs as well as the expert does on average, givenenough time.

Original: Put differently, we do not penalize the learner for initial mistakes as long as they areable to learn enough from them to match expert performance.

Original: We will proceed by studying the AIGproperties of on-policy and off-policy imitation learning algorithms.

Original: We are now ready to state our main result:
Revised: We are now ready to state our main results:

Original: Theorem 5.4 (AVE of IL Algorithms) .
Revised: Theorem 3.3 (Interactive AVE) .

Original: Define lim sup t →∞ ϵ on ( t ) = ϵ on ( ∞ ) , lim T →∞ (cid:80) T t ϵ off ( t ) +δ off ( t ) =
Revised: For all MDPs and π s.t.

Original: Σ off ( ∞ ) , and lim sup t →∞ ϵ rew ( t ) =
Revised: lim sup t →∞ ϵ on ( t ) =

Original: ϵ rew ( ∞ ) .
Revised: ϵ on ( ∞ ) ,

Original: For all (C)MDPs and π ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation] [Equation] [Equation] In words, if either an on- Q or reward-matching learner is able to achieve 0 moment matching errorasymptotically, they will achieve the same average value as the expert.
Revised: Thus, if ϵ on ( ∞ ) = 0 , the on-policy learner will achieve AVE.

Original: Even if they are unable to doso, the AIG will be bounded by a constant.
Revised: Even if it does not, the AIG will bebounded by a constant.

Original: In contrast,  the  result for off-policy algorithms is muchweaker.
Revised: In contrast, we have the following result for off-policy algorithms:

Original: Notice that instead of taking a lim sup which, roughly speaking, captures the asymptoticerror, the off-policy bound is in terms of the sum of errors, which factors in errors made early on.

Original: This result indicates that an off-policy learner that makes mistakes early on in the episode (as is likelyfor contextual problems) might be unable to eventually match expert performance, even when theproblem is recoverable.

Original: We prove that there exist certain problems for which this bound is tight.

Original: Theorem 5.5 (Off-Policy AVE Lower Bound) .
Revised: Theorem 3.4 (Off-Policy AVE) .

Original: There exist CMDPs and π s.t.
Revised: For all MDPs and π s.t.

Original: lim sup t →∞ ϵ off (t ) =
Revised: lim sup t →∞ ϵ off (

Original: 0for which
Revised: [Equation] off ( [Equation]

Original: [Equation]
Revised: [Equation]

Original: In words, this theorem says that for certain problems, even if an off-policy learner can drive downerror asymptotically, they might be doomed as far as AVE because of mistakes made earlier on.
Revised: This result indicates that unless an off-policy learner is able to reduce its moment-matching error at arate ∝ 1 T , the learner will have a growing AIG and therefore not achieve AVE. We see a similar resultfor online IL algorithms to that for interactive IL algorithms: Theorem 3.5 (Online AVE) . For all MDPs and π

Original:  We prove these results in the appendix.
Revised: [Equation] [Equation] [Equation] We prove these results in the appendix.

Original: To come full circle, we now argue that CMDPs (including ourcausal bandit problem) that satisfy the asymptotic moment identifiability conditions along with anasymptotic realizability condition  on the policy class enable the learner to match expert performance:
Revised: To come full circle, we now argue that a subset of CMDPs(including our causal bandit problem) satisfies a condition that makes it possible to drive ϵ off ( ∞ ) ,ϵ on ( ∞ ) , and ϵ rew ( ∞ ) to 0. We now make two assumptions that characterize this subset:

Original: Assumption 5.6 (Asymptotic Realizability) .
Revised: Assumption 3.6 (Asymptotic Identifiability) .

Original: Asymptotic Moment Identifiability holds and ∃ π ∈ Πs.t.
Revised: There exists a function g such that for any context c ,and for all t ∈ [ T ] , and any history of length t generated by any π ∈ Π , g ( h t ) =

Original: lim t →∞ ϵ on ( t ) = 0 , lim t →∞ ϵ off ( t )= 0 , and lim t →∞ ϵ rew ( t ) = 0 .
Revised: c with probability1 − δ ( t ) , where lim t →∞ δ ( t ) = 0 .

Original: Note this assumption is weaker than a standard realizability assumption – it is saying that alongcertain moments ofinterest, we can asymptotically match the expert. Plugging in this assumptioninto Theorem 5.4 tells us that on such problems, an on-policy learner will be able to achieve AVE.
Revised: This assumption means that regardless of the sequence of observations the learner receives, theywill eventually be able to pin down the confounder. One way to ensure this condition holds is toadd exploration noise to the learner’s actions such that they try all actions with sufficiently highprobability and assuming an underlying ergodicity of the MDP. Assumption 3.7 (Realizability) .

Original: We now turn our attention to efficiently computing such a policy. We prove thatby solving anapproximate equilibrium computation game over the space of history-based policies and the space ofobservable moments (which can be done efficiently with no-regret algorithms [Freund and Schapire,1997]), one can find a policy that achieves a low AIG.
Revised: Π contains π E ( s t , g ( h t )) , where g is defined as above. This is a relatively weak assumption which simply means that the learner can act as the expert wouldonce they have nailed down the confounder. Under these two assumptions, a CMDP is really just anexample of the more general phenomenon of AR.

Original: [Figure]
Revised: We prove the following result in the appendix:

Original: Theorem 5.7.
Revised: Theorem 3.8.

Original: For any contextual MDP and policy class that satisfies Asymptotic Realizability,let π be an ϵ -approximate Nash equilibrium strategy for the infinite horizon reward-matching oron- Q -matching game. Then, we know that lim T →∞ AIG( π, T ) ≤ ϵ or lim T →∞ AIG( π, T )≤ Hϵ . In short, by solving an on-policy moment-matching problem over policies that have access to history,we have strong guarantees of matching expert performance on asymptotically realizable problems. We re-iterate that we have no such guarantees for off-policy algorithms.
Revised: For any contextual MDP and policy class that satisfy the preceding two assumptions,any no-regret learner over Π will be able to drive ϵ on ( ∞ ) and ϵ rew ( ∞ ) to 0.

Original: We also note that our causalbandit problem satisfies these assumptions, providing theoretical justification for our results.
Revised: We also note that our causal bandit problem satisfies these assumptions:

Original: Corollary 5.8.
Revised: Corollary 3.9.

Original: There exists a singleton observable moment class such that the Causal Bandit Problem satisfies Asymptotic Realizability, implying that the iterates produced by an on-policymoment matching algorithm will achieve AVE.
Revised: The Causal Bandit Problem satisfies Asymptotic Identifiability and Realizability andhence achieve AVE for any on-policy moment matching algorithm.

Original: Putting it all together, for asymptotically realizable problems, on-policy imitation learning algorithms have stronger guarantees than their off-policy counterparts with respect to asymptotic valueequivalence.
Revised: In short, for asymptotically realizable problems, on-policy imitation learning algorithms have strongerguarantees than their off-policy counterparts with respect to asymptotic value equivalence.

Original: If a contextual MDP satisfies these conditions (like with our Causal Bandit Problem),the preceding results apply. We see that our theory matches our experiments: on-policy algorithmsappear to work on all identifiable instances of the causal bandit problems, while off-policy algorithmshave much weaker performance.
Revised: Contextual MDPs are a specific example of AR problems if they satisfy an identifiability condition. One specificexample of such a CMDP is our Causal Bandit Problem.

Original: We now turn our attention to another CMDP that satisfies ourassumptions to further validate our theory empirically.
Revised: We now turn our attention to another CMDPthat satisfies our assumptions to further validate our theory empirically.

Original:  Experiments
Revised:  Experiments

Original:  We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai[2016] that is inspired by the multi-task reinforcement learning setups of Finn et al. 
Revised: [Figure] We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai[2016] that draws heavily on the multi-task reinforcement learning setups of Finn et al. [2017] and Kumar et al.

Original: Inthese tasks, the agent is rewarded for running at a particular velocity that is randomly sampled at thebeginning of each episode.
Revised: In these tasks, the agent is rewarded for running at a particular velocity thatis randomly sampled at the beginning of each episode.

Original: We train expert policies that have access to this privileged information.
Revised: We train expert policies that have access tothis privileged information.

Revised: We append an indicator to the learner’s state of whether their currentvelocity is below or above the target velocity for the episode – by observing a few such indicators,one might hope that the learner is able to hone in on what their velocity should be via a sort of binarysearch.

Original: Wecompare two algorithms: the off-policy behavioral cloning ( BC )
Revised: We compare two algorithms: the off-policy behavioral cloning ( BC )

Original: [Pomerleau, 1989] and the on-policy DAgger ( DAgger )
Revised: [Pomerleau, 1989] andthe on-policy DAgger ( DAgger )

Original: [Ross et al., 2010] that either have access to the immediate state or the last fivetimesteps of history (for which we use the suffix -SEQ ).
Revised: [Ross et al., 2010]. We use two model classes. The base class is atwo-layer MLP with 256 hidden units and ReLu activations. The history-dependent class (for whichwe use the suffix -SEQ ) is an LSTM with a hidden unit size of 256 that is preceded by a 1-layer MLPwith 256 units and a ReLu activation.

Revised: We give the learner access to the last 5 timesteps of history.

Revised: Weemphasize that the expert operates only using the current state.

Original: Inthe bar plots of Fig. 4, we see that without access to history, BC performs poorly.
Revised: In Fig. 4 Left, we see that without access to history, BC performs poorly.

Original: As our theorypredicts, when we equip theBC learner with history, we actually see it perform worse on average,exhibiting the latching behavior that has been observed repeatedly in practice.
Revised: Interestingly, when weequip the learner with history, we actually see it perform worse on average, reflecting the shortof latching-style behavior that has been observed repeatedly in practice.

Original: In contrast, we see thatequipping our on-policy learner with access to the last few observations and actions does not lead toa sharp decline in terms of performance.
Revised: In contrast, we see thatequipping our on-policy learner with access to the last few observations and actions does not lead toa sharp decline in terms of performance.

Original: We use MSE as a surrogate for moment-matching error.
Revised: For the center and right plots, we use MSE as a surrogatefor moment-matching error.

Revised: In the center plot of Fig.

Original: Wesee that on both environments, on-policy methods are far better at matching expert actions on theirown rollout distribution.
Revised: 4, we see that, as expected, on-policy methodsare far better at matching expert actions on their own rollout distribution.

Original: Wealso see that with enough time, DAgger-SEQ is able to predict actionsbetter than DAgger . 
Revised: We see that with enough time, DAgger-SEQ is able to predict actions marginally better than DAgger . In Fig.

Revised: 4 right, we plotthe product of ϵ off ( t ) and t .

Revised: If this curve was roughly constant or decaying (reflecting an off-policymoment matching error that decays at or faster than 1 /t ), our theory would indicate that the learnerwould perform as well as the expert under an arbitrary reward function.

Revised: However, we actually see alinear growth , indicating that our off-policy learners should not achieve value equivalence, whichechoes what we see in the left plot.

Revised: Thus, we find that our theory accurately predicts these results.

Original: Putting together these observations, it appears as though on-policy methods withaccess to history are the most robust against partial information, agreeing with our theory.
Revised: We study imitation learning problems for which the expert has access to more information than thelearner. We find that as long as the learner can identify this missing information and recover from themistakes it made during identification, on-policy algorithms are able to match time-averaged expertvalue, in contrast to off-policy algorithms. We believe our results point towards a unified explanationof both empirical successes and failures.

Revised: Limitations.

Revised: Our theoretical analyses primarily focus on the so-called asymptotic regime where

Revised: T goes to infinity.

Revised: For many applications, finite time bounds are equally important.

Revised: There will alsobe value in deriving more refined finite-sample bounds.

Revised: In particular, it will be interesting to see ifthere are concrete structure in the problem instances that can replace our asymptotic identifiabilityassumption.

Revised: Moreover, our framework currently relies on the inherent randomness in the environmentto establish identifiability of the context.

Revised: While this is a natural assumption for many environments,we believe there is also a problem space of designing more sample-efficient exploration algorithmsthat leverage the structure of the underlying MDPs.

Revised: Societal impacts.

Revised: Our paper aims to address an important issue in data-driven decision-makingprocesses in the real world—that is, the data collection in many environments may miss contextualinformation that is critical to the problem.

Revised: Our work focuses on this issue in the domain of imitationlearning, and establishes the theoretical guarantees for each type of algorithms.

Revised: We believe that ourstudies will promote appropriate reliance on different types of algorithms when we deploy imitationlearning technologies in the real world.