Original: Causal Imitation Learning with Unobserved Contexts
Revised: Sequence Model Imitation Learning with Unobserved Contexts

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider imitation learning problems where the expert has access to a perepisode context that is hidden from the learner, both in the demonstrations and attest-time.
Revised: We consider imitation learning problems where the learner’s ability to mimic the expert increases throughout the course of an episode as more information is revealed.

Original: While the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions,they might be able to eventually identify the  context and act as the expert would.
Revised: One example of this is when the expert has access to privileged information: while the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions, they might be able to eventually identify the hidden context and act as the expert would.

Original: We show that on-policy imitation learning algorithms (with or without access to aqueryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods.
Revised: We prove that on-policy imitation learning algorithms (with or without access to a queryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods.

Revised: This is because onpolicy algorithms provably learn to recover from their initially suboptimal actions, while off-policy methods treat their suboptimal past actions as though they came from the expert.

Revised: This often manifests as a latching behavior: a naive repetition of past actions.

Original: We conduct experiments in a toy banditdomain that show that there exist sharp phase transitions of whether off-policyapproaches are able to match expert performance asymptotically, in contrast to theuniformly good performance of on-policy approaches.
Revised: We conduct experiments in a toy bandit domain that show that there exist sharp phase transitions of whether off-policy approaches are able to match expert performance asymptotically, in contrast to the uniformly good performance of on-policy approaches.

Original: We demonstrate that onseveral continuous control tasks, off-policy approaches arenot able to effectivelyuse history to reduce their uncertainty over the context, in contrast to on-policymethods.
Revised: We demonstrate that on several continuous control tasks, on-policy approaches are able to use history to identify the context while off-policy approaches actually perform worse when given access to history.

Original:  Introduction
Revised:  Introduction

Original: An unstated assumption in much of the work in imitation learning (IL) is that the learner and theexpert have access to the same state information.
Revised: An unstated assumption in much of the work in imitation learning (IL) is that the learner and the expert have access to the same state information.

Original: With powerful enough statistical models, thisassumption places the problem in the realizable setting – i.e. the imitator can actually behave likethe expert.
Revised: With powerful enough statistical models, this assumption places us in the realizable setting – i.e. the imitator can actually behave like the expert.

Original: In practice, however, an expert might have more information than the learner does.
Revised: In practice however, an expert might have more information than the learner does.

Original: Forexample, an expert policy might be trained in simulation with privileged access to state before beingused to supervise a learner policy that  has access to a subset of features, a recipethat has enjoyedsuccess in domains from motion planning with obstacles [Choudhury et al., 2018], to autonomousdriving [Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].
Revised: For example, an expert policy might be trained in simulation with privileged access to state before being used to supervise a learner policy that only has access to a subset of features. This recipe has enjoyed success in domains from motion planning with obstacles [Choudhury et al., 2018], to autonomous driving [Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].

Original: Recent theoretical work has established “no-go” results for successfully imitating an expert that hasaccess to more information [Zhang et al., 2020, Kumor et al., 2021].
Revised: Recent theoretical work has established “no-go” results for successfully imitating an expert that has access to more information [Zhang et al., 2020, Kumor et al., 2021].

Original: The core of their argumentsis that without seeing some feature that influences expert behavior but is not echoed elsewherein the state, the learner might not be able to properly ground the expert actions in the observedstate.
Revised: The core of their arguments is that without seeing some feature that inﬂuences expert behavior but is not echoed elsewhere in the state, the learner might not be able to properly ground the expert actions in the observed state.

Original: In causal inference terms, this hidden information acts as an unobserved confounder whichprevents identification of the desired treatment effect (the expert action).
Revised: In causal inference terms, this hidden information acts as an unobserved confounder which prevents identiﬁcation of the desired causal estimand (the expert action).

Original: Despite these results,impressive empirical successes have been achieved even when the learner has a more impoverishedstate representation than the expert.
Revised: Despite these results, impressive empirical successes have been achieved even when the learner has a more impoverished state representation than the expert.

Original: In this work, we reconcile theory and practice by considering a broad class of problems where thelearner’s ability to mimic expert actions increases as more observations are revealed.
Revised: In this work, we reconcile theory and practice by considering a broad class of problems where the learner’s ability to mimic expert actions increases as more observations are revealed.

Original: We study thelarge-horizon limit to tease out what is key to good performance.
Revised: We study the large-horizon limit to tease out what is key to good performance.

Original: We find that off-policy approaches(e.g. behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisionscan lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.
Revised: We ﬁnd that off-policy approaches (e.g. behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisions can lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.

Original: Weshow that for some problem families, there exists a sharp phase transition in problem parameterswhere behavior cloning shifts between being consistent to having arbitrarily poor performance.
Revised: We show that for some problem families, there exists a sharp phase transition in problem parameters where off-policy IL shifts between being consistent to having arbitrarily poor performance.

Original: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Rosset al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal
Revised: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Ross et al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal

Original: Control [Bagnell, 2015])
Revised: Control [Bagnell, 2015])

Original: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent ofparameters) asymptotically consistent on these problems.
Revised: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent of parameters) asymptotically consistent on these problems.

Original: This strong separation between on- and offpolicy approaches we believe explains empirical findings of poor performance of behavioral cloningeven in regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,
Revised: We believe this strong separation between on- and off- policy approaches helps explain both the poor performance of behavioral cloning even in regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,

Original: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kuefler et al., 2017] and also explainsthe success of on-policy methods  above.
Revised: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kueﬂer et al., 2017] and the success of on-policy methods mentioned above.

Original: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,
Revised: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,

Original: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and
Revised: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and

Original: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expertperformance [Swamy et al., 2021].
Revised: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expert performance [Swamy et al., 2021].

Original: Thus, for our experiments, we introduce partial observability toensure that our results are closer to the real world.
Revised: Thus, for our experiments, we introduce partial observability to ensure that we are focused on part of what makes imitation learning hard in practice.

Original: We study in detail Contextual Markov Decision Process (CMDP)
Revised: We study in detail Contextual Markov Decision Process (CMDPs)

Original: [Hallak et al., 2015] that satisfyan identifiability condition.
Revised: [Hallak et al., 2015] that satisfy an asymptotic realizability condition.

Original: Intuitively, this means we can expect that proper utilization of historyeventually enables accurate prediction of the context.
Revised: Intuitively, this means we can expect that proper utilization of history to eventually enable accurate prediction of the context.

Original: A key result we show is that for identifiable CMDPs where the learner can recover from mistakes early on in an episode, on-policy imitationlearning algorithms that operate in the space of histories are able to asymptotically match expertvalue, while off-policy approaches struggle to do so.
Revised: A key result we show is that for identiﬁable CMDPs where the learner can recover from mistakes early on in an episode, on-policy imitation learning algorithms that operate in the space of histories are able to asymptotically match time-averaged expert value, while off-policy approaches struggle to do so.

Original: More concretely, our work makes three contributions:
Revised: More concretely, our work makes three contributions:

Original: We show that under appropriate identifiability and recoverability conditions, the context- dependent expert policy might become asymptotically realizable , enabling on-policy imita- tion learning algorithms to match (or nearly match)  expert performance.2.
Revised: We show that under appropriate identiﬁability and recoverability conditions, the contextdependent expert policy becomes asymptotically realizable , enabling on-policy imitation learning algorithms to match (or nearly match) time-averaged expert performance.

Revised: More generally, we show that when longer history allows the learner to get closer to realizing the expert policy, on-policy methods are able to take advantage of this property while offpolicy methods are stuck with the consequences of their mistakes early on in an episode.

Original:  We conduct experiments in a simplified bandit domain which show that there exist sharpphase transitions in terms of when off-policy imitation learning algorithms equipped withhistory work in contrast to the uniform value-equivalence of the policies produced byon-policy approaches.3.
Revised: This manifests as off-policy methods producing policies that merely repeat previous actions. We conduct experiments in a simpliﬁed bandit domain which show that there exist sharp phase transitions in terms of when off-policy imitation learning algorithms match expert performance in contrast to the uniform value-equivalence of the policies produced by onpolicy approaches.

Original: We  conduct experiments on continuous control tasks that show that on-policy algorithmsare able to take advantage of history to correctly identify the context in a way that off-policymethods are not.
Revised: We also conduct experiments on continuous control tasks that show that on-policy algorithms are able to take advantage of history to correctly identify the context in a way that off-policy methods are not.

Original: We begin with a discussion of related work.
Revised: We begin with a discussion of related work.

Original:  Related Work
Revised:  Related Work

Original: One of the fundamental challenges of imitation learning (or any sequential prediction task where thelearner consumes some function of its own prior predictions) is the likelihood of significant covariateshift between training-time data and test-time observations
Revised: One of the fundamental challenges of imitation learning (or any sequential prediction task where the learner consumes some function of its own prior predictions) is the likelihood of signiﬁcant covariate shift between training-time data and test-time observations

Original: [Ross and Bagnell, 2010].
Revised: [Ross and Bagnell, 2010].

Original: In short, thishappens because the learner might end up in states not seen in the demonstrations and is thus unsurehow to act.
Revised: In short, this happens because the learner might end up in states not seen in the demonstrations and is thus unsure how to act.

Original: Early work in this area includes that of Daumé et al.
Revised: Early work in this area includes that of Daumé et al.

Original: [2009] in the natural languageprocessing and that of Ross et al.
Revised: [2009] in the natural language processing and that of Ross et al.

Original: [2010] in  robotics, both of which come to the preceding conclusion.
Revised: [2010] in imitation learning and robotics, both of which come to the preceding conclusion.

Original: Recent work by Spencer et al.
Revised: Recent work by Spencer et al.

Original: [2021] shows that there are actually more than one regime of covariateshift.
Revised: [2021] shows that there are actually more than one regime of covariate shift.

Original: In the “easy” regime where the expert is realizable, off-policy methods like behavior cloningmatch expert performance when data and model capacity are large enough.
Revised: In the “easy” regime where the expert is realizable, off-policy methods like behavior cloning match expert performance when data and model capacity are large enough.

Original: However, in harderregimes where the expert is non-realizable due to model misspecification (or effectively so due todata scarcity), off-policy methods compound in error and one must rely on on-policy methods, eitherthose that require an interactive expert [Ross et al., 2010] or an interactive simulator Ziebart et al.[2008], Swamy et al.
Revised: However, in harder regimes where the expert is non-realizable due to model misspeciﬁcation, off-policy methods compound in error and one must rely on on-policy methods, either those that require an interactive expert [Ross et al., 2010] or an interactive simulator [Ziebart et al., 2008, Swamy et al., 2021].

Original: Onepractically important instance of model misspecification is when the learner is denied stateinformation that the expert uses.
Revised: One instance of model misspeciﬁcation of practical interest is when the learner is denied state information that the expert uses.

Original: For instance, in self-driving, the human expert has richer contextabout the scene than the limited perception system of the learner.
Revised: For instance, in self-driving, the human expert might have richer context about the scene than the limited perception system of the car.

Original: While the standard solution is toadd a history of past states and actions to the model, practitioners have often noted that accelerations.
Revised: While the standard solution is to add a history of past states and actions to the model, practitioners have often noted that this leads to a “latching effect” where the learner simply repeats the past action. For example, Muller et al. [2006] note such latching with steering actions, Kueﬂer et al. [2017], Bansal et al.

Revised: [2018] note this with braking actions, and Codevilla et al.

Revised: [2019] with accelerations.

Original: Recent work by Ortega et al.
Revised: Recent work by Ortega et al.

Original: [2021] also points out this latching behavior which theyterm as “self-delusion”.
Revised: [2021] also points out this latching behavior which they term as “self-delusion.”

Original: Once one identifies the downstream effect of missing context as covariate shift, a natural questionis whether an extension of prior covariate-shift-robust imitation learning methods to the space ofhistories would be able to learn effectively in confounded settings.
Revised: Once one identiﬁes the downstream effect of missing context as covariate shift, a natural question is whether an extension of prior covariate-shift-robust imitation learning methods to the space of histories would be able to learn effectively in partial information settings.

Original: Prior work has answered parts ofthis question.
Revised: Prior work has answered parts of this question.

Original: For example, Choudhury et al.
Revised: For example, Choudhury et al.

Original: [2018] proves that interactive imitation learning overthe space of histories converges to the QMDP approximation of the expert’s policy [Littman et al.
Revised: [2018] proves that interactive imitation learning over the space of histories converges to the QMDP approximation of the expert’s policy [Littman et al., 1995].

Original: In fact, the proposed algorithm of Ortega et al.
Revised: Recent work Ortega et al.

Original: [2021]  (counterfactual teaching) is identical tothe earlier FORWARD algorithm of Ross and Bagnell [2010] under log-loss.
Revised: [2021] views these kind of partial information setting through a causal lens [Pearl et al., 2016], and provides an algorithm (counterfactual teaching) equivalent to the interactive-expert FORWARD algorithm of Ross and Bagnell [2010] under log-loss.

Original: Our main contributionprovides conditions under which such approaches will converge to a policy that is equivalent in valueto that of the expert in the presence of unobserved contexts.
Revised: We build upon these analyses by providing conditions under which such approaches will converge to a policy that is equivalent in value to that of the expert in the presence of unobserved contexts.

Original: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,
Revised: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,

Original: Kumor et al., 2021], context that is updated throughout the episode and is only reflected for a singlestep prevents the learner from refining its estimates via considering history.
Revised: Kumor et al., 2021], context that is updated throughout the episode and is only reﬂected for a single step prevents the learner from reﬁning its estimates via considering history.

Original: We would expect ourresults to generalize to settings beyond the CMDP where a similar asymptotic realizability propertyholds true.
Revised: Our results apply to settings beyond the CMDP where a similar asymptotic realizability property holds.

Original: Tennenholtz et al.
Revised: Tennenholtz et al.

Original: [2021] consider IL in contextual MDPs but give the learner access tothe confounder at test time and do not consider policies that operate over the space of histories.
Revised: [2021] consider IL in contextual MDPs but give the learner access to the confounder at test time and do not consider policies that operate over the space of histories.

Original: Recently, Swamy et al.
Revised: de Haan et al.

Original: [2021] proved that for a wide class of interactive expert imitation learningalgorithms, one should expect learner sub-optimality to grow as O ( ϵHT ) ,where T is the horizon ofthe problem and H is a recoverability constant that upper bounds the number of steps it takes for theexpert to recover from an arbitrary learner mistake Leike et al.
Revised: [2019] also consider imitation learning from a causal inference perspective but focus on issues of covariate shift [Spencer et al., 2021], rather than those of missing information.  The Latching Effect in Off-Policy Imitation Learning Consider a ﬁnite-horizon Contextual Markov Decision Process (CMDP) parameterized by h S , A , C , T , r, T i where S , A , C are the state, action, and context spaces, T : S ⇥ A ⇥ C !

Original: [2016], Ross et al.
Revised:  ( S ) is the transition operator, r :

Original: We arguethat for infinite horizon problems, one requires a finite H to argue value equivalence for the policythat an on-policy learner converges to – otherwise, a mistake early on can doom the learner for therest of the horizon.  Imitation Learning with Unobserved Contexts
Revised: S ⇥ A ⇥ C !

Original: We begin with a toy example studied by Ortega et al.
Revised: [  1 , 1] is the reward function, and T is the horizon. At the beginning of each episode, a context is sampled from p ( c ) and is held ﬁxed until the next reset. Intuitively, there exists a family of reward and transition functions that are indexed by the context for each episode. We use h t 2 H to denote a t -step history: ( s 1 , a 1 , . . .

Revised: , s t ) .

Original: 3.1 Phase Transitions for Off-Policy Imitation Learning
Revised: We see trajectories generated by an expert policy ⇡ E : S ⇥ C !  ( A ) and search over time-varying policies ⇡ : H !  ( A ) . We use ⇡ 1 ...t to refer to the sequence of policies that comprise this time-varying policy.

Revised: We assume that our policy class ⇧ is convex and compact.

Revised: We begin with a toy example of the latching effect and how it leads to poor policy performance.

Original: Problem 3.1 (Causal Bandit Problem) .
Revised: Problem 3.1 (Causal Bandit Problem, [Ortega et al., 2021]) .

Original: Consider a K -armed bandit.
Revised: Consider an episodic MDP with K actions (arms) and a single state.

Original: At the beginning of each episode,a context c ∈ [ K ] is chosen uniformly at random and represents the correct arm for that episode.
Revised: At the beginning of each episode, a context c 2 [ K ] is chosen uniformly at random and represents the correct arm for that episode.

Original: Pulling an arm leads to binary feedback: + if the arm was correct and − otherwise.
Revised: Pulling an arm leads to binary feedback: + if the arm was correct and  otherwise.

Original: This feedbackis flipped with probability ϵ obs ∈ [0 , 1] .
Revised: This feedback is ﬂipped with probability ✏ obs 2 [0 , 1] .

Original: The learner observes expert demonstrations where at eachtimestep, the expert plays the correct arm with probability 1 − ϵ exp ∈ [0 , 1] and another arm uniformlyat random otherwise.
Revised: The learner observes expert demonstrations where at each timestep, the expert plays the correct arm with probability 1  ✏ exp 2 [0 , 1] and another arm uniformly at random otherwise.

Original: The reward function is 1 for pulling the correct arm and 0 otherwise.
Revised: The reward function is 1 for pulling the correct arm and 0 otherwise.

Original: Weemphasize that the learner does not observe the rewards, just noisy binary feedback as an observationafter each pull.
Revised: We emphasize that the learner does not observe the rewards, just noisy binary feedback as an observation after each pull.

Original: [Figure]
Revised: As the learner does not observe the correct arm, we are in the partial information setting.

Original: Intuitively, by tabulating the noisy observations, the learner should be able to narrow down what thecorrect arm is.
Revised: However, as one might expect, by pulling all arms enough times and observing the noisy feedback, the learner can narrow down which arm they should pull for the rest of the episode. Note that the learner stays 0.8  exp 0.  o b s

Revised: Consistency of  off 0.

Original: We focus on    this problem as it is trivially recoverable – the learner is reset to the samestate after each action so they are free to explore without consequences.
Revised: 0.8  exp 0.  o b s Consistency of  on    2000 t 0. P r o b .

Revised: o f C o rr e c t A r m  obs = 0.3,  exp = 0.

Revised:  on  off 1   exp 1/ K in the same state after each action so they are free to perform this exploration without long-term consequences.

Revised: Observation 1: Off-Policy Methods Have Consistency Phase Transitions.

Original: In Fig.
Revised: In Fig.

Original: 1, we plot, for avariety of settings of the problem parameters ( ϵ exp , ϵ obs ) whether the learner makes mistakes with asignificantly higher probability than the expert  for behavioral cloning (an off-policy algorithm) and DAgger (an on-policy algorithm which utilizes a queryable expert).
Revised: 1, we plot, for a variety of settings of the problem parameters ( ✏ exp , ✏ obs ) whether the learner makes mistakes with a signiﬁcantly higher probability than the expert does for an off-policy IL algorithm and an on-policy IL algorithm.

Original: We give all learners access to thefull history of interactions (i.e. arms pulled and noisy feedback observed).
Revised: We give all learners access to the full history of interactions (i.e. arms pulled and noisy feedback observed).

Original: With exactly ϵ obs = 0 .
Revised: With exactly ✏ obs = 0 .

Original: 5 , the learner gets no information from observation, preventing any algorithmfrom learning properly.
Revised: 5 , the learner gets no information from the observations, preventing any algorithm from learning properly.

Original: This means that under this particular setting, the problem instance is notidentifiable , a concept we formalize below.
Revised: This means that under this particular setting, the problem instance is not identiﬁable , a concept we develop further below.

Original: In contrast to the uniform consistency of DAgger wheneverthe problem is identifiable, we see sharp phase transitions in terms of where where BC is consistent –a small change to either problem parameter can lead to a drastically different result.
Revised: In contrast to the uniform consistency of the on-policy approach whenever the problem is identiﬁable, we see sharp phase transitions in terms of where the off-policy approach is consistent – a small change to either problem parameter can lead to a drastically different result.

Revised: Observation 2: Off-Policy Methods Produce Latching Policies.

Revised: At the ﬁrst timestep, all learners pick an arm uniformly at random as they have no information about the context.

Revised: In the cases where the off-policy approach is inconsistent, it continues to pick this arm ad inﬁnitum as it treats its own past actions as though they were the expert’s.

Original: This means thatit can be rather hard to predict whether BC will work for  a given problem, rendering it dangerous toapply in practice.
Revised: This is what leads to the 1 K = 15 success rate seen in the rightmost part of Fig. 1, even after 2000 timesteps of experience telling the learner that another arm should be pulled. Put differently, the off-policy learner collapses its uncertainty over the correct arm too quickly for the negative feedback it receives to push it to a different arm. Concerningly, this effect appears to be extremely sensitive to the parameters of the problem, rendering it difﬁcult to predict and hedge against.

Revised: Observation 3: Low Density Ratios Help Off-Policy Methods.

Revised: Looking at the left side of Fig.

Revised: 1, a natural question might be why, as we increase ✏ exp , the off-policy approach is consistent for a wider spectrum of ✏ obs values.

Original: As we increase ϵ exp , the expert has a higher chance of making a mistake, bringing its trajectorydistribution closer to that of the uniform policy and lowering the worst-case density ratio betweenlearner and expert trajectory distributions.
Revised: Observe that as we increase ✏ exp , the expert has a higher chance of making a mistake, bringing its trajectory distribution closer to that of the uniform policy and lowering the maximum density ratio between learner and expert trajectory distributions.

Original: As was established by Spencer et al.
Revised: As was established by Spencer et al.

Original: [2021], a low densityratio is a sufficient condition for behavioral cloning to be consistent.
Revised: [2021], a low maximum density ratio is a sufﬁcient condition for behavioral cloning to be consistent.

Original: Our experiments echo thiscondition as a higher ϵ exp leads to greater flexibility in terms of choice of ϵ obs while maintainingconsistency.
Revised: This experiment appears to echo their theoretical results.

Original: The symmetry around the ϵ obs = 0 . 5 axis is due to the fact that the learner can stillfigure which arm is different from the rest on the basis of negated feedback.
Revised: Putting together Fig.

Original: The main take-away from this experiment is that on-policy approaches appear to be able to handlehidden context given access to history, in contrast to off-policyapproaches, which can do so but in away that depends heavily on problem parameters.  
Revised: 1, on-policy approaches appear to be able to handle hidden context given access to history while off-policy can do so only in a way that depends heavily on problem parameters.  A Bayesian Perspective on the Latching Effect

Original: To begin to explain this difference in behavior ofon-policy and off-policy algorithms, we take a brief detour into the structural causal model (SCM)each of these classes of algorithms is implicitly assuming. 
Revised: To begin to explain this difference in behavior of on-policy and off-policy algorithms, we consider the structural causal model (SCM) each of these classes of algorithms is implicitly assuming. sssaaac

Original: Consider a finite-horizon Contextual Markov Decision Process (CMDP) parameterized by⟨S , A , C , T, r, T ⟩ where S , A , C are the state, action, and context spaces, T : S × A × C → ∆( S ) 154 is the transition operator, r :
Revised: T T ⇡ E ⇡ E ⇡ E ⇡ E ⇡ E (a) ⌧ ⇠ ⇡ E hhhaaac

Original: S ×
Revised: T

Original: A × C → [ − 1 , 1] is the reward function, and T is the horizon. At 155 the beginning of each episode, a context is sampled from p ( c ) and is held fixed until the next reset. Intuitively, there exists a family of reward and transition functions that are indexed by the context foreach episode. We use h t ∈ H to denote a t -step history: ( s 1 , a 1 , . . .
Revised: T ⇡⇡⇡(b) On-Policy hhhaaac

Original: , s t ) .
Revised: T

Original: We see trajectories generated by an expert policy π E : S × C → ∆( A ) and search over time-varying policies π : H → ∆( A ) . 159 We use π 1 ...t to refer to the sequence of policies that comprise this time-varying policy. We assumethat our policy class Π is convex and compact.
Revised: T ⇡ E ⇡ E ⇡⇡ E (c) Off-Policy 4.1 SCMs for Imitation Learning

Original: Expert trajectories of length t are generated according to
Revised: We observe that expert trajectories of length t are generated according to

Original: [Equation]
Revised: [Equation]

Original: while learner trajectories are generated according to
Revised: while learner trajectories are generated according to

Original: [Equation]
Revised: [Equation]

Original: We use τ ∼ π E and τ ∼ π to denote T -step trajectories sampled according to the above distributions.
Revised: We use ⌧ ⇠ ⇡ E and ⌧ ⇠ ⇡ to denote T -step trajectories sampled according to the above distributions.

Original: The key difference between on-policy and off-policy imitation learning algorithms is the differencebetween the center and right of Fig. 2:
Revised: We deﬁne our value and Q functions as usual: V ⇡ ( s ) = E ⌧ ⇠ ⇡ | s t = s [ P [Equation]

Revised: )] , Q ⇡

Revised: [Equation]

Original: ) on policy assume the data they have seen thusfar is generated by executing learner π while policy assumes it is generated by executingthe expert policy π E  .
Revised: ) = E ⌧ ⇠ ⇡ | s t = s,a t = a [ P Tt 0 =1 r ( s t 0 , a t 0 )] .

Original: The off-policy approximation is what leads to the latching behavior observedempirically: even if the first action was chosen at random, by treating it as though it was produced bythe expert π E (who sees the context c and picks actions that are correlated across time), the learner islikely to continue to repeat the past action. This is also what we observe empirically in the causal sssaaac
Revised: Lastly, let performance be J ( ⇡ ) = E ⌧ ⇠ ⇡ [

Original: T
Revised: P

Original: T π E π E π E π E π E hhhaaac T
Revised: Tt =1 r ( s t , a t )] .

Original: T πππhhhaaac
Revised: The key difference between on-policy and off-policy imitation learning algorithms is the difference between the center and right SCMs of Fig.

Original: T
Revised: On-policy algorithms assume the data they have seen thus far is generated by executing learner ⇡ (b) while off-policy algorithms assume it is generated by executing the expert policy ⇡ E (c). The off-policy approximation is what leads to the latching behavior observed empirically: even if the ﬁrst action was chosen at random, by treating it as though it was produced by the expert ⇡ E (who sees the context c and therefore picks actions that are correlated across time), the learner is likely to continue to repeat the past action .

Original: T π E π E ππ E bandit problem in the cases where behavioral cloning does not work: the learner continues to play thefirst arm it chose, ignoring the feedback it gets that it is repeatedly making a mistake.
Revised: This is also what we observe empirically in the causal bandit problem in the cases where off-policy IL does not work: the learner continues to play the ﬁrst arm it chose, ignoring the feedback it gets that it is repeatedly making a mistake, and only matching expert performance on a 1 K of episodes.

Revised: We note this gives the off-policy learner a choice between ﬁre and pyre [Eliot, 1942]: they need to use history to narrow down the context but if they do, they can learn a latching policy that performs poorly at test-time.

Revised: 4.2 Off Policy Methods Have an Incorrect Context Posterior

Original: Another way of seeing this point is by considering what the posterior over the confounder would beunder samples from each of these SCMs.
Revised: Another way of seeing this point is by considering what the posterior over the confounder would be under samples from each of these SCMs.

Revised: If the learner was able to accurately pin down the correct arm, they would be able to easily reproduce the expert policy.

Original: Via Bayes Rule and assuming a uniform prior over contexts,we know that
Revised: Thus, we can focus on correct-arm identiﬁcation via Bayes Rule. Assuming a uniform prior over contexts,

Original: [Equation]
Revised: [Equation]

Original: Under the off-policy graphical model, p ( c, h t ) ∝ p ( τ ; π E ) – the probability of a history h t under theexpert’s distribution (the left SCM in Fig. 2).
Revised: Under the off-policy graphical model, p ( c, h t ) / p ( ⌧ ; ⇡ E ) : the probability of a history h t under the expert’s distribution (Fig. 2, (a)).

Original: Expanding terms, we arrive at 
Revised: Expanding terms, we arrive at the following expression

Revised: Proposition 4.1.

Revised: The off-policy posterior over contexts is

Original: [Equation]
Revised: [Equation]

Original: Notice how under the expert’s generative model, the context c directly influences the actions so it isuseful to condition on them when attempting to predict c .
Revised: Notice that for the expert policy, the context c directly inﬂuences actions so conditioning on actions when attempting to predict c is correct.

Original: We highlight this term that encodes thisdependence in green.
Revised: We highlight in green the term that encodes this dependence.

Original: In contrast, in the  on-policy graphical model (Fig. 2, center), we could treatthe actions as interventions rather than as variables with a path to c and still end up with the sameposterior over contexts.
Revised: In contrast, if we assume the learner has no access to the context except via its inﬂuence on the history of states, we should instead treat actions as interventions [Pearl et al., 2016] that provide no further information about the context. The on-policy graphical model (Fig. 2, (b)) does just this, allowing us to compute the correct posterior over contexts.

Original: Mathematically,p
Revised: Formally, Proposition 4.2. The on-policy posterior over contexts is

Original: [Equation]
Revised: [Equation]

Original: where the equality follows from the fact that
Revised: where the equality follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: (6) and the standard do () -calculus rules [Pearl et al., 2016].
Revised: and the standard do () -calculus rules [Pearl et al., 2016].

Original: Intuitively, we are leveraging the fact thatthe learner’s actions have all their dependence on the context mediated through the history of states toignore them in our posterior calculation.
Revised: Intuitively, we are leveraging the fact that the learner’s actions have all their dependence on the context mediated through the history of states to ignore them in our posterior calculation.

Original: Notice that this expression matches the off-policy posteriorexcept for the term in green – because the on-policy learner knows that the actions in the history werenot produced by the expert, it does not weight them by the expert’s probability of playing them in it’sposterior update.
Revised: Notice that this expression matches the off-policy posterior except for the term in green: because the on-policy learner knows that the actions in the history were not produced by the expert, it does not weight them by the expert’s probability of playing them in its posterior update .

Original: Graphically, as far as the posterior over the context is concerned, we could be in thefollowing SCM:
Revised: Graphically, as far as the posterior over the context is concerned, on-policy approaches correspond to the SCM in Fig. 3, left.

Original: [Figure]
Revised: [Figure]

Original: By ignoring the incorrect actions early on, the on-policy learner does not learn to merely repeat thefirst thing it tried.
Revised: By ignoring its own incorrect actions early on, the on-policy learner doesn’t over-index on them and come to a false conclusion as to what the value of the context is, as illustrated in Fig.

Original: While this single-term might seem rather unimportant, it can lead to markedlydifferent results. In fact, the learners we used to generate Fig. 1 had policies given by
Revised: 3, right. We’ve already seen evidence of how the addition or removal of the green term can lead to markedly different results: the learners we used to generate Fig. 1 had policies given by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Note that these policies are exactly equivalent except for the green term in the posterior of the offpolicy π bc , yet have markedly different consistency properties as the parameters of the problem arechanged.
Revised: Note that these policies are exactly equivalent except for the green term in the posterior of the off-policy ⇡ o ↵ , yet respond quite differently as the parameters of the problem are changed.

Original: We now turn our attention to generalizing this argument beyond likelihood-based learners(i.e. those that match sufficient statistics of expert behavior – moments – rather than maintainingexplicit posteriors) and providing value equivalence guarantees for on-policy learners in the limit ofinfinite timesteps. 3.3 Asymptotic Realizability in Imitation Learning One might hope that  with the use of history and the on-policy graphical model, one might be able toeventually match expert performance in contextual problems.
Revised: Also note that p on ( c | h t ) requires interaction with the environment to evaluate, while p o ↵ ( c | h t ) does not.

Original:  This is an example of a more generalphenomenon we term asymptotic realizability (AR), in which the learner is able toperform as well asthe expert does with high probability after observing an arbitrary history of some length. We beginby defining our goal in AR problems: Definition 3.2 (Asymptotic Value Equivalence (AVE)) . We say that policy π is asymptoticallyvalue-equivalent (AVE) when the following condition holds true as T → ∞ :
Revised:  A Moment-Matching Solution to the Latching Effect

Original: [Equation]

Original: Intuitively, this condition means that the learner performs as well as the expert does on average, givenenough time. In the finite horizon, we use average imitation gap (AIG) to refer to the followingquantity: [Equation] A sufficient condition for AVE is that the AIG goes to 0 as T → ∞ .
Revised: We now turn our attention to generalizing this argument beyond Bayesian learners, including to those that match sufﬁcient statistics of expert behavior – moments – rather than maintaining explicit posteriors, and providing value equivalence guarantees for history-equipped on-policy learners.

Original: We will now study the AIGproperties of on-policy and off-policy imitation learning algorithms. While up to this point wehave focused on likelihood-based methods (e.g. behavioral cloning or history-based methods thatmaintain an explicit posterior and perform Bayesian filtering), our theory generalizes to the broaderclass of moment-matching algorithms, which includes, beyond the previously discussed approaches, AdVIL/AdRIL/DAeQuIL [Swamy et al., 2021], GAIL [Ho and Ermon, 2016], and MaxEnt IRL[Ziebart et al., 2008].
Revised: 5.1 A Quick Review of Moment-Matching in Imitation Learning

Original: First, as in Swamy et al.
Revised: First, as in Swamy et al.

Original: [2021], we define F Q E as the set of on- Q moments , with f ∈ F Q E satisfying221 type signature
Revised: [2021], we deﬁne F Q E as the set of on- Q moments , with f 2 F Q E satisfying type signature

Original: S ×
Revised: S ⇥

Original: A × C → [ − T, T ] .
Revised: A ⇥ C ! [  T, T ] .

Original: Intuitively, F Q E spans the set of possible expert Q -functions.
Revised: Intuitively, F Q E spans the set of possible expert Q -functions.

Original: We require the actual expert Q -function to be contained: Q π E ∈ F Q E .
Revised: We require the actual expert Q -function to be contained: Q ⇡ E 2 F Q E .

Original: We assume that F Q E is convex,223 compact, closed under negation, and finite dimensional.
Revised: We assume that F Q E is convex, compact, closed under negation, and ﬁnite dimensional.

Original: We use
Revised: Second, deﬁne

Original: F on [Equation] t ) =
Revised: F Q be the class of off- Q moments (i.e. 8 ⇡ 2 ⇧ , Q ⇡ 2 F Q ) and satisfy the same function-class assumptions as

Original: { f/
Revised: F Q E .

Original: [Equation] T − t ) : f ∈ F Q E } 224 to refer to a scaled version of  the on- Q moments. We define two key quantities:
Revised: Third, let F r denote the class of reward moments and also satisfy the same function-class assumptions. All f 2 F r satisfy type signature S ⇥

Original: [Equation]
Revised: A ⇥ C ! [  1 , 1] and we assume that r 2 F r . Swamy et al.

Original: [Equation]
Revised: [2021] prove that if one is able to approximately solve a two-player zero-sum momentmatching game between a policy player (that picks from ⇧ ) and a discriminator (that picks from F Q E , F Q , or F r ), one has a bound on the performance difference between the learner and the expert. Depending on which class the discriminator selects from, one ends up with a different bound. For example, if one solves the game with the following payoff and f 2 F r to an ✏ -approximate Nash equilibrium,

Original: [Equation]
Revised: [Equation]

Original: 11) where ϵ on ( t ) is the moment matching error at time t between actions selected by the expert and thelearner on states visited by the learner, H is a moment recoverability constant which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake Leike et al. [2016], Rosset al. [2010], Swamy et al. These can be seen as the natural history-space generalizations ofthe analogous concepts in Swamy et al.
Revised: one has a guarantee that J ( ⇡ E )  J ( ⇡ )  ✏ T .

Original: [2021] and govern the performance of interactive imitationlearning algorithms.
Revised: Unfortunately, directly solving such a game is not possible in the hidden context setting as we do not see the contexts and therefore cannot evaluate the moment functions. 1 We turn our attention to adapting moment-matching to the contextual setting. 5.2 Moment-Matching with Unobserved Contexts

Original:  A similar moment-matching error can be defined for off-policy algorithms, this time computed onexpert rollouts and as a function of learner Q moments.
Revised: For each moment class, we assume the existence of an observable moment class that operates over the space of histories (i.e. H ⇥ A ! R ) and has members that eventually produce outputs close to that of their context-dependent counterparts. In math, Assumption 5.1 (Asymptotic On- Q Moment Identiﬁability) . [Equation]

Original: Let F Q be the class of learner Q momentsand satisfy the same assumptions as F QE .
Revised: Analogous conditions can be deﬁned for F r and F Q , giving us ˜ F r and ˜ F Q .

Original: Analogously to above, we define F off ( t ) 
Revised: We note that these condi- tions are asymptotic in nature – we use  ( t ) to refer to the ﬁnite-horizon expected difference between the observable and context-dependent moments. We use H to denote the moment recoverability constant , which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake [Swamy et al., 2021]: [Equation] [Equation]

Revised: For problems where the expert is able to effectively correct learner mistakes, this quantity can be signiﬁcantly smaller than the horizon.

Original:  = {  f/ 2( T− t ) : 236 f ∈ F Q } to refer to a scaled version of the off- Q moments.
Revised: Deﬁne ˜ F on = { ˜ f/ 2 H : ˜ f 2 ˜ F Q E } and ˜ F o ↵ = { ˜ f/ 2 T : ˜ f 2 ˜ F Q } to be scaled-down versions of the observable moments such that their range is [  1 , 1] .

Original: We can now define our off-policy 237 moment matching error:
Revised: We can now deﬁne our moment-matching errors:

Original: [Equation]
Revised: [Equation]

Original: We can also define a moment-matching error for online imitation learning algorithms (i.e. thosethat have access to the environment but not a queryable expert like GAIL [Ho and Ermon, 2016]or MaxEnt IRL
Revised: As argued by Swamy et al. [2021], ✏ on ( t ) governs the performance of on- Q algorithms like DAgger [Ross et al., 2010], ✏ o ↵ ( t ) the performance of off- Q algorithms like behavioral cloning [Pomerleau, 1989], and ✏ rew ( t ) the performance of reward-matching algorithms like MaxEnt IRL

Original: [Ziebart et al., 2008]) in terms of reward moments that span the class of rewardfunctions. Let f ∈ F r satisfy type signature S
Revised: [Ziebart et al., 2008] and GAIL [Ho and Ermon, 2016]. With these deﬁnitions laid out, we can now prove how well each class of algorithms handles unobserved contexts.

Original: × A × C
Revised: For some partial information problems, the use of history coupled with on-policy feedback might allow the learner to eventually match expert performance. This is an example of a more general phenomenon we term asymptotic realizability (AR), in which the learner is able to perform as well as the expert does with high probability after observing an arbitrary history of some length. We begin by deﬁning the average imitation gap or AIG for short.

Original: → [ − 1 , 1] and assume that r ∈ F r . 
Revised: Deﬁnition 5.2 (AIG( ⇡ , T)) . [Equation]

Original: We 242 define the reward moment-matching error as
Revised: We now deﬁne our performance target in AR problems. Deﬁnition 5.3 (Asymptotic Value Equivalence (AVE)) . We say that policy ⇡ is asymptotically value-equivalent (AVE) when the following condition holds true:

Original: [Equation]
Revised: [Equation]

Revised: Intuitively, this condition means that the learner performs as well as the expert does on average, given enough time.

Revised: Put differently, we do not penalize the learner for initial mistakes as long as they are able to learn enough from them to match expert performance.

Revised: We will proceed by studying the AIG properties of on-policy and off-policy imitation learning algorithms.

Original: We are now ready to state our main results:
Revised: We are now ready to state our main result:

Original: Theorem 3.3 (Interactive AVE) .
Revised: Theorem 5.4 (AVE of IL Algorithms) .

Original: For all MDPs and π s.t.
Revised: Deﬁne lim sup t !

Original: lim sup t →∞ ϵ on ( t ) = 
Revised: 1 ✏ on ( t ) = ✏ on [Equation]

Original: ϵ on ( ∞ ) , 
Revised: 1 ) , lim T ! [Equation] ↵

Original: [Equation]
Revised: [Equation]

Original: Thus, if ϵ on ( ∞ ) = 0 , the on-policy learner will achieve AVE.
Revised:  o ↵ ( t ) = ⌃ o ↵ ( 1 ) , and lim sup t ! 1 ✏ rew ( t ) = ✏ rew ( 1 ) .

Revised: For all (C)MDPs and ⇡ ,

Revised: [Equation]

Revised: In words, if either an on- Q or reward-matching learner is able to achieve 0 moment matching error asymptotically, they will achieve the same average value as the expert.

Original: Even if it does not, the AIG will bebounded by a constant.
Revised: Even if they are unable to do so, the AIG will be bounded by a constant.

Original: In contrast,we have thefollowing result for off-policy algorithms:
Revised: In contrast, the result for off-policy algorithms is much weaker.

Revised: Notice that instead of taking a lim sup which, roughly speaking, captures the asymptotic error, the off-policy bound is in terms of the sum of errors, which factors in errors made early on.

Revised: This result indicates that an off-policy learner that makes mistakes early on in the episode (as is likely for contextual problems) might be unable to eventually match expert performance, even when the problem is recoverable.

Revised: We prove that there exist certain problems for which this bound is tight.

Original: Theorem 3.4 (Off-Policy AVE) .
Revised: Theorem 5.5 (Off-Policy AVE Lower Bound) .

Original: For all MDPs and π s.t.
Revised: There exist CMDPs and ⇡ s.t.

Original: lim sup t →∞ ϵ off ( [Equation] off ( 
Revised: lim sup t ! 1 ✏ o ↵ ( t ) =for which

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This result indicates that unless an off-policy learner is able to reduce its moment-matching error at arate ∝ 1 T , the learner will have a growing AIG and therefore not achieve AVE. We see a similar resultfor online IL algorithms to that for interactive IL algorithms: Theorem 3.5 (Online AVE) . For all MDPs and π
Revised: In words, this theorem says that for certain problems, even if an off-policy learner can drive down error asymptotically, they might be doomed as far as AVE because of mistakes made early on.

Original: [Equation]

Original: [Equation] [Equation] We prove these results in the appendix. To come full circle, we now argue that a subset of CMDPs(including our causal bandit problem)satisfies a condition that makes it possible to drive ϵ off ( ∞ ) ,ϵ on ( ∞ ) , and ϵ rew ( ∞ ) to 0.
Revised: We prove these results in Appendix A. To come full circle, we now argue that CMDPs (including our causal bandit problem) that satisfy the asymptotic moment identiﬁability conditions along with an asymptotic realizability condition on the policy class enable the learner to match expert performance:

Original: We now make two assumptions that characterize this subset:
Revised: [Figure]

Original: Assumption 3.6 (Asymptotic Identifiability) .
Revised: Assumption 5.6 (Asymptotic Realizability) .

Original: There exists a function g such that for any context c ,and for all t ∈ [ T ] , and any history of length t generated by any π ∈ Π , g ( h t ) = c with probability1 − δ ( t ) , where lim t →∞ δ ( t ) = 0 . This assumption means that regardless of the sequence of observations the learner receives, theywill eventually be able to pin down the confounder.
Revised: Asymptotic Moment Identiﬁability holds and 9 ⇡ 2 ⇧ s.t.

Original: One way to ensure this condition holds is toadd exploration noise to the learner’s actions such that they try all actions with sufficiently highprobability and assuming an underlying ergodicity of the MDP. Assumption 3.7 (Realizability) .
Revised: lim t !

Original: Π contains π E (s t  , g (h t )) , where g is defined as above.
Revised: 1 ✏ on ( t ) = 0 , lim t ! 1 ✏ o ↵ ( t ) = 0 , and lim t ! 1 ✏ rew ( t ) = 0 .

Original: This is  a relatively weak assumption which simply means that the learner can act as the expert wouldonce they have nailed down the confounder. Under these two assumptions, a CMDP is really just anexample of  the more general phenomenon of AR.
Revised: Note this assumption is weaker than a standard realizability assumption – it is saying that along certain moments of interest, we can asymptotically match the expert. Plugging in this assumption into Theorem 5.4 tells us that on such problems, an on-policy learner will be able to achieve AVE.

Original: We  prove  the following result in the appendix:
Revised: We now turn our attention to efﬁciently computing such a policy. We prove that by solving an approximate equilibrium computation game over the space of history-based policies and the space of observable moments (which can be done efﬁciently with no-regret algorithms, [Freund and Schapire, 1997]), one can ﬁnd a policy that achieves a low AIG.

Original: Theorem 3.8.
Revised: Theorem 5.7.

Original: For any contextual MDP and policy class that satisfy the preceding two assumptions,any no-regret learner over Π will be able to drive ϵ on ( ∞ ) and ϵ rew ( ∞ ) to 0.
Revised: For any contextual MDP and policy class that satisﬁes Asymptotic Realizability, let ⇡ be an ✏ -approximate Nash equilibrium strategy for the inﬁnite horizon reward-matching or on- Q -matching game. Then, we know that lim T ! 1 AIG( ⇡ , T )  ✏ or lim T ! 1 AIG( ⇡ , T )  H ✏ .

Revised: In short, by solving an on-policy moment-matching problem over policies that have access to history, we have strong guarantees of matching expert performance on asymptotically realizable problems.

Revised: We re-iterate that we have no such guarantees for off-policy algorithms.

Original: We also note that our causal bandit problem satisfies these assumptions:
Revised: We also note that our causal bandit problem satisﬁes these assumptions, providing theoretical justiﬁcation for our results.

Original: Corollary 3.9.
Revised: Corollary 5.8.

Original: The Causal Bandit Problem satisfies Asymptotic Identifiability and Realizability andhence achieve AVE for any on-policy moment matching algorithm.
Revised: There exists a singleton observable moment class such that the Causal Bandit Problem satisﬁes Asymptotic Realizability, implying that the iterates produced by an on-policy moment matching algorithm will achieve AVE.

Original: In short, for asymptotically realizable problems, on-policy imitation learning algorithms have strongerguarantees than their off-policy counterparts with respect to asymptotic value equivalence.
Revised: Putting it all together, for asymptotically realizable problems, on-policy imitation learning algorithms have stronger guarantees than their off-policy counterparts with respect to asymptotic value equivalence.

Original: Contextual MDPs are a specific example of AR problems if they satisfy an identifiability condition. One specificexample of such a CMDP is our Causal Bandit Problem.
Revised: If a contextual MDP satisﬁes these conditions (like with our Causal Bandit Problem), the preceding results apply. We see that our theory matches our experiments: on-policy algorithms appear to work on all identiﬁable instances of the causal bandit problems, while off-policy algorithms have much weaker performance.

Original: We now turn our attention to another CMDPthat satisfies our assumptions to further validate our theory empirically.
Revised: We now turn our attention to more complex CMDPs that satisfy our assumptions to further validate our theory empirically.

Original:  Experiments
Revised:  Experiments

Original: [Figure] We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai[2016] that draws heavily on the multi-task reinforcement learning setups of Finn et al.[2017] and Kumar et al.
Revised: We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai [2016] that is inspired by the multi-task reinforcement learning setups of Finn et al.

Original: In these tasks, the agent is rewarded for running at a particular velocity thatis randomly sampled at the beginning of each episode.
Revised: In these tasks, the agent is rewarded for running at a particular velocity that is randomly sampled at the beginning of each episode.

Original: We train expert policies that have access tothis privileged information.
Revised: We train expert policies that have access to this privileged information.

Original: We append an indicator to the learner’s state of whether their currentvelocity is below or above the target velocity for the episode – by observing a few such indicators,one might hope that the learner is able to hone in on what their velocity should be via a sort of binarysearch.

Original: We compare two algorithms: the off-policy behavioral cloning ( BC )
Revised: We compare two algorithms: the off-policy behavioral cloning ( BC )

Original: [Pomerleau, 1989] andthe on-policy DAgger ( DAgger )
Revised: [Pomerleau, 1989] and the on-policy DAgger ( DAgger )

Original: [Ross et al., 2010]. We use two model classes. The base class is atwo-layer MLP with 256 hidden units and ReLu activations. The history-dependent class (for whichwe use the suffix -SEQ ) is an LSTM with a hidden unit size of 256 that is preceded by a 1-layer MLPwith 256 units and a ReLu activation.
Revised: [Ross et al., 2010] that either have access to the immediate state or the last ﬁve timesteps of history (for which we use the sufﬁx -SEQ ).

Original: We give the learner access to the last 5 timesteps of history.

Original: Weemphasize that the expert operates only using the current state.

Original: In  Fig. 4 Left, we see that without access to history, BC performs poorly.
Revised: In the bar plots of Fig. 4, we see that without access to history, BC performs poorly.

Original: Interestingly, when weequip the  learner with history, we actually see it perform worse on average, reflecting the shortof latching-style behavior that has been observed repeatedly in practice.
Revised: As our theory predicts, when we equip the BC learner with history, we actually see it perform worse on average, exhibiting the latching behavior that has been observed repeatedly in practice.

Original: In contrast, we see thatequipping our on-policy learner with access to the last few observations and actions does not lead toa sharp decline in terms of performance.
Revised: In contrast, we see that equipping our on-policy learner with access to the last few observations and actions does not lead to a sharp decline in terms of performance.

Original: For the center and right plots, we use MSE as a surrogatefor moment-matching error.
Revised: We use MSE as a surrogate for moment-matching error.

Original: In the center plot of Fig.

Original: 4, we see that, as expected, on-policy methodsare far better at matching expert actions on their own rollout distribution.
Revised: We see that on both environments, on-policy methods are far better at matching expert actions on their own rollout distribution.

Original: We  see that with enough time, DAgger-SEQ is able to predict actionsmarginally better than DAgger .
Revised: We also see that with enough time, DAgger-SEQ is able to predict actions better than DAgger (lower ✏ on (1000) ).

Original: In Fig. 4 right, we plotthe product of ϵ off ( t ) and t . If this curve was roughly constant or decaying (reflecting an off-policymoment matching error that decays at or faster than 1 /t ), our theory would indicate that the learnerwould perform as well as  the expert under an arbitrary reward function. However, we actually see alinear growth , indicating that our off-policy learners should not achieve value equivalence, whichechoes what we see in the left plot.
Revised: Putting together these observations, it appears as though on-policy methods with access to history are the most robust against partial information, agreeing with our theory.

Original: Thus, we find that our theory accurately predicts these results.
Revised: We release our code at https://github.com/gkswamy98/sequence_model_il .

Original:  Conclusion
Revised:  Conclusion

Original: We study  imitation learning problems for which the expert has access to more information than thelearner.
Revised: We study asymptotically realizable imitation learning problems where the learner’s ability to mimic the expert increases over the timesteps of the problem.

Original: We find that as long as the learner can identify this missing information and recover from themistakes it made during identification, on-policy algorithms are able to match time-averaged expertvalue, in contrast to off-policy algorithms.
Revised: We ﬁnd that under identiﬁability and recoverability assumptions, on-policy algorithms are able to match time-averaged expert value, in contrast to off-policy algorithms which might pay for their early mistakes for the rest of the horizon. We demonstrate that empirically, off-policy algorithms are not able to take advantage of history to uncover missing information.

Original: We believe our results point towards a unified explanationof both empirical successes and failures.
Revised: We believe our results point towards a uniﬁed explanation of both empirical successes and failures.

Original:  Limitations.
Revised:  Acknowledgments

Original: Our theoretical analyses primarily focus on the so-called asymptotic regime where T goes to infinity. For many applications, finite time bounds are equally important.
Revised: ZSW is supported in part by the NSF FAI Award #1939606, a Google Faculty Research Award, a J.P.

Original: There will alsobe value in deriving more refined finite-sample bounds. In particular, it will be interesting to see ifthere are concrete structure in the problem instances that can replace our asymptotic identifiabilityassumption. Moreover, our framework currently relies on the inherent randomness in the environmentto establish identifiability of the context. While this is a natural assumption for many environments,we believe there is also a problem space of designing more sample-efficient exploration algorithmsthat leverage the structure of the underlying MDPs.
Revised: Morgan Faculty Award, a Facebook Research Award, an Okawa Foundation Research Grant, and a Mozilla Research Grant.

Original: Societal impacts. Our paper aims to address an important issue in data-driven decision-makingprocesses in the real world—that is, the data collection in many environments may miss contextualinformation that is critical to the problem. Our work focuses on this issue in the domain of imitationlearning, and establishes the theoretical guarantees for each type of algorithms. We believe that ourstudies will promote appropriate reliance on different types of algorithms when we deploy imitationlearning technologies in the real world.
Revised: GS is supported computationally by a GPU award from NVIDIA and emotionally by his family and friends.