Original: Sequence Model Imitation Learning with Unobserved Contexts
Revised: Sequence Model Imitation Learning with Unobserved Contexts

Original: Abstract
Revised: Abstract

Original: We consider imitation learning problems where the learner’s ability to mimic the expert increases throughout the course of an episode as more information is revealed.
Revised: We consider imitation learning problems where the learner’s ability to mimic the expert increases throughout the course of an episode as more information is revealed.

Original: One example of this is when the expert has access to privileged information: while the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions, they might be able to eventually identify the hidden context and act as the expert would.
Revised: One example of this is when the expert has access to privileged information: while the learner might not be able to accurately reproduce expert behavior early on in an episode, by considering the entire history of states and actions, they might be able to eventually identify the hidden context and act as the expert would.

Original: We prove that on-policy imitation learning algorithms (with or without access to a queryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods.
Revised: We prove that on-policy imitation learning algorithms (with or without access to a queryable expert) are better equipped to handle these sorts of asymptotically realizable problems than off-policy methods.

Original: This is because onpolicy algorithms provably learn to recover from their initially suboptimal actions, while off-policy methods treat their suboptimal past actions as though they came from the expert.
Revised: This is because onpolicy algorithms provably learn to recover from their initially suboptimal actions, while off-policy methods treat their suboptimal past actions as though they came from the expert.

Original: This often manifests as a latching behavior: a naive repetition of past actions.
Revised: This often manifests as a latching behavior: a naive repetition of past actions.

Original: We conduct experiments in a toy bandit domain that show that there exist sharp phase transitions of whether off-policy approaches are able to match expert performance asymptotically, in contrast to the uniformly good performance of on-policy approaches.
Revised: We conduct experiments in a toy bandit domain that show that there exist sharp phase transitions of whether off-policy approaches are able to match expert performance asymptotically, in contrast to the uniformly good performance of on-policy approaches.

Original: We demonstrate that on several continuous control tasks, on-policy approaches are able to use history to identify the context while off-policy approaches actually perform worse when given access to history.
Revised: We demonstrate that on several continuous control tasks, on-policy approaches are able to use history to identify the context while off-policy approaches actually perform worse when given access to history.

Original:  Introduction
Revised:  Introduction

Original: An unstated assumption in much of the work in imitation learning (IL) is that the learner and the expert have access to the same state information.
Revised: An unstated assumption in much of the work in imitation learning (IL) is that the learner and the expert have access to the same state information.

Original: With powerful enough statistical models, this assumption places us in the realizable setting – i.e. the imitator can actually behave like the expert.
Revised: With powerful enough statistical models, this assumption places us in the realizable setting – i.e. the imitator can actually behave like the expert.

Original: In practice however, an expert might have more information than the learner does.
Revised: In practice however, an expert might have more information than the learner does.

Original: For example, an expert policy might be trained in simulation with privileged access to state before being used to supervise a learner policy that only has access to a subset of features.
Revised: For example, an expert policy might be trained in simulation with privileged access to state before being used to supervise a learner policy that only has access to a subset of features.

Original: This recipe has enjoyed success in domains from motion planning with obstacles [Choudhury et al., 2018], to autonomous driving [Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].
Revised: This recipe has enjoyed success in domains from motion planning with obstacles [Choudhury et al., 2018], to autonomous driving [Chen et al., 2019], to legged locomotion [Lee et al., 2020, Kumar et al., 2021].

Original: Recent theoretical work has established “no-go” results for successfully imitating an expert that has access to more information [Zhang et al., 2020, Kumor et al., 2021].
Revised: Recent theoretical work has established “no-go” results for successfully imitating an expert that has access to more information [Zhang et al., 2020, Kumor et al., 2021].

Original: The core of their arguments is that without seeing some feature that inﬂuences expert behavior but is not echoed elsewhere in the state, the learner might not be able to properly ground the expert actions in the observed state.
Revised: The core of their arguments is that without seeing some feature that influences expert behavior but is not echoed elsewhere in the state, the learner might not be able to properly ground the expert actions in the observed state.

Original: In causal inference terms, this hidden information acts as an unobserved confounder which prevents identiﬁcation of the desired causal estimand (the expert action).
Revised: In causal inference terms, this hidden information acts as an unobserved confounder which prevents identification of the desired causal estimand (the expert action).

Original: Despite these results, impressive empirical successes have been achieved even when the learner has a more impoverished state representation than the expert.
Revised: Despite these results, impressive empirical successes have been achieved even when the learner has a more impoverished state representation than the expert.

Original: In this work, we reconcile theory and practice by considering a broad class of problems where the learner’s ability to mimic expert actions increases as more observations are revealed.
Revised: In this work, we reconcile theory and practice by considering a broad class of problems where the learner’s ability to mimic expert actions increases as more observations are revealed.

Original: We study the large-horizon limit to tease out what is key to good performance.
Revised: We study the large-horizon limit to tease out what is key to good performance.

Original: We ﬁnd that off-policy approaches (e.g. behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisions can lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.
Revised: We find that off-policy approaches (e.g. behavioral cloning) that ignore the resulting covariate shift from initially sub-optimal decisions can lead to poor results, even when there exists a policy that in the large-horizon limit is optimal.

Original: We show that for some problem families, there exists a sharp phase transition in problem parameters where off-policy IL shifts between being consistent to having arbitrarily poor performance.
Revised: We show that for some problem families, there exists a sharp phase transition in problem parameters where off-policy IL shifts between being consistent to having arbitrarily poor performance.

Original: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Ross et al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal
Revised: In contrast, we show that on-policy approaches that leverage interaction with the demonstrator [Ross et al., 2010] or take advantage of interaction with the environment (in the style of Inverse Optimal

Original: Control [Bagnell, 2015])
Revised: Control [Bagnell, 2015])

Original: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent of parameters) asymptotically consistent on these problems.
Revised: [Ziebart et al., 2008, Ho and Ermon, 2016] are always (i.e. independent of parameters) asymptotically consistent on these problems.

Original: We believe this strong separation between on- and off- policy approaches helps explain both the poor performance of behavioral cloning even in regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,
Revised: We believe this strong separation between on- and off- policy approaches helps explain both the poor performance of behavioral cloning even in regimes with large data and powerful model classes [Spencer et al., 2021, Muller et al., 2006,

Original: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kueﬂer et al., 2017] and the success of on-policy methods mentioned above.
Revised: Codevilla et al., 2019, de Haan et al., 2019, Bansal et al., 2018, Kuefler et al., 2017] and the success of on-policy methods mentioned above.

Original: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,
Revised: We note that, in contrast to the hidden state that is common in real-world problems [Boots et al., 2011,

Original: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and
Revised: Kumar et al., 2021, Lee et al., 2020], standard benchmarks like the PyBullet suite [Coumans and

Original: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expert performance [Swamy et al., 2021].
Revised: Bai, 2016] are fully observed, enabling off-policy algorithms like behavioral cloning to match expert performance [Swamy et al., 2021].

Original: Thus, for our experiments, we introduce partial observability to ensure that we are focused on part of what makes imitation learning hard in practice.
Revised: Thus, for our experiments, we introduce partial observability to ensure that we are focused on part of what makes imitation learning hard in practice.

Original: We study in detail Contextual Markov Decision Process (CMDPs)
Revised: We study in detail Contextual Markov Decision Process (CMDPs)

Original: [Hallak et al., 2015] that satisfy an asymptotic realizability condition.
Revised: [Hallak et al., 2015] that satisfy an asymptotic realizability condition.

Original: Intuitively, this means we can expect that proper utilization of history to eventually enable accurate prediction of the context.
Revised: Intuitively, this means we can expect that proper utilization of history to eventually enable accurate prediction of the context.

Original: A key result we show is that for identiﬁable CMDPs where the learner can recover from mistakes early on in an episode, on-policy imitation learning algorithms that operate in the space of histories are able to asymptotically match time-averaged expert value, while off-policy approaches struggle to do so.
Revised: A key result we show is that for identifiable CMDPs where the learner can recover from mistakes early on in an episode, on-policy imitation learning algorithms that operate in the space of histories are able to asymptotically match time-averaged expert value, while off-policy approaches struggle to do so.

Original: More concretely, our work makes three contributions:
Revised: More concretely, our work makes three contributions:

Original: We show that under appropriate identiﬁability and recoverability conditions, the contextdependent expert policy becomes asymptotically realizable , enabling on-policy imitation learning algorithms to match (or nearly match) time-averaged expert performance.
Revised: We show that under appropriate identifiability and recoverability conditions, the contextdependent expert policy becomes asymptotically realizable , enabling on-policy imitation learning algorithms to match (or nearly match) time-averaged expert performance.

Original: More generally, we show that when longer history allows the learner to get closer to realizing the expert policy, on-policy methods are able to take advantage of this property while offpolicy methods are stuck with the consequences of their mistakes early on in an episode.
Revised: More generally, we show that when longer history allows the learner to get closer to realizing the expert policy, on-policy methods are able to take advantage of this property while offpolicy methods are stuck with the consequences of their mistakes early on in an episode.

Original: This manifests as off-policy methods producing policies that merely repeat previous actions.
Revised: This manifests as off-policy methods producing policies that merely repeat previous actions.

Original: We conduct experiments in a simpliﬁed bandit domain which show that there exist sharp phase transitions in terms of when off-policy imitation learning algorithms match expert performance in contrast to the uniform value-equivalence of the policies produced by onpolicy approaches.
Revised: We conduct experiments in a simplified bandit domain which show that there exist sharp phase transitions in terms of when off-policy imitation learning algorithms match expert performance in contrast to the uniform value-equivalence of the policies produced by onpolicy approaches.

Original: We also conduct experiments on continuous control tasks that show that on-policy algorithms are able to take advantage of history to correctly identify the context in a way that off-policy methods are not.
Revised: We also conduct experiments on continuous control tasks that show that on-policy algorithms are able to take advantage of history to correctly identify the context in a way that off-policy methods are not.

Original: We begin with a discussion of related work.
Revised: We begin with a discussion of related work.

Original:  Related Work
Revised:  Related Work

Original: One of the fundamental challenges of imitation learning (or any sequential prediction task where the learner consumes some function of its own prior predictions) is the likelihood of signiﬁcant covariate shift between training-time data and test-time observations
Revised: One of the fundamental challenges of imitation learning (or any sequential prediction task where the learner consumes some function of its own prior predictions) is the likelihood of significant covariate shift between training-time data and test-time observations

Original: [Ross and Bagnell, 2010].
Revised: [Ross and Bagnell, 2010].

Original: In short, this happens because the learner might end up in states not seen in the demonstrations and is thus unsure how to act.
Revised: In short, this happens because the learner might end up in states not seen in the demonstrations and is thus unsure how to act.

Original: Early work in this area includes that of Daumé et al.
Revised: Early work in this area includes that of Daumé et al.

Original: [2009] in the natural language processing and that of Ross et al.
Revised: [2009] in the natural language processing and that of Ross et al.

Original: [2010] in imitation learning and robotics, both of which come to the preceding conclusion.
Revised: [2010] in imitation learning and robotics, both of which come to the preceding conclusion.

Original: Recent work by Spencer et al.
Revised: Recent work by Spencer et al.

Original: [2021] shows that there are actually more than one regime of covariate shift.
Revised: [2021] shows that there are actually more than one regime of covariate shift.

Original: In the “easy” regime where the expert is realizable, off-policy methods like behavior cloning match expert performance when data and model capacity are large enough.
Revised: In the “easy” regime where the expert is realizable, off-policy methods like behavior cloning match expert performance when data and model capacity are large enough.

Original: However, in harder regimes where the expert is non-realizable due to model misspeciﬁcation, off-policy methods compound in error and one must rely on on-policy methods, either those that require an interactive expert [Ross et al., 2010] or an interactive simulator [Ziebart et al., 2008,
Revised: However, in harder regimes where the expert is non-realizable due to model misspecification, off-policy methods compound in error and one must rely on on-policy methods, either those that require an interactive expert [Ross et al., 2010] or an interactive simulator [Ziebart et al., 2008,

Original: Swamy et al., 2021].
Revised: Swamy et al., 2021].

Original: One instance of model misspeciﬁcation of practical interest is when the learner is denied state information that the expert uses.
Revised: One instance of model misspecification of practical interest is when the learner is denied state information that the expert uses.

Original: For instance, in self-driving, the human expert might have richer context about the scene than the limited perception system of the car.
Revised: For instance, in self-driving, the human expert might have richer context about the scene than the limited perception system of the car.

Original: While the standard solution is to add a history of past states and actions to the model, practitioners have often noted that this leads to a “latching effect” where the learner simply repeats the past action.
Revised: While the standard solution is to add a history of past states and actions to the model, practitioners have often noted that this leads to a “latching effect” where the learner simply repeats the past action.

Original: For example, Muller et al.
Revised: For example, Muller et al.

Original: [2006] note such latching with steering actions, Kueﬂer et al.
Revised: [2006] note such latching with steering actions, Kuefler et al.

Original: [2017], Bansal et al.
Revised: [2017], Bansal et al.

Original: [2018] note this with braking actions, and Codevilla et al.
Revised: [2018] note this with braking actions, and Codevilla et al.

Original: [2019] with accelerations.
Revised: [2019] with accelerations.

Original: Recent work by Ortega et al.
Revised: Recent work by Ortega et al.

Original: [2021] also points out this latching behavior which they term as “self-delusion.”
Revised: [2021] also points out this latching behavior which they term as “self-delusion.”

Original: Once one identiﬁes the downstream effect of missing context as covariate shift, a natural question is whether an extension of prior covariate-shift-robust imitation learning methods to the space of histories would be able to learn effectively in partial information settings.
Revised: Once one identifies the downstream effect of missing context as covariate shift, a natural question is whether an extension of prior covariate-shift-robust imitation learning methods to the space of histories would be able to learn effectively in partial information settings.

Original: Prior work has answered parts of this question.
Revised: Prior work has answered parts of this question.

Original: For example, Choudhury et al.
Revised: For example, Choudhury et al.

Original: [2018] proves that interactive imitation learning over the space of histories converges to the QMDP approximation of the expert’s policy [Littman et al., 1995].
Revised: [2018] proves that interactive imitation learning over the space of histories converges to the QMDP approximation of the expert’s policy [Littman et al., 1995].

Original: Recent work Ortega et al.
Revised: Recent work Ortega et al.

Original: [2021] views these kind of partial information setting through a causal lens [Pearl et al., 2016], and provides an algorithm (counterfactual teaching) equivalent to the interactive-expert FORWARD algorithm of Ross and Bagnell [2010] under log-loss.
Revised: [2021] views these kind of partial information setting through a causal lens [Pearl et al., 2016], and provides an algorithm (counterfactual teaching) equivalent to the interactive-expert FORWARD algorithm of Ross and Bagnell [2010] under log-loss.

Original: We build upon these analyses by providing conditions under which such approaches will converge to a policy that is equivalent in value to that of the expert in the presence of unobserved contexts.
Revised: We build upon these analyses by providing conditions under which such approaches will converge to a policy that is equivalent in value to that of the expert in the presence of unobserved contexts.

Original: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,
Revised: Our focus on the contextual MDP [Hallak et al., 2015] is because, as argued by [Zhang et al., 2020,

Original: Kumor et al., 2021], context that is updated throughout the episode and is only reﬂected for a single step prevents the learner from reﬁning its estimates via considering history.
Revised: Kumor et al., 2021], context that is updated throughout the episode and is only reflected for a single step prevents the learner from refining its estimates via considering history.

Original: Our results apply to settings beyond the CMDP where a similar asymptotic realizability property holds.
Revised: Our results apply to settings beyond the CMDP where a similar asymptotic realizability property holds.

Original: Tennenholtz et al.
Revised: Tennenholtz et al.

Original: [2021] consider IL in contextual MDPs but give the learner access to the confounder at test time and do not consider policies that operate over the space of histories.
Revised: [2021] consider IL in contextual MDPs but give the learner access to the confounder at test time and do not consider policies that operate over the space of histories.

Original: de Haan et al.
Revised: de Haan et al.

Original: [2019] also consider imitation learning from a causal inference perspective but focus on issues of covariate shift [Spencer et al., 2021], rather than those of missing information. 
Revised: [2019] also consider imitation learning from a causal inference perspective but focus on issues of covariate shift [Spencer et al., 2021], rather than those of missing information. [Swamy et al., 2022] also consider unobserved confounders in IL but focus on correlated action perturbations rather than partial observability. Wen et al. [2020, 2021, 2022], Chuang et al.

Original:   The Latching Effect in Off-Policy Imitation Learning
Revised: [2022] propose various offline approaches to mitigating the latching effect (which they term the copycat problem). However, as we prove below, online interaction with the environment is both necessary and sufficient to prevent compounding errors and the associated latching effect. It is therefore difficult to provide strong theoretical guarantees for their work or argue that their results in simulation would necessarily transfer to real-world problems.  The Latching Effect in Off-Policy Imitation Learning

Original: Consider a ﬁnite-horizon Contextual Markov Decision Process (CMDP) parameterized by h S , A , C , T , r, T i where S , A , C are the state, action, and context spaces, T : S ⇥
Revised: Consider a finite-horizon Contextual Markov Decision Process (CMDP) parameterized by ⟨S , A , C , T , r, T ⟩ where S , A , C are the state, action, and context spaces, T : S ×

Original: A ⇥ C !  ( S ) is the transition operator, r :
Revised: A × C → ∆( S ) is the transition operator, r :

Original: S ⇥
Revised: S ×

Original: A ⇥ C ! [  1 , 1] is the reward function, and T is the horizon.
Revised: A × C → [ − 1 , 1] is the reward function, and T is the horizon.

Original: At the beginning of each episode, a context is sampled from p ( c ) and is held ﬁxed until the next reset.
Revised: At the beginning of each episode, a context is sampled from p ( c ) and is held fixed until the next reset.

Original: Intuitively, there exists a family of reward and transition functions that are indexed by the context for each episode.
Revised: Intuitively, there exists a family of reward and transition functions that are indexed by the context for each episode.

Original: We use h t 2 H to denote a t -step history: ( s 1 , a 1 , . . .
Revised: We use h t ∈ H to denote a t -step history: ( s 1 , a 1 , . . .

Original: , s t ) .
Revised: , s t ) .

Original: We see trajectories generated by an expert policy ⇡ E : S ⇥ C !  ( A ) and search over time-varying policies ⇡ : H !  ( A ) .
Revised: We see trajectories generated by an expert policy π E : S × C → ∆( A ) and search over time-varying policies π : H → ∆( A ) .

Original: We use ⇡ 1 ...t to refer to the sequence of policies that comprise this time-varying policy.
Revised: We use π 1 ...t to refer to the sequence of policies that comprise this time-varying policy.

Original: We assume that our policy class ⇧ is convex and compact.
Revised: We assume that our policy class Π is convex and compact.

Original: We begin with a toy example of the latching effect and how it leads to poor policy performance.
Revised: We begin with a toy example of the latching effect and how it leads to poor policy performance.

Original: Problem 3.1 (Causal Bandit Problem, [Ortega et al., 2021]) .
Revised: Problem 3.1 (Causal Bandit Problem, [Ortega et al., 2021]) .

Original: Consider an episodic MDP with K actions (arms) and a single state.
Revised: Consider an episodic MDP with K actions (arms) and a single state.

Original: At the beginning of each episode, a context c 2 [ K ] is chosen uniformly at random and represents the correct arm for that episode.
Revised: At the beginning of each episode, a context c ∈ [ K ] is chosen uniformly at random and represents the correct arm for that episode.

Original: Pulling an arm leads to binary feedback: + if the arm was correct and  otherwise.
Revised: Pulling an arm leads to binary feedback: + if the arm was correct and − otherwise.

Original: This feedback is ﬂipped with probability ✏ obs 2 [0 , 1] .
Revised: This feedback is flipped with probability ϵ obs ∈ [0 , 1] .

Original: The learner observes expert demonstrations where at each timestep, the expert plays the correct arm with probability 1  ✏ exp 2 [0 , 1] and another arm uniformly at random otherwise.
Revised: The learner observes expert demonstrations where at each timestep, the expert plays the correct arm with probability 1 − ϵ exp ∈ [0 , 1] and another arm uniformly at random otherwise.

Original: Thereward function is 1 for pulling the correct arm and 0 otherwise. We emphasize that the learner does not observe the rewards, just noisy binary feedback as an observation after each pull. As the learner does not observe the correct arm, we are in the partial information setting. However, as one might expect, by pulling all arms enough times and observing the noisy feedback, the learner can narrow down which arm they should pull for the rest of the episode.
Revised: The

Original: Note that the learner stays

Original: 0.8  exp 0.
Revised: 0.8 e exp 0.

Original:  o b s
Revised: e o b s

Original: Consistency of  off 0.
Revised: Consistency of π off 0.

Original: 0.8  exp 0.
Revised: 0.8 e exp 0.

Original:  o b s
Revised: e o b s

Original: Consistency of  on    2000 t 0.
Revised: Consistency of π on    2000 t 0.

Original: P r o b .
Revised: P r o b .

Original: o f C o rr e c t A r m  obs = 0.3,  exp = 0.
Revised: o f C o rr e c t A r m e obs = 0.3, e exp = 0.

Revised: π on π off 1 − e exp 1/ K reward function is 1 for pulling the correct arm and 0 otherwise.

Revised: We emphasize that the learner does not observe the rewards, just noisy binary feedback as an observation after each pull.

Revised: As the learner does not observe the correct arm, we are in the partial information setting.

Revised: However, as one might expect, by pulling all arms enough times and observing the noisy feedback, the learner can narrow down which arm they should pull for the rest of the episode.

Original:  on  off 1   exp 1/ K in the same state after each action so they are free to perform this exploration without long-term consequences.
Revised: Note that the learner stays in the same state after each action so they are free to perform this exploration without long-term consequences.

Original: Observation 1: Off-Policy Methods Have Consistency Phase Transitions.
Revised: Observation 1: Off-Policy Methods Have Consistency Phase Transitions.

Original: In Fig.
Revised: In Fig.

Original: 1, we plot, for a variety of settings of the problem parameters ( ✏ exp , ✏ obs ) whether the learner makes mistakes with a signiﬁcantly higher probability than the expert does for an off-policy IL algorithm and an on-policy IL algorithm.
Revised: 1, we plot, for a variety of settings of the problem parameters ( ϵ exp , ϵ obs ) whether the learner makes mistakes with a significantly higher probability than the expert does for an off-policy IL algorithm and an on-policy IL algorithm.

Original: We give all learners access to the full history of interactions (i.e. arms pulled and noisy feedback observed).
Revised: We give all learners access to the full history of interactions (i.e. arms pulled and noisy feedback observed).

Original: With exactly ✏ obs = 0 .
Revised: With exactly ϵ obs = 0 .

Original: 5 , the learner gets no information from the observations, preventing any algorithm from learning properly.
Revised: 5 , the learner gets no information from the observations, preventing any algorithm from learning properly.

Original: This means that under this particular setting, the problem instance is not identiﬁable , a concept we develop further below.
Revised: This means that under this particular setting, the problem instance is not identifiable , a concept we develop further below.

Original: In contrast to the uniform consistency of the on-policy approach whenever the problem is identiﬁable, we see sharp phase transitions in terms of where the off-policy approach is consistent – a small change to either problem parameter can lead to a drastically different result.
Revised: In contrast to the uniform consistency of the on-policy approach whenever the problem is identifiable, we see sharp phase transitions in terms of where the off-policy approach is consistent – a small change to either problem parameter can lead to a drastically different result.

Original: Observation 2: Off-Policy Methods Produce Latching Policies.
Revised: Observation 2: Off-Policy Methods Produce Latching Policies.

Original: At the ﬁrst timestep, all learners pick an arm uniformly at random as they have no information about the context.
Revised: At the first timestep, all learners pick an arm uniformly at random as they have no information about the context.

Original: In the cases where the off-policy approach is inconsistent, it continues to pick this arm ad inﬁnitum as it treats its own past actions as though they were the expert’s.
Revised: In the cases where the off-policy approach is inconsistent, it continues to pick this arm ad infinitum as it treats its own past actions as though they were the expert’s.

Original: This is what leads to the 1 K = 15 success rate seen in the rightmost part of Fig.
Revised: This is what leads to the 1 K = 15 success rate seen in the rightmost part of Fig.

Original: 1, even after 2000 timesteps of experience telling the learner that another arm should be pulled.
Revised: 1, even after 2000 timesteps of experience telling the learner that another arm should be pulled.

Original: Put differently, the off-policy learner collapses its uncertainty over the correct arm too quickly for the negative feedback it receives to push it to a different arm.
Revised: Put differently, the off-policy learner collapses its uncertainty over the correct arm too quickly for the negative feedback it receives to push it to a different arm.

Original: Concerningly, this effect appears to be extremely sensitive to the parameters of the problem, rendering it difﬁcult to predict and hedge against.
Revised: Concerningly, this effect appears to be extremely sensitive to the parameters of the problem, rendering it difficult to predict and hedge against.

Original: Observation 3: Low Density Ratios Help Off-Policy Methods.
Revised: Observation 3: Low Density Ratios Help Off-Policy Methods.

Original: Looking at the left side of Fig.
Revised: Looking at the left side of Fig.

Original: 1, a natural question might be why, as we increase ✏ exp , the off-policy approach is consistent for a wider spectrum of ✏ obs values.
Revised: 1, a natural question might be why, as we increase ϵ exp , the off-policy approach is consistent for a wider spectrum of ϵ obs values.

Original: Observe that as we increase ✏ exp , the expert has a higher chance of making a mistake, bringing its trajectory distribution closer to that of the uniform policy and lowering the maximum density ratio between learner and expert trajectory distributions.
Revised: Observe that as we increase ϵ exp , the expert has a higher chance of making a mistake, bringing its trajectory distribution closer to that of the uniform policy and lowering the maximum density ratio between learner and expert trajectory distributions.

Original: As was established by
Revised: As was established by

Original: Spencer et al.
Revised: Spencer et al.

Original: [2021], a low maximum density ratio is a sufﬁcient condition for behavioral cloning to be consistent.
Revised: [2021], a low maximum density ratio is a sufficient condition for behavioral cloning to be consistent.

Original: This experiment appears to echo their theoretical results.
Revised: This experiment appears to echo their theoretical results.

Original: Putting together Fig.
Revised: Putting together Fig.

Original: 1, on-policy approaches appear to be able to handle hidden context given access to history while off-policy can do so only in a way that depends heavily on problem parameters.
Revised: 1, on-policy approaches appear to be able to handle hidden context given access to history while off-policy can do so only in a way that depends heavily on problem parameters.

Original: A Bayesian Perspective on the Latching Effect

Original: To begin to explain this difference in behavior of on-policy and off-policy algorithms, we consider the structural causal model (SCM) each of these classes of algorithms is implicitly assuming.

Original: sssaaac
Revised: sssaaac

Original: T
Revised: T

Original: T ⇡ E ⇡ E ⇡ E ⇡ E ⇡ E (a) ⌧ ⇠ ⇡ E hhhaaac
Revised: T π E π E π E π E π E (a) τ ∼ π E hhhaaac

Original: T
Revised: T

Original: T ⇡⇡⇡(b) On-Policy hhhaaac
Revised: T πππ(b) On-Policy hhhaaac

Original: T
Revised: T

Original: T ⇡ E ⇡ E ⇡⇡ E (c) Off-Policy
Revised: T π E π E ππ E (c) Off-Policy

Revised:  A Bayesian Perspective on the Latching Effect

Revised: To begin to explain this difference in behavior of on-policy and off-policy algorithms, we consider the structural causal model (SCM) each of these classes of algorithms is implicitly assuming.

Original: 4.1 SCMs for Imitation Learning
Revised: 4.1 SCMs for Imitation Learning

Original: We observe that expert trajectories of length t are generated according to
Revised: We observe that expert trajectories of length t are generated according to

Original: [Equation]
Revised: [Equation]

Original: while learner trajectories are generated according to
Revised: while learner trajectories are generated according to

Original: [Equation]
Revised: [Equation]

Original: We use ⌧ ⇠ ⇡ E and ⌧ ⇠ ⇡ to denote T -step trajectories sampled according to the above distributions.
Revised: We use τ ∼ π E and τ ∼ π to denote T -step trajectories sampled according to the above distributions.

Original: We deﬁne our value and Q functions as usual: V ⇡ ( s ) = E ⌧ ⇠ ⇡ | s t = s [ P
Revised: We define our value and Q functions as usual: V π

Original: [Equation]
Revised: [Equation]

Original: )] , Q ⇡
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  ) =
Revised: Q π ( s, a ) =

Original: E ⌧ ⇠ ⇡ | s t = s,a t = a [ 
Revised: E τ ∼ π | s t = s,a t = a [ (cid:80)

Original: P
Revised: Tt ′

Original: Tt 0 =1 r ( s t 0 , a t 0 )] .
Revised: =1 r ( s t ′ , a t ′ )] .

Original: Lastly, let performance be J ( ⇡ ) =
Revised: Lastly, let performance be J ( π ) =

Original: E ⌧ ⇠ ⇡ [ P
Revised: E τ ∼ π

Original:  Tt =1 r ( s t , a t )] .
Revised: [ (cid:80) Tt =1 r ( s t , a t )] .

Original: The key difference between on-policy and off-policy imitation learning algorithms is the difference between the center and right SCMs of Fig.
Revised: The key difference between on-policy and off-policy imitation learning algorithms is the difference between the center and right SCMs of Fig.

Original: On-policy algorithms assume the data they have seen thus far is generated by executing learner ⇡ (b) while off-policy algorithms assume it is generated by executing the expert policy ⇡ E (c).
Revised: On-policy algorithms assume the data they have seen thus far is generated by executing learner π (b) while off-policy algorithms assume it is generated by executing the expert policy π E (c).

Original: The off-policy approximation is what leads to the latching behavior observed empirically: even if the ﬁrst action was chosen at random, by treating it as though it was produced by the expert ⇡ E (who sees the context c and therefore picks actions that are correlated across time), the learner is likely to continue to repeat the past action .
Revised: The off-policy approximation is what leads to the latching behavior observed empirically: even if the first action was chosen at random, by treating it as though it was produced by the expert π E (who sees the context c and therefore picks actions that are correlated across time), the learner is likely to continue to repeat the past action .

Original: This is also what we observe empirically in the causal bandit problem in the cases where off-policy IL does not work: the learner continues to play the ﬁrst arm it chose, ignoring the feedback it gets that it is repeatedly making a mistake, and only matching expert performance on a 1 K of episodes.
Revised: This is also what we observe empirically in the causal bandit problem in the cases where off-policy IL does not work: the learner continues to play the first arm it chose, ignoring the feedback it gets that it is repeatedly making a mistake, and only matching expert performance on a 1 K of episodes.

Original: We note this gives the off-policy learner a choice between ﬁre and pyre [Eliot, 1942]: they need to use history to narrow down the context but if they do, they can learn a latching policy that performs poorly at test-time.
Revised: We note this gives the off-policy learner a choice between fire and pyre (in the words of Eliot [1942]): they need to use history to narrow down the context but if they do, they can learn a latching policy that performs poorly at test-time.

Original: 4.2 Off Policy Methods Have an Incorrect Context Posterior
Revised: 4.2 Off Policy Methods Have an Incorrect Context Posterior

Original: Another way of seeing this point is by considering what the posterior over the confounder would be under samples from each of these SCMs.
Revised: Another way of seeing this point is by considering what the posterior over the confounder would be under samples from each of these SCMs.

Original: If the learner was able to accurately pin down the correct arm, they would be able to easily reproduce the expert policy.
Revised: If the learner was able to accurately pin down the correct arm, they would be able to easily reproduce the expert policy.

Original: Thus, we can focus on correct-arm identiﬁcation via Bayes Rule.
Revised: Thus, we can focus on correct-arm identification via Bayes Rule.

Original: Assuming a uniform prior over contexts,
Revised: Assuming a uniform prior over contexts,

Original: [Equation]
Revised: [Equation]

Original: Under the off-policy graphical model, p ( c, h t ) / p ( ⌧ ; ⇡ E ) : the probability of a history h t under the expert’s distribution (Fig. 2, (a)).
Revised: Under the off-policy graphical model, p ( c, h t ) ∝ p ( τ ; π E ) : the probability of a history h t under the expert’s distribution (Fig. 2, (a)).

Original: Expanding terms, we arrive at the following expression
Revised: Expanding terms, we arrive at the following expression

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The off-policy posterior over contexts is
Revised: The off-policy posterior over contexts is

Original: [Equation]
Revised: [Equation]

Original: Notice that for the expert policy, the context c directly inﬂuences actions so conditioning on actions when attempting to predict c is correct.
Revised: Notice that for the expert policy, the context c directly influences actions so conditioning on actions when attempting to predict c is correct.

Original: We highlight in green the term that encodes this dependence.
Revised: We highlight in green the term that encodes this dependence.

Original: In contrast, if we assume the learner has no access to the context except via its inﬂuence on the history of states, we should instead treat actions as interventions [Pearl et al., 2016] that provide no further information about the context.
Revised: In contrast, if we assume the learner has no access to the context except via its influence on the history of states, we should instead treat actions as interventions [Pearl et al., 2016] that provide no further information about the context.

Original: The on-policy graphical model (Fig. 2, (b)) does just this, allowing us to compute the correct posterior over contexts.
Revised: The on-policy graphical model (Fig. 2, (b)) does just this, allowing us to compute the correct posterior over contexts.

Original: Formally,
Revised: Formally,

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: The on-policy posterior over contexts is 
Revised: The on-policy posterior over contexts is p on

Original: [Equation]
Revised: [Equation]

Revised: c, h

Revised: [Equation]

Revised: ∝

Revised: [Equation]

Original: where the equality follows from the fact that
Revised: where the equality follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: and the standard do () -calculus rules [Pearl et al., 2016].
Revised: and the standard do () -calculus rules [Pearl et al., 2016].

Original: Intuitively, we are leveraging the fact that the learner’s actions have all their dependence on the context mediated through the history of states to ignore them in our posterior calculation.
Revised: Intuitively, we are leveraging the fact that the learner’s actions have all their dependence on the context mediated through the history of states to ignore them in our posterior calculation.

Original: Notice that this expression matches the off-policy posterior except for the term in green: because the on-policy learner knows that the actions in the history were not produced by the expert, it does not weight them by the expert’s probability of playing them in its posterior update .
Revised: Notice that this expression matches the off-policy posterior except for the term in green: because the on-policy learner knows that the actions in the history were not produced by the expert, it does not weight them by the expert’s probability of playing them in its posterior update .

Original: Graphically, as far as the posterior over the context is concerned, on-policy approaches correspond to the SCM in Fig.
Revised: Graphically, as far as the posterior over the context is concerned, on-policy approaches correspond to the SCM in Fig.

Original: 3, left.
Revised: 3, left.

Original: [Figure]
Revised: hhhaaac T T C0.

Revised: 1 Off-Policy, t=

Revised: Learner picks arm 1 randomly

Revised: C 2 C

Original:  By ignoring its own incorrect actions early on, the on-policy learner doesn’t over-index on them and come to a false conclusion as to what the value of the context is, as illustrated in Fig.
Revised: C0. 1 Off-Policy, t= C 2 C By ignoring its own incorrect actions early on, the on-policy learner doesn’t over-index on them and come to a false conclusion as to what the value of the context is, as illustrated in Fig.

Original: 3, right.
Revised: 3, right.

Original: We’ve already seen evidence of how the addition or removal of the green term can lead to markedly different results: the learners we used to generate Fig.
Revised: We’ve already seen evidence of how the addition or removal of the green term can lead to markedly different results: the learners we used to generate Fig.

Original: 1 had policies given by
Revised: 1 had policies given by

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: (7) and

Original: [Equation]
Revised: [Equation]

Original: Note that these policies are exactly equivalent except for the green term in the posterior of the off-policy ⇡ o ↵ , yet respond quite differently as the parameters of the problem are changed.
Revised: Note that these policies are exactly equivalent except for the green term in the posterior of the off-policy π off , yet respond quite differently as the parameters of the problem are changed.

Original: Also note that p on ( c | h t ) requires interaction with the environment to evaluate, while p o ↵ ( c | h t ) does not.
Revised: Also note that p on ( c | h t ) requires interaction with the environment to evaluate, while p off ( c | h t ) does not.

Original:  A Moment-Matching Solution to the Latching Effect
Revised:  A Moment-Matching Solution to the Latching Effect

Original: We now turn our attention to generalizing this argument beyond Bayesian learners, including to those that match sufﬁcient statistics of expert behavior – moments – rather than maintaining explicit posteriors, and providing value equivalence guarantees for history-equipped on-policy learners.
Revised: We now turn our attention to generalizing this argument beyond Bayesian learners, including to those that match sufficient statistics of expert behavior – moments – rather than maintaining explicit posteriors, and providing value equivalence guarantees for history-equipped on-policy learners.

Original: 5.1 A Quick Review of Moment-Matching in Imitation Learning
Revised: 5.1 A Quick Review of Moment-Matching in Imitation Learning

Original: First, as in Swamy et al.
Revised: First, as in Swamy et al.

Original: [2021], we deﬁne F Q E as the set of on- Q moments , with f 2 F Q E satisfying type signature
Revised: [2021], we define F Q E as the set of on- Q moments , with f ∈ F Q E satisfying type signature

Original: S ⇥
Revised: S ×

Original: A ⇥ C ! [  T, T ] .
Revised: A × C → [ − T, T ] .

Original: Intuitively, F Q E spans the set of possible expert Q -functions.
Revised: Intuitively, F Q E spans the set of possible expert Q -functions.

Original: We require the actual expert Q -function to be contained: Q ⇡ E 2 F Q E .
Revised: We require the actual expert Q -function to be contained: Q π E ∈ F Q E .

Original: We assume that F Q E is convex, compact, closed under negation, and ﬁnite dimensional.
Revised: We assume that F Q E is convex, compact, closed under negation, and finite dimensional.

Original: Second, deﬁne
Revised: Second, define

Original: F Q be the class of off- Q moments (i.e. 8 ⇡ 2 ⇧ , Q ⇡ 2 F Q ) and satisfy the same function-class assumptions as
Revised: F Q be the class of off- Q moments (i.e. ∀ π ∈ Π , Q π ∈ F Q ) and satisfy the same function-class assumptions as

Original: F Q E .
Revised: F Q E .

Original: Third, let F r denote the class of reward moments and also satisfy the same function-class assumptions.
Revised: Third, let F r denote the class of reward moments and also satisfy the same function-class assumptions.

Original: All f 2 F r satisfy type signature S ⇥ A ⇥ C !
Revised: All f ∈ F r satisfy type signature S × A × C →

Original: [  1 , 1] and we assume that r 2 F r .
Revised: [ − 1 , 1] and we assume that r ∈ F r .

Original: Swamy et al.
Revised: Swamy et al.

Original: [2021] prove that if one is able to approximately solve a two-player zero-sum momentmatching game between a policy player (that picks from ⇧ ) and a discriminator (that picks from
Revised: [2021] prove that if one is able to approximately solve a two-player zero-sum momentmatching game between a policy player (that picks from Π ) and a discriminator (that picks from

Original: F Q E , F Q , or F r ), one has a bound on the performance difference between the learner and the expert.
Revised: F Q E , F Q , or F r ), one has a bound on the performance difference between the learner and the expert.

Original: Depending on which class the discriminator selects from, one ends up with a different bound.
Revised: Depending on which class the discriminator selects from, one ends up with a different bound.

Original: For example, if one solves the game with the following payoff and f 2 F r to an ✏ -approximate Nash equilibrium,
Revised: For example, if one solves the game with the following payoff and f ∈ F r to an ϵ -approximate Nash equilibrium,

Original: [Equation]
Revised: [Equation]

Original: one has a guarantee that J ( ⇡ E )  J ( ⇡ )  ✏ T .
Revised: one has a guarantee that J ( π E ) − J ( π ) ≤ ϵT .

Original: Unfortunately, directly solving such a game is not possible in the hidden context setting as we do not see the contexts and therefore cannot evaluate the moment functions.
Revised: Unfortunately, directly solving such a game is not possible in the hidden context setting as we do not see the contexts and therefore cannot evaluate the moment functions.

Original: 1 We turn our attention to adapting moment-matching to the contextual setting.
Revised: 1 We turn our attention to adapting moment-matching to the contextual setting.

Original: 5.2 Moment-Matching with Unobserved Contexts
Revised: 5.2 Moment-Matching with Unobserved Contexts

Original: For each moment class, we assume the existence of an observable moment class that operates over the space of histories (i.e. H ⇥ A ! R ) and has members that eventually produce outputs close to that of their context-dependent counterparts.
Revised: For each moment class, we assume the existence of an observable moment class that operates over the space of histories (i.e. H × A → R ) and has members that eventually produce outputs close to that of their context-dependent counterparts.

Original: In math,
Revised: In math,

Original: Assumption 5.1 (Asymptotic On- Q Moment Identiﬁability) .
Revised: Assumption 5.1 (Asymptotic On- Q Moment Identifiability) .

Original: [Equation]
Revised: [Equation]

Original: Analogous conditions can be deﬁned for F r and F Q , giving us ˜ F r and ˜ F Q .
Revised: Analogous conditions can be defined for F r and F Q , giving us ˜ F r and ˜ F Q .

Original: We note that these condi- tions are asymptotic in nature – we use  ( t ) to refer to the ﬁnite-horizon expected difference between the observable and context-dependent moments.
Revised: We note that these condi- tions are asymptotic in nature – we use δ ( t ) to refer to the finite-horizon expected difference between the observable and context-dependent moments.

Original: We use H to denote the moment recoverability constant , which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake [Swamy et al., 2021]:[Equation]
Revised: We use H to denote the moment recoverability constant , which bounds how much total cost is incurred for the expert to recover from an arbitrary mistake [Swamy et al., 2021]:

Original: [Equation]
Revised: [Equation]

Original: For problems where the expert is able to effectively correct learner mistakes, this quantity can be signiﬁcantly smaller than the horizon.
Revised: For problems where the expert is able to effectively correct learner mistakes, this quantity can be significantly smaller than the horizon.

Original: Deﬁne ˜ F on = { ˜ f/ 2 H : ˜ f 2 ˜ F Q E } and ˜ F o ↵ = { ˜ f/ 2 T : ˜ f 2 ˜
Revised: Define ˜ F on = { ˜ f/ 2 H : ˜ f ∈ ˜ F Q E } and ˜ F off = { ˜ f/ 2 T : ˜ f ∈ ˜

Original: F Q } to be scaled-down versions of the observable moments such that their range is [  1 , 1] .
Revised: F Q } to be scaled-down versions of the observable moments such that their range is [ − 1 , 1] .

Original: We can now deﬁne our moment-matching errors:
Revised: We can now define our moment-matching errors:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As argued by Swamy et al.
Revised: As argued by Swamy et al.

Original: [2021], ✏ on ( t ) governs the performance of on- Q algorithms like DAgger [Ross et al., 2010], ✏ o ↵ ( t ) the performance of off- Q algorithms like behavioral cloning [Pomerleau, 1989], and ✏ rew ( t ) the performance of reward-matching algorithms like MaxEnt IRL
Revised: [2021], ϵ on ( t ) governs the performance of on- Q algorithms like DAgger [Ross et al., 2010], ϵ off ( t ) the performance of off- Q algorithms like behavioral cloning [Pomerleau, 1989], and ϵ rew ( t ) the performance of reward-matching algorithms like MaxEnt IRL

Original: [Ziebart et al., 2008] and GAIL [Ho and Ermon, 2016].
Revised: [Ziebart et al., 2008] and GAIL [Ho and Ermon, 2016].

Original: With these deﬁnitions laid out, we can now prove how well each class of algorithms handles unobserved contexts.
Revised: With these definitions laid out, we can now prove how well each class of algorithms handles unobserved contexts.

Original: 5.3 Asymptotic Realizability in Imitation Learning
Revised: 5.3 Asymptotic Realizability in Imitation Learning

Original: For some partial information problems, the use of history coupled with on-policy feedback might allow the learner to eventually match expert performance.
Revised: For some partial information problems, the use of history coupled with on-policy feedback might allow the learner to eventually match expert performance.

Original: This is an example of a more general phenomenon we term asymptotic realizability (AR), in which the learner is able to perform as well as the expert does with high probability after observing an arbitrary history of some length.
Revised: This is an example of a more general phenomenon we term asymptotic realizability (AR), in which the learner is able to perform as well as the expert does with high probability after observing an arbitrary history of some length.

Original: We begin by deﬁning the average imitation gap or AIG for short.
Revised: We begin by defining the average imitation gap or AIG for short.

Original: Deﬁnition 5.2 (AIG( ⇡ , T)) .
Revised: Definition 5.2 (AIG( π , T)) .

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne our performance target in AR problems.
Revised: We now define our performance target in AR problems.

Original: Deﬁnition 5.3 (Asymptotic Value Equivalence (AVE)) .
Revised: Definition 5.3 (Asymptotic Value Equivalence (AVE)) .

Original: We say that policy ⇡ is asymptotically value-equivalent (AVE) when the following condition holds true:
Revised: We say that policy π is asymptotically value-equivalent (AVE) when the following condition holds true:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this condition means that the learner performs as well as the expert does on average, given enough time.
Revised: Intuitively, this condition means that the learner performs as well as the expert does on average, given enough time.

Original: Put differently, we do not penalize the learner for initial mistakes as long as they are able to learn enough from them to match expert performance.
Revised: Put differently, we do not penalize the learner for initial mistakes as long as they are able to learn enough from them to match expert performance.

Original: We will proceed by studying the AIG properties of on-policy and off-policy imitation learning algorithms.
Revised: We will proceed by studying the AIG properties of on-policy and off-policy imitation learning algorithms.

Original: We are now ready to state our main result:
Revised: We are now ready to state our main result:

Original: Theorem 5.4 (AVE of IL Algorithms) .
Revised: Theorem 5.4 (AVE of IL Algorithms) .

Original: Deﬁne lim sup t ! 1 ✏ on ( t ) =
Revised: Define lim sup t →∞ ϵ on ( t ) =

Original: ✏ on [Equation] 1 ) , lim T !
Revised: ϵ on ( ∞ ) , lim T →∞ (cid:80) Tt

Original: [Equation]
Revised: [Equation]

Original: ↵ [Equation]  o ↵ ( t ) =
Revised: off ( t )+ δ off ( t ) =

Original: ⌃ o ↵ ( 1 ) , and lim sup t ! 1 ✏ rew ( t ) =
Revised: Σ off ( ∞ ) , and lim sup t →∞ ϵ rew ( t ) =

Original: ✏ rew ( 1 ) .
Revised: ϵ rew ( ∞ ) .

Original: For all (C)MDPs and ⇡ ,
Revised: For all (C)MDPs and π ,

Original: [Equation]
Revised: [Equation]

Original: In words, if either an on- Q or reward-matching learner is able to achieve 0 moment matching error asymptotically, they will achieve the same average value as the expert.
Revised: In words, if either an on- Q or reward-matching learner is able to achieve 0 moment matching error asymptotically, they will achieve the same average value as the expert.

Original: Even if they are unable to do so, the AIG will be bounded by a constant.
Revised: Even if they are unable to do so, the AIG will be bounded by a constant.

Original: In contrast, the result for off-policy algorithms is much weaker.
Revised: In contrast, the result for off-policy algorithms is much weaker.

Original: Notice that instead of taking a lim sup which, roughly speaking, captures the asymptotic error, the off-policy bound is in terms of the sum of errors, which factors in errors made early on.
Revised: Notice that instead of taking a lim sup which, roughly speaking, captures the asymptotic error, the off-policy bound is in terms of the sum of errors, which factors in errors made early on.

Original: This result indicates that an off-policy learner that makes mistakes early on in the episode (as is likely for contextual problems) might be unable to eventually match expert performance, even when the problem is recoverable.
Revised: This result indicates that an off-policy learner that makes mistakes early on in the episode (as is likely for contextual problems) might be unable to eventually match expert performance, even when the problem is recoverable.

Original: We prove that there exist certain problems for which this bound is tight.
Revised: We prove that there exist certain problems for which this bound is tight.

Original: Theorem 5.5 (Off-Policy AVE Lower Bound) .
Revised: Theorem 5.5 (Off-Policy AVE Lower Bound) .

Original: There exist CMDPs and ⇡ s.t.
Revised: There exist CMDPs and π s.t.

Original: lim sup t ! 1 ✏ o ↵ ( t ) =for which
Revised: lim sup t →∞ ϵ off ( t ) = for which

Original: [Equation]
Revised: [Equation]

Original:  In words, this theorem says that for certain problems, even if an off-policy learner can drive down error asymptotically, they might be doomed as far as AVE because of mistakes made early on.
Revised: [Equation] [Figure] In words this theorem says that for certain problems, even if an off-policy learner can drive down error asymptotically, they might be doomed as far as AVE because of mistakes made early on.

Original: We prove these results in Appendix A. To come full circle, we now argue that CMDPs (including our causal bandit problem) that satisfy the asymptotic moment identiﬁability conditions along with an asymptotic realizability condition on the policy class enable the learner to match expert performance:[Figure]
Revised: We prove these results in Appendix A. To come full circle, we now argue that CMDPs (including our causal bandit problem) that satisfy the asymptotic moment identifiability conditions along with an asymptotic realizability condition on the policy class enable the learner to match expert performance:

Original: Assumption 5.6 (Asymptotic Realizability) .
Revised: Assumption 5.6 (Asymptotic Realizability) .

Original: Asymptotic Moment Identiﬁability holds and 9 ⇡ 2 ⇧ s.t.
Revised: Asymptotic Moment Identifiability holds and ∃ π ∈ Π

Original: lim t !
Revised: s.t.

Original: 1 ✏ on ( t ) = 0 , lim t ! 1 ✏ o ↵ ( t ) = 0 , and lim t ! 1 ✏ rew ( t ) =0 .
Revised: lim t →∞ ϵ on ( t ) = 0 , lim t →∞ ϵ off ( t ) = 0 , and lim t →∞ ϵ rew ( t ) =

Original: Note this assumption is weaker than a standard realizability assumption – it is saying that along certain moments of interest, we can asymptotically match the expert.
Revised: Note this assumption is weaker than a standard realizability assumption – it is saying that along certain moments of interest, we can asymptotically match the expert.

Original: Plugging in this assumption into Theorem 5.4 tells us that on such problems, an on-policy learner will be able to achieve AVE.
Revised: Plugging in this assumption into Theorem 5.4 tells us that on such problems, an on-policy learner will be able to achieve AVE.

Original: We now turn our attention to efﬁciently computing such a policy.
Revised: We now turn our attention to efficiently computing such a policy.

Original: We prove that by solving an approximate equilibrium computation game over the space of history-based policies and the space of observable moments (which can be done efﬁciently with no-regret algorithms, [Freund and Schapire, 1997]), one can ﬁnd a policy that achieves a low AIG.
Revised: We prove that by solving an approximate equilibrium computation game over the space of history-based policies and the space of observable moments (which can be done efficiently with no-regret algorithms, [Freund and Schapire, 1997]), one can find a policy that achieves a low AIG.

Original: Theorem 5.7.
Revised: Theorem 5.7.

Original: For any contextual MDP and policy class that satisﬁes Asymptotic Realizability, let ⇡ be an ✏ -approximate Nash equilibrium strategy for the inﬁnite horizon reward-matching or on- Q -matching game.
Revised: For any contextual MDP and policy class that satisfies Asymptotic Realizability, let π be an ϵ -approximate Nash equilibrium strategy for the infinite horizon reward-matching or on- Q -matching game.

Original: Then, we know that lim T !
Revised: Then, we know that lim T →∞ AIG

Original: 1 AIG( ⇡ , T )  ✏ or lim T !
Revised: [Equation] π, T ) [Equation] ϵ or lim T →∞ AIG

Original: 1 AIG( ⇡ , T )  H ✏ .
Revised: [Equation] [Equation] ≤ Hϵ .

Original: In short, by solving an on-policy moment-matching problem over policies that have access to history, we have strong guarantees of matching expert performance on asymptotically realizable problems.
Revised: In short, by solving an on-policy moment-matching problem over policies that have access to history, we have strong guarantees of matching expert performance on asymptotically realizable problems.

Original: We re-iterate that we have no such guarantees for off-policy algorithms.
Revised: We re-iterate that we have no such guarantees for off-policy algorithms.

Original: We also note that our causal bandit problem satisﬁes these assumptions, providing theoretical justiﬁcation for our results.
Revised: We also note that our causal bandit problem satisfies these assumptions, providing theoretical justification for our results.

Original: Corollary 5.8.
Revised: Corollary 5.8.

Original: There exists a singleton observable moment class such that the Causal Bandit
Revised: There exists a singleton observable moment class such that the Causal Bandit

Original: Problem satisﬁes Asymptotic Realizability, implying that the iterates produced by an on-policy moment matching algorithm will achieve AVE.
Revised: Problem satisfies Asymptotic Realizability, implying that the iterates produced by an on-policy moment matching algorithm will achieve AVE.

Original: Putting it all together, for asymptotically realizable problems, on-policy imitation learning algorithms have stronger guarantees than their off-policy counterparts with respect to asymptotic value equivalence.
Revised: Putting it all together, for asymptotically realizable problems, on-policy imitation learning algorithms have stronger guarantees than their off-policy counterparts with respect to asymptotic value equivalence.

Original: If a contextual MDP satisﬁes these conditions (like with our Causal Bandit Problem), the preceding results apply.
Revised: If a contextual MDP satisfies these conditions (like with our Causal Bandit Problem), the preceding results apply.

Original: We see that our theory matches our experiments: on-policy algorithms appear to work on all identiﬁable instances of the causal bandit problems, while off-policy algorithms have much weaker performance.
Revised: We see that our theory matches our experiments: on-policy algorithms appear to work on all identifiable instances of the causal bandit problems, while off-policy algorithms have much weaker performance.

Original: We now turn our attention to more complex CMDPs that satisfy our assumptions to further validate our theory empirically.
Revised: We now turn our attention to more complex CMDPs that satisfy our assumptions to further validate our theory empirically.

Original:  Experiments
Revised:  Experiments

Original: We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai [2016] that is inspired by the multi-task reinforcement learning setups of Finn et al.
Revised: We conduct experiments in a CMDP extension of the standard PyBullet tasks Coumans and Bai [2016] that is inspired by the multi-task reinforcement learning setups of Finn et al.

Original: In these tasks, the agent is rewarded for running at a particular velocity that is randomly sampled at the beginning of each episode.
Revised: In these tasks, the agent is rewarded for running at a particular velocity that is randomly sampled at the beginning of each episode.

Original: We train expert policies that have access to this privileged information.
Revised: We train expert policies that have access to this privileged information.

Original: We compare two algorithms: the off-policy behavioral cloning ( BC )
Revised: We compare two algorithms: the off-policy behavioral cloning ( BC )

Original: [Pomerleau, 1989] and the on-policy
Revised: [Pomerleau, 1989] and the on-policy

Original: DAgger ( DAgger )
Revised: DAgger ( DAgger )

Original: [Ross et al., 2010] that either have access to the immediate state or the last ﬁve timesteps of history (for which we use the sufﬁx -SEQ ).
Revised: [Ross et al., 2010] that either have access to the immediate state or the last five timesteps of history (for which we use the suffix -SEQ ).

Original: In the bar plots of Fig. 4, we see that without access to history, BC performs poorly.
Revised: In the bar plots of Fig. 4, we see that without access to history, BC performs poorly.

Original: As our theory predicts, when we equip the BC learner with history, we actually see it perform worse on average, exhibiting the latching behavior that has been observed repeatedly in practice.
Revised: As our theory predicts, when we equip the BC learner with history, we actually see it perform worse on average, exhibiting the latching behavior that has been observed repeatedly in practice.

Original: In contrast, we see that equipping our on-policy learner with access to the last few observations and actions does not lead to a sharp decline in terms of performance.
Revised: In contrast, we see that equipping our on-policy learner with access to the last few observations and actions does not lead to a sharp decline in terms of performance.

Original: We use MSE as a surrogate for moment-matching error.
Revised: We use MSE as a surrogate for moment-matching error.

Original: We see that on both environments, on-policy methods are far better at matching expert actions on their own rollout distribution.
Revised: We see that on both environments, on-policy methods are far better at matching expert actions on their own rollout distribution.

Original: We also see that with enough time, DAgger-SEQ is able to predict actions better than DAgger (lower ✏ on (1000) ).
Revised: We also see that with enough time, DAgger-SEQ is able to predict actions better than DAgger (lower ϵ on (1000) ).

Original: Putting together these observations, it appears as though on-policy methods with access to history are the most robust against partial information, agreeing with our theory.
Revised: Putting together these observations, it appears as though on-policy methods with access to history are the most robust against partial information, agreeing with our theory.

Original: We release our code at https://github.com/gkswamy98/sequence_model_il .
Revised: We release our code at https://github.com/gkswamy98/sequence_model_il .

Original:  Conclusion
Revised:  Conclusion

Original: We study asymptotically realizable imitation learning problems where the learner’s ability to mimic the expert increases over the timesteps of the problem.
Revised: We study asymptotically realizable imitation learning problems where the learner’s ability to mimic the expert increases over the timesteps of the problem.

Original: We ﬁnd that under identiﬁability and recoverability assumptions, on-policy algorithms are able to match time-averaged expert value, in contrast to off-policy algorithms which might pay for their early mistakes for the rest of the horizon.
Revised: We find that under identifiability and recoverability assumptions, on-policy algorithms are able to match time-averaged expert value, in contrast to off-policy algorithms which might pay for their early mistakes for the rest of the horizon.

Original: We demonstrate that empirically, off-policy algorithms are not able to take advantage of history to uncover missing information.
Revised: We demonstrate that empirically, off-policy algorithms are not able to take advantage of history to uncover missing information.

Original: We believe our results point towards a uniﬁed explanation of both empirical successes and failures.
Revised: We believe our results point towards a unified explanation of both empirical successes and failures.

Original:  Acknowledgments
Revised:  Acknowledgments

Original: ZSW is supported in part by the NSF FAI Award #1939606, a Google Faculty Research Award, a J.P.
Revised: ZSW is supported in part by the NSF FAI Award #1939606, a Google Faculty Research Award, a J.P.

Original: Morgan Faculty Award, a Facebook Research Award, an Okawa Foundation Research Grant, and a Mozilla Research Grant.
Revised: Morgan Faculty Award, a Facebook Research Award, an Okawa Foundation Research Grant, and a Mozilla Research Grant.

Original: GS is supported computationally by a GPU award from NVIDIA and emotionally by his family and friends.
Revised: GS is supported computationally by a GPU award from NVIDIA and emotionally by his family and friends.