Revised: R OBUST

Revised: I MITATION

Revised: L

Original: I MITATION L EARNING FROM C ORRUPTED D EMONSTRATIONS
Revised: EARNING FROM C ORRUPTED D EMONSTRATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider ofﬂine Imitation Learning from corrupted demonstrations where a constant fraction of data can be noise or even arbitrary outliers.
Revised: We consider ofﬂine Imitation Learning from corrupted demonstrations where a constant fraction of data can be noise or even arbitrary outliers.

Original: Classical approaches such as Behavior Cloning assumes that demonstrations are collected by an presumably optimal expert, hence may fail drastically when learning from corrupted demonstrations.
Revised: Classical approaches such as Behavior Cloning assumes that demonstrations are collected by an presumably optimal expert, hence may fail drastically when learning from corrupted demonstrations.

Original: We propose a novel robust algorithm by minimizing a
Revised: We propose a novel robust algorithm by minimizing a

Original: Median-of-Means (MOM) objective which guarantees the accurate estimation of policy, even in the presence of constant fraction of outliers.
Revised: Median-of-Means (MOM) objective which guarantees the accurate estimation of policy, even in the presence of constant fraction of outliers.

Original: Our theoretical analysis shows that our robust method in the corrupted setting enjoys nearly the same error scaling and sample complexity guarantees as the classical Behavior Cloning in the expert demonstration setting.
Revised: Our theoretical analysis shows that our robust method in the corrupted setting enjoys nearly the same error scaling and sample complexity guarantees as the classical Behavior Cloning in the expert demonstration setting.

Original: Our experiments on continuous-control benchmarks validate that existing algorithms are fragile under corrupted demonstration while our method exhibits the predicted robustness and effectiveness.
Revised: Our experiments on continuous-control benchmarks validate that existing algorithms are fragile under corrupted demonstration while our method exhibits the predicted robustness and effectiveness.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have witnessed the success of using autonomous agent to learn and adapt to complex tasks and environments in a range of applications such as playing games (e.g. Mnih et al., 2015; Silver et al., 2018;
Revised: Recent years have witnessed the success of using autonomous agent to learn and adapt to complex tasks and environments in a range of applications such as playing games (e.g. Mnih et al., 2015; Silver et al., 2018;

Original: Vinyals et al., 2019), autonomous driving (e.g. Kendall et al., 2019; Bellemare et al., 2020), robotics (Haarnoja et al., 2017), medical treatment (e.g. Yu et al., 2019) and recommendation system and advertisement (e.g. Li et al., 2011; Thomas et al., 2017).
Revised: Vinyals et al., 2019), autonomous driving (e.g. Kendall et al., 2019; Bellemare et al., 2020), robotics (Haarnoja et al., 2017), medical treatment (e.g. Yu et al., 2019) and recommendation system and advertisement (e.g. Li et al., 2011; Thomas et al., 2017).

Original: Previous success for sequential decision making often requires two key components: (1) a careful design reward function that can provide the supervision signal during learning and (2) an unlimited number of online interactions with the real-world environment  to query new unseen region.
Revised: Previous success for sequential decision making often requires two key components: (1) a careful design reward function that can provide the supervision signal during learning and (2) an unlimited number of online interactions with the real-world environment (or a carefully designed simulator) to query new unseen region.

Original: However, in many scenarios, both components are not allowed.
Revised: However, in many scenarios, both components are not allowed.

Original: For example, it is hard to deﬁne the reward signal in uncountable many extreme situations in autonomous driving; and it is dangerous and risky to directly deploy a learning policy on human to gather information in autonomous medical treatment.
Revised: For example, it is hard to deﬁne the reward signal in uncountable many extreme situations in autonomous driving; and it is dangerous and risky to directly deploy a learning policy on human to gather information in autonomous medical treatment (Yu et al., 2019).

Original: Therefore an ofﬂine sequential decision making algorithm without reward signal is in demand.
Revised: Therefore an ofﬂine sequential decision making algorithm without reward signal is in demand.

Original: Ofﬂine Imitation Learning (IL) offers an elegant way to train intelligent agents for complex task without the knowledge of reward functions.
Revised: Ofﬂine Imitation Learning (IL) offers an elegant way to train intelligent agents for complex task without the knowledge of reward functions or using a simulator.

Original: Since the ofﬂine imitation learning does not interact with the environment, in order to guide intelligent agents to correct behaviors, it is crucial to have high quality expert demonstrations.
Revised: Since the ofﬂine imitation learning does not interact with the environment, in order to guide intelligent agents to correct behaviors, it is crucial to have high quality expert demonstrations.

Original: The well-known Behavior Cloning (BC) algorithm
Revised: The well-known Behavior Cloning (BC) algorithm

Original: (Pomerleau, 1988) requires that the demonstrations given for training are all presumably optimal and it aims to learn that mapping from state to action via expert demonstration data set.
Revised: (Pomerleau, 1988) requires that the demonstrations given for training are all presumably optimal and it aims to learn that mapping from state to action from expert demonstration data set.

Original: However in real world scenario, since the demonstration is often collected from human, we cannot guarantee that all the demonstrations wehave collected provide successful behaviors.
Revised: However in real world scenario, since the demonstration is often collected from human, we cannot guarantee that all the demonstrations we collected have high quality.

Original: An human expert tends to make mistakes by accident or due to the hardness of a complicated scenario (e.g., medical diagnosis).
Revised: An human expert can make mistakes by accident or due to the hardness of a complicated scenario (e.g., medical diagnosis).

Original: Furthermore, even an expert demonstrates a successful behavior, the recorder or the recording system can have a chance to contaminate the data by accident or on purpose (e.g. Eykholt et al., 2018; Neff & Nagy, 2016). 
Revised: Furthermore, even an expert demonstrates a successful behavior, the recorder or the recording system can have a chance to contaminate the data by accident or on purpose (e.g. Eykholt et al., 2018; Neff & Nagy, 2016). [Figure]

Original: This leads to the central question of the paper:
Revised: This leads to the central question of the paper:

Original: Can the presumably optimal assumption on  demonstrations be weakened or even tolerate arbitrary outliers under ofﬂine imitation learning settings?
Revised: Can the optimality assumption on expert demonstrations be weakened or even tolerate arbitrary outliers under ofﬂine imitation learning settings?

Original: [Figure]

Original: More concretely, we consider corrupted demonstrations setting where the majority of the demonstration data is collected by an expert policy (presumably optimal), and the remaining data can be even arbitrary outliers (the formal deﬁnition is presented in Deﬁnition 2.1).
Revised: More concretely, we consider corrupted demonstrations setting where the majority of the demonstration data is collected by an expert policy (presumably optimal), and the remaining data can be even arbitrary outliers (the formal deﬁnition is presented in Deﬁnition 2.1).

Original: This has great signiﬁcance in many applications, such as automated medical diagnosis for healthcare (Yu et al.
Revised: This has great signiﬁcance in many applications, such as automated medical diagnosis for healthcare (Yu et al.

Original: and autonomous driving (Ma et al., 2018), where the historical data (demonstration) is often complicated and noisy which requires robustness consideration.
Revised: and autonomous driving (Ma et al., 2018), where the historical data (demonstration) is often complicated and noisy which requires robustness consideration.

Original: However, the classical ofﬂine imitation learning approaches such as Behavior Cloning (BC) fails drastically under this corrupted demonstration settings.
Revised: However, the classical ofﬂine imitation learning approaches such as Behavior Cloning (BC) fails drastically under this corrupted demonstration settings.

Original: We illustrated this phenomenon in Figure 1.
Revised: We illustrated this phenomenon in Figure 1.

Original: We use BC on a continuous control environment, and the performance of the policy learned by BC drops drastically as the fraction of corruptions increases in the ofﬂine demonstration data set.
Revised: We use BC on a continuous control environment, and the performance of the policy learned by BC drops drastically as the fraction of corruptions increases in the ofﬂine demonstration data set.

Original: However, our proposed algorithm – Robust Behavior Cloning (Algorithm 1) – is resilient to corruptions in the ofﬂine demonstrations.
Revised: However, our proposed algorithm – Robust Behavior Cloning (Algorithm 1) – is resilient to corruptions in the ofﬂine demonstrations.

Original: The detailed experimental setup is included in Section 5.
Revised: The detailed experimental setup is included in Section 5.

Original: We now summarize our contributions as follows.
Revised: We now summarize our contributions as follows.

Original: 1. M AIN C ONTRIBUTIONS
Revised: M AIN C ONTRIBUTIONS

Original: • (Algorithm)
Revised: • (Algorithm)

Original: We consider robustness in ofﬂine imitation learning where we have corrupted demonstrations.
Revised: We consider robustness in ofﬂine imitation learning where we have corrupted demonstrations.

Original: Our deﬁnition for corrupted demonstrations signiﬁcantly weakens the presumably optimal assumption on demonstration data, and can tolerate a constant fraction of state-action pairs to be arbitrarily corrupted
Revised: Our deﬁnition for corrupted demonstrations signiﬁcantly weakens the presumably optimal assumption on demonstration data, and can tolerate a constant (cid:15) -fraction of state-action pairs to be arbitrarily corrupted.

Original: We refer to Deﬁnition 1 for a more precise statement
Revised: We refer to Deﬁnition 2.1 for a more precise statement.

Original: To deal with this issue, we propose a novel algorithm Robust Behavior Cloning (Algorithm 1) for robust imitation learning.
Revised: To deal with this issue, we propose a novel algorithm Robust Behavior Cloning (Algorithm 1) for robust imitation learning.

Original: Our algorithm works in the ofﬂine setting, without any further interaction with the environment.
Revised: Our algorithm works in the ofﬂine setting, without any further interaction with the environment.

Original: The core ingredient of our robust algorithm is using a novel median of means objective in policy estimation compared to classical Behavior Cloning.
Revised: The core ingredient of our robust algorithm is using a novel median of means objective in policy estimation compared to classical Behavior Cloning.

Original: Hence, it’s simple to implement, and computationally efﬁcient
Revised: Hence, it’s simple to implement, and computationally efﬁcient.

Original: • (Theoretical guarantees)
Revised: • (Theoretical guarantees)

Original: We analyze our Robust Behavior Cloning algorithm when there exists a constant fraction of outliers in the demonstrations under the ofﬂine setting.
Revised: We analyze our Robust Behavior Cloning algorithm when there exists a constant fraction of outliers in the demonstrations under the ofﬂine setting.

Original: We show that our RBC achieves  the same error scaling and sample complexity compared to vanilla BC with expertdemonstrations.
Revised: We show that our RBC achieves nearly the same error scaling and sample complexity compared to vanilla BC with expert

Original:  To this end, our algorithm guarantees robustness to corrupted demonstrations at no cost of statistical error.
Revised: demonstrations. To this end, our algorithm guarantees robustness to corrupted demonstrations at no cost of statistical error.

Original: This is the content of Section 4.
Revised: This is the content of Section 4.

Original: • (Empirical support)
Revised: • (Empirical support)

Original: We validate the predicted robustness and show the effectiveness of our algorithm on several continuous control benchmarks – the vanilla BC is fragile indeed with corrupted demonstrations, and our Robust Behavior Cloning achieves nearly the same performance compared to vanilla BC with expert demonstrations.
Revised: We validate the predicted robustness and show the effectiveness of our algorithm on different high-dimensional continuous control benchmarks – the vanilla BC is fragile indeed with corrupted demonstrations, and our Robust Behavior Cloning achieves nearly the same performance compared to vanilla BC with expert demonstrations.

Original: This is the content of Section 5.
Revised: This is the content of Section 5.

Original:  P ROBLEM S ETUP 2. R EINFORCEMENT L EARNING AND I MITATION L EARNING
Revised:  P ROBLEM S ETUP 2. R EINFORCEMENT L EARNING AND I MITATION L EARNING

Original: Markov Decision Process and Reinforcement Learning.
Revised: Markov Decision Process and Reinforcement Learning.

Original: We start the problem setup by introducing the Markov decision process (MDP).
Revised: We start the problem setup by introducing the Markov decision process (MDP).

Original: An MDP M = (cid:104)S , A , r, P , µ 0 , γ (cid:105) consists of a state space S , an action space A , an unknown reward function r : S ×
Revised: An MDP M = (cid:104)S , A , r, P , µ 0 , γ (cid:105) consists of a state space S , an action space A , an unknown reward function r : S ×

Original: A → [0 , R max ] , an unknown transition kernel P : S ×
Revised: A → [0 , R max ] , an unknown transition kernel P : S ×

Original: A → ∆( S ) , an initial state distribution µ 0 ∈ ∆( S ) , and a discounted factor γ ∈ (0 , 1) .
Revised: A → ∆( S ) , an initial state distribution µ 0 ∈ ∆( S ) , and a discounted factor γ ∈ (0 , 1) .

Original: We use ∆ to denote the probability distributions on the simplex.
Revised: We use ∆ to denote the probability distributions on the simplex.

Original: An agent acts in a MDP following a policy π ( ·| s ) , which prescribes a distribution over the action space
Revised: An agent acts in a MDP following a policy π ( ·| s ) , which prescribes a distribution over the action space

Original: A given each state s ∈ S .
Revised: A given each state s ∈ S .

Original: Running the policy starting from the initial distribution s 1 ∼ µyields a stochastic trajectory T :
Revised: Running the policy starting from the initial distribution s 1 ∼ µyields a stochastic trajectory T :

Original: = { s t , a t , r t } 1 ≤ t ≤∞ , where s t , a t , r t represent the state, action, reward at time t respectively, with a t ∼ π ( ·| s t ) and the next state s t +1 follows the unknown transition kernel s t +1 ∼ P ( ·| s t , a t ) .
Revised: = { s t , a t , r t } 1 ≤ t ≤∞ , where s t , a t , r t represent the state, action, reward at time t respectively, with a t ∼ π ( ·| s t ) and the next state s t +1 follows the unknown transition kernel s t +1 ∼ P ( ·| s t , a t ) .

Original: We denote ρ π,t ∈ ∆( S × A ) as the marginal joint stationary distribution for state, action at time step t , and we deﬁne ρ π = (1 − γ ) (cid:80) ∞
Revised: We denote ρ π,t ∈ ∆( S × A ) as the marginal joint stationary distribution for state, action at time step t , and we deﬁne ρ π = (1 − γ ) (cid:80) ∞

Original: i =1 γ t ρ π,t as visitation distribution for policy π .
Revised: i =1 γ t ρ π,t as visitation distribution for policy π .

Original: For simplicity, we reuse the notation
Revised: For simplicity, we reuse the notation

Original: ρ π ( s ) = (cid:82) a ∈A ρ π ( s, a ) da to denote the marginal distribution over state.
Revised: ρ π ( s ) = (cid:82) a ∈A ρ π ( s, a ) da to denote the marginal distribution over state.

Original: The goal of reinforcement learning is to ﬁnd the best policy π to maximize the expected cumulative return J π = E T ∼ π
Revised: The goal of reinforcement learning is to ﬁnd the best policy π to maximize the expected cumulative return J π = E T ∼ π

Original: [ (cid:80) ∞ i =1 γ t r t ] .
Revised: [ (cid:80) ∞ i =1 γ t r t ] .

Revised: Common RL algorithms (e.g., please refer to Szepesv´ari (2010)) requires online interaction and exploration with the environments.

Revised: However, this is prohibited in the ofﬂine setting.

Original: Imitation Learning.
Revised: Imitation Learning.

Original: Imitation learning (IL) aims to obtain a policy to mimic expert’s behavior with demonstration data set D
Revised: Imitation learning (IL) aims to obtain a policy to mimic expert’s behavior with demonstration data set D

Original: = { ( s i , a i ) } Ni =1 where N is the sample size of D .
Revised: = { ( s i , a i ) } Ni =1 where N is the sample size of D .

Original: Note that we do not need any reward signal.
Revised: Note that we do not need any reward signal.

Original: Tradition imitation learning assumes perfect (or near-optimal) expert demonstration – for simpliﬁcation we assume that each state, action pair ( s i , a i ) is drawn from the joint stationary distribution of an expert policy π E :
Revised: Tradition imitation learning assumes perfect (or near-optimal) expert demonstration – for simpliﬁcation we assume that each state-action pair ( s i , a i ) is drawn from the joint stationary distribution of an expert policy π E :

Original: [Equation]
Revised: [Equation]

Revised: Learning from demonstrations with or without online interactions has a long history (e.g., Pomerleau (1988);

Revised: Ho & Ermon (2016)).

Original: The goal of ofﬂine IL is to learn a policy (cid:98) π IL =
Revised: The goal of ofﬂine IL is to learn a policy (cid:98) π IL =

Original: A ( D ) through an IL algorithm A , given the demonstration data set D , without further interaction with the unknown true transition dynamic P .
Revised: A ( D ) through an IL algorithm A , given the demonstration data set D , without further interaction with the unknown true transition dynamic P .

Original: Behavior Cloning.
Revised: Behavior Cloning.

Original: The Behavior Cloning (BC) is the well known algorithm (Pomerleau, 1988) for IL which only uses ofﬂine demonstration data without any interaction with the environment.
Revised: The Behavior Cloning (BC) is the well known algorithm (Pomerleau, 1988) for IL which only uses ofﬂine demonstration data without any interaction with the environment.

Original: More speciﬁcally, BC solves the following Maximum Likelihood Estimation (MLE) problem, which minimizes the average Negative Log-Likelihood (NLL) for all samples in ofﬂine demonstrations D :
Revised: More speciﬁcally, BC solves the following Maximum Likelihood Estimation (MLE) problem, which minimizes the average Negative Log-Likelihood (NLL) for all samples in ofﬂine demonstrations D :

Original: [Equation]
Revised: [Equation]

Original: Recent work (Rajaraman et al., 2020) has shown that BC is optimal under the ofﬂine setting, and can only be improved with the knowledge of transition dynamic P  .
Revised: Recent works (Agarwal et al., 2019; Rajaraman et al., 2020; Xu et al., 2021) have shown that BC is optimal under the ofﬂine setting, and can only be improved with the knowledge of transition dynamic P in the worst case .

Original: Also, a line of research considers improving BC with further online interaction of the environment (Brantley et al., 2019) or actively querying an expert (Ross et al., 2011).
Revised: Also, another line of research considers improving BC with further online interaction of the environment (Brantley et al., 2019) or actively querying an expert (Ross et al., 2011; Ross & Bagnell, 2014).

Original: 2. L EARNING FROM CORRUPTED DEMONSTRATIONS
Revised: 2. L EARNING FROM CORRUPTED DEMONSTRATIONS

Original: However, it is sometimes unrealistic to assume that the demonstration data set is collected through a presumably optimal expert policy.
Revised: However, it is sometimes unrealistic to assume that the demonstration data set is collected through a presumably optimal expert policy.

Original: In this paper, we propose Deﬁnition 2.1 for the corrupted demonstrations, which tolerates gross corruption or model mismatch in ofﬂine data set.
Revised: In this paper, we propose Deﬁnition 2.1 for the corrupted demonstrations, which tolerates gross corruption or model mismatch in ofﬂine data set.

Original: Deﬁnition 2.1 (Corrupted Demonstrations) .
Revised: Deﬁnition 2.1 (Corrupted Demonstrations) .

Original: Let the state-action pair ( s i , a i ) Ni
Revised: Let the state-action pair ( s i , a i ) Ni

Original: =1 drawn from the joint stationary distribution of an expert policy π E .
Revised: =1 drawn from the joint stationary distribution of a presumably optimal expert policy π E .

Original: The corrupted demonstration data D are generated by the following process: an adversary can choose an arbitrary (cid:15) -fraction  of the samples in [ N ] and modiﬁes them with arbitrary values.
Revised: The corrupted demonstration data D are generated by the following process: an adversary can choose an arbitrary (cid:15) -fraction ( (cid:15) < 0 . 5 ) of the samples in [ N ] and modiﬁes them with arbitrary values.

Revised: We note that (cid:15) is a constant independent of the dimensions of the problem.

Original: After the corruption, we use D to denote the corrupted demonstration data set.
Revised: After the corruption, we use D to denote the corrupted demonstration data set.

Original: This corruption process can represent gross corruptions or model mismatch in the demonstration data set.
Revised: This corruption process can represent gross corruptions or model mismatch in the demonstration data set.

Original: To the best of our knowledge, Deﬁnition 2.1 is the ﬁrst deﬁnition for corrupted demonstrations in imitation learning which tolerates arbitrary corruptions.
Revised: To the best of our knowledge, Deﬁnition 2.1 is the ﬁrst deﬁnition for corrupted demonstrations in imitation learning which tolerates arbitrary corruptions.

Original: In the supervised learning, the wellknown Huber’s contamination model (Huber (1964)) considers ( x , y ) iid ∼ (1 − (cid:15) ) P + (cid:15)Z, where x ∈ R d is the explanatory variable (feature) and y ∈ R is the response variable.
Revised: In the supervised learning, the well-known Huber’s contamination model (Huber (1964)) considers ( x , y ) iid ∼ (1 − (cid:15) ) P + (cid:15)Z, where x ∈ R d is the explanatory variable (feature) and y ∈ R is the response variable.

Original: Here, P denotes the authentic statistical distribution such as Normal mean estimation or linear regression model, and Z denotes the outliers.
Revised: Here, P denotes the authentic statistical distribution such as Normal mean estimation or linear regression model, and Z denotes the outliers.

Original: Dealing with corrupted x and y  has a long history in the robust statistics community (e.g. Rousseeuw, 1984; Chen et al., 2013; 2017; Klivans et al., 2018;
Revised: Dealing with corrupted x and y in high dimensions has a long history in the robust statistics community (e.g. Rousseeuw, 1984; Chen et al., 2013; 2017; Yin et al., 2018).

Original: Yin et al., 2018;
Revised: However, it’s only until recently that robust statistical methods can handle constant (cid:15) -fraction (independent of dimensionality R d ) of outliers in x and y (Klivans et al., 2018;

Original: Prasad et al., 2020; Diakonikolas et al., 2019; Liu et al., 2019; 2020; Shen & Sanghavi, 2019; Lugosi & Mendelson, 2019; Lecu´e & Lerasle, 2020; Jalal et al., 2020).
Revised: Prasad et al., 2020; Diakonikolas et al., 2019; Liu et al., 2019; 2020; Shen & Sanghavi, 2019; Lugosi & Mendelson, 2019; Lecu ´ e & Lerasle, 2020; Jalal et al., 2020).

Original: We note that in Imitation Learning, the data collecting process for the demonstrations does not obey i.i.d.
Revised: We note that in Imitation Learning, the data collecting process for the demonstrations does not obey i.i.d.

Original: assumption in traditional supervised learning due to the temporal dependency.
Revised: assumption in traditional supervised learning due to the temporal dependency.

Original: Notations.
Revised: Notations.

Original: Throughout this paper, we use { c i } i =1 , 2 , 3 to denote the universal positive constant.
Revised: Throughout this paper, we use { c i } i =1 , 2 , 3 to denote the universal positive constant.

Original: We utilize the big- O notation f ( n ) =
Revised: We utilize the big- O notation f ( n ) =

Original: O ( g ( n )) to denote that there exists a positive constant c 1 and a natural number n 0 such that, for all n ≥ n 0 , we have f ( n ) ≤ c 1 g ( n ) .
Revised: O ( g ( n )) to denote that there exists a positive constant c 1 and a natural number n 0 such that, for all n ≥ n 0 , we have f ( n ) ≤ c 1 g ( n ) .

Original:  O UR A LGORITHMS
Revised:  O UR A LGORITHMS

Original: It is well known that the Median-of-Means (MOM) estimator achieves sub-Gaussian concentration bound for one-dimensional mean estimation even though the underlying distribution only has second moment bound (heavy tailed distribution) (interested readers are referred to textbooks such as
Revised: It is well known that the Median-of-Means (MOM) estimator achieves sub-Gaussian concentration bound for one-dimensional mean estimation even though the underlying distribution only has second moment bound (heavy tailed distribution) (interested readers are referred to textbooks such as

Original: Nemirovsky & Yudin (1983); Jerrum et al.
Revised: Nemirovsky & Yudin (1983); Jerrum et al.

Original: ; Alon et al.
Revised: ; Alon et al.

Original: The vanilla MOM estimator for one-dimensional mean estimation works like following: (1) randomly partition N samples into M batches; (2) calculates the mean for each batch; (3) outputs the median of these batch mean.
Revised: The vanilla MOM estimator for one-dimensional mean estimation works like following: (1) randomly partition N samples into M batches; (2) calculates the mean for each batch; (3) outputs the median of these batch mean.

Original: Very recently, MOM estimators are used for high dimensional robust regression (Brownlees et al., 2015; Hsu & Sabato, 2016) by applying MOM estimator on the loss function of empirical risk minimization process.
Revised: Very recently, MOM estimators are used for high dimensional robust regression (Brownlees et al., 2015; Hsu & Sabato, 2016) by applying MOM estimator on the loss function of empirical risk minimization process.

Original: 3. R OBUST B EHAVIOR C LONING
Revised: 3. R OBUST B EHAVIOR C LONING

Original: Motivated by using MOM estimators on the loss function, we propose Deﬁnition 3.1 which uses a
Revised: Motivated by using MOM estimators on the loss function, we propose Deﬁnition 3.1 which uses a

Original: MOM objective to handle arbitrary outliers in demonstration data set ( s , a ) ∈ D .
Revised: MOM objective to handle arbitrary outliers in demonstration data set ( s , a ) ∈ D .

Original: Deﬁnition 3.1 (Robust Behavior Cloning) .
Revised: Deﬁnition 3.1 (Robust Behavior Cloning) .

Original: We split the corrupted demonstrations D into M batches randomly 1 : { B j } Mj =1  .
Revised: We split the corrupted demonstrations D into M batches randomly 1 : { B j } M j =1 , with the batch size b ≤ 1 3 (cid:15) .

Original: The Robust Behavior Cloning solves the following optimization
Revised: The Robust Behavior Cloning solves the following optimization

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Table]

Original: where the loss function (cid:96) j
Revised: where the loss function (cid:96) j

Original: ( π ) is the average Negative Log-Likelihood in the batch B j : 
Revised: ( π ) is the average Negative Log-Likelihood in the batch B j : [Equation]

Original: [Equation]
Revised: [Equation]

Original: The workhorse of Deﬁnition 3.1 is eq.
Revised: The workhorse of Deﬁnition 3.1 is eq.

Original: (3), which uses a novel variant of Median-of-Means (MOM) tournament procedure (Le Cam (2012);
Revised: (3), which uses a novel variant of Median-of-Means (MOM) tournament procedure (Le Cam (2012);

Original: Lugosi & Mendelson (2019); Lecu´e & Lerasle (2020); Jalal et al.
Revised: Lugosi & Mendelson (2019); Lecu ´ e & Lerasle (2020); Jalal et al.

Original: In eq.
Revised: In eq.

Original: (4), we calculate the average Negative Log-Likelihood (NLL) for a single batch, and (cid:98) π RBC is the solution of a min-max formulation based on the batch loss (cid:96)
Revised: (4), we calculate the average Negative Log-Likelihood (NLL) for a single batch, and (cid:98) π RBC is the solution of a min-max formulation based on the batch loss (cid:96)

Original: j ( π ) .
Revised: j ( π ) .

Original: Though our algorithm minimizes the robust version of NLL, we do not utilize the traditional iid assumption in the supervised learning. 
Revised: Though our algorithm minimizes the robust version of NLL, we do not utilize the traditional iid assumption in the supervised learning. To gain some intuition of the formulation eq.

Revised: (3), if we replace the median operator by the mean operator, then RBC is equivalent to BC which just minimizes the empirical average of Negative

Revised: Log-Likelihood.

Revised: This is due to the linearity of using the mean operator.

Revised: However, this is not robust to corrupted demonstration.

Original:  The intuition behind solving this min-max formulation is that the inner variable π (cid:48) needs to get close to π E to maximize the difference of loss function, and the outer variable π also need to get close to π E .
Revised: Hence, we use the median operator on the loss function. The intuition behind solving this min-max formulation is that the inner variable π (cid:48) needs to get close to π E to maximize the difference of loss function, and the outer variable π also need to get close to π E .

Original: Hence we can guarantee that (cid:98) π RBC will be close to π E .
Revised: Hence we can guarantee that (cid:98) π RBC will be close to π E .

Original: In Section 4, we show that under corrupted demonstrations, (cid:98) π RBC in eq.
Revised: In Section 4, we show that under corrupted demonstrations, (cid:98) π RBC in eq.

Original: (3) has the same error scaling and sample complexity compared to π E .
Revised: (3) has the same error scaling and sample complexity compared to π E .

Original: In  Deﬁnition 3.1 the objective function eq.
Revised: In Section 4, we provide rigorous statistical guarantees for Deﬁnition 3.1. However, the objective function eq.

Original: (3)  is not convex (in general), hence we use Algorithmas a computational heuristic to solve it.
Revised: (3) in Deﬁnition 3.1 is not convex (in general), hence we use Algorithm 1 as a computational heuristic to solve it.

Original: In each iteration of Algorithm 1, we randomly partition the demonstration data set D into M batches, and calculate the loss (cid:96) j ( π ) − (cid:96) j ( π (cid:48) ) by eq.
Revised: In each iteration of Algorithm 1, we randomly partition the demonstration data set D into M batches, and calculate the loss (cid:96) j ( π ) − (cid:96) j ( π (cid:48) ) by eq.

Original: We then pick the batch B Med with the median loss, and evaluate the gradient on that batch.
Revised: We then pick the batch B Med with the median loss, and evaluate the gradient on that batch.

Original: We use gradient descent on π for the arg min part and gradient ascent on π (cid:48) for the arg max part.
Revised: We use gradient descent on π for the arg min part and gradient ascent on π (cid:48) for the arg max part.

Original: In Section 5, we empirically show that this gradient-based heuristic Algorithm 1 is able to minimize this objective and has good convergence properties.
Revised: In Section 5, we empirically show that this gradientbased heuristic Algorithm 1 is able to minimize this objective and has good convergence properties.

Original: As for the time complexity, when using back-propagation on one batch of samples, our RBC incurs overhead costs compared to vanilla BC, in order to evaluate the loss function for all samples via forward propagation.
Revised: As for the time complexity, when using back-propagation on one batch of samples, our RBC incurs overhead costs compared to vanilla BC, in order to evaluate the loss function for all samples via forward propagation.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we provide theoretical guarantees for our RBC algorithm.
Revised: In this section, we provide theoretical guarantees for our RBC algorithm.

Original: Since our method (Deﬁnition 3.1) directly estimates the conditional probability π ( a | s ) over the ofﬂine demonstrations, our theoretical analysis provides guarantees on E s ∼ ρ π E (cid:13)(cid:13)(cid:98) π RBC ( ·| s ) − π E ( ·| s ) (cid:13)(cid:13) 2 TV , which upper bounds the total variation norm compared to π E under the expectation of s ∼ ρ π E .
Revised: Since our method (Deﬁnition 3.1) directly estimates the conditional probability π ( a | s ) over the ofﬂine demonstrations, our theoretical analysis provides guarantees on E s ∼ ρ π E (cid:13)(cid:13)(cid:98) π RBC ( ·| s ) − π E ( ·| s ) (cid:13)(cid:13) 2 TV , which upper bounds the total variation norm compared to π E under the expectation of s ∼ ρ π E .

Original: The ultimate goal of the learned policy is to maximize the expected cumulative return, thus we then provide an upper bound for the sub -optimality
Revised: The ultimate goal of the learned policy is to maximize the expected cumulative return, thus we then provide an upper bound for the sub-optimality J π E − J (cid:98) π RBC .

Original: [Equation] We begin the theoretical analysis by Assumption 4.1, which simpliﬁes our analysis and is common in literature (Agarwal et al., 2019; 2020).
Revised: We begin the theoretical analysis by Assumption 4.1, which simpliﬁes our analysis and is common in literature (Agarwal et al., 2019; 2020).

Original: By assuming that the policy class Π is discrete, our upper bounds depend on the quantity log( | Π | ) /N , which matches the error rates and sample complexity for using BC with expert demonstrations (Agarwal et al., 2019; 2020).
Revised: By assuming that the policy class Π is discrete, our upper bounds depend on the quantity log( | Π | ) /N , which matches the error rates and sample complexity for using BC with expert demonstrations (Agarwal et al., 2019; 2020).

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: We assume that the policy class Π is discrete, and realizable, i.e., π E ∈ Π .
Revised: We assume that the policy class Π is discrete, and realizable, i.e., π E ∈ Π .

Original: We ﬁrst present Theorem 4.1, which shows that minimizing the MOM objective via eq.
Revised: We ﬁrst present Theorem 4.1, which shows that minimizing the MOM objective via eq.

Original: (3) guarantees the closeness of robust policy to optimal policy in total variation distance.
Revised: (3) guarantees the closeness of robust policy to optimal policy in total variation distance.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Suppose we have corrupted demonstration data set D with sample size N from Definition 2.1, and there exists a(sufﬁciently small) constant corruption ratio (cid:15)  .
Revised: Suppose we have corrupted demonstration data set D with sample size N from Deﬁnition 2.1, and there exists a constant corruption ratio (cid:15) < 0 . 5 .

Original: Under Assumption 4.1, let τ to be the output objective value with (cid:98) π RBC in the optimization eq.
Revised: Under Assumption 4.1, let τ to be the output objective value with (cid:98) π RBC in the optimization eq.

Original: (3)  , then with probability at least 1 − c 1 δ , we have
Revised: (3) with the batch size b ≤ 13 (cid:15) , then with probability at least 1 − c 1 δ , we have

Original: [Equation]
Revised: [Equation]

Original: The proof is collected in Appendix A.
Revised: The proof is collected in Appendix A.

Original: We note that the data collection process does not follow the iid assumption, hence we use martingale analysis similar to (Agarwal et al., 2019; 2020). 
Revised: We note that the data collection process does not follow the iid assumption, hence we use martingale analysis similar to (Agarwal et al., 2019; 2020). The ﬁrst part of eq. (5) is the statistical error log( | Π | /δ ) N .

Original: The  ﬁnal objective value in the optimization eq.
Revised: The second part is the ﬁnal objective value in the optimization eq.

Original: (3) τ  includes two parts – the ﬁrst part  is inherent stochastic distribution of π E and the fraction of corruption  (cid:15) , which cannot be improved.
Revised: (3) τ which includes two parts – the ﬁrst part scales with O ( 1 b ) , which is equivalent to the fraction of corruption O ( (cid:15) ) .

Original: The second part is the sub-optimality gap due to the solving the non-convex optimization.
Revised: The second part is the sub-optimality gap due to the solving the non-convex optimization.

Original: Our main theorem  Theorem 4. guarantees that a small value of the ﬁnal objective implies an accurate estimation of policy and hence we can certify estimation quality using the obtained ﬁnal value of the objective.
Revised: Our main theorem – Theorem 4.1 – guarantees that a small value of the ﬁnal objective implies an accurate estimation of policy and hence we can certify estimation quality using the obtained ﬁnal value of the objective.

Original: Next, we present Theorem 4.2, which guarantees the reward performance of the learned robust policy (cid:98) π RBC .
Revised: Next, we present Theorem 4.2, which guarantees the reward performance of the learned robust policy (cid:98) π RBC .

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Under the same setting as Theorem 4.1, we have
Revised: Under the same setting as Theorem 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − c 1 δ .
Revised: with probability at least 1 − c 1 δ .

Original: The proof is also collected in Appendix A.
Revised: The proof is also collected in Appendix A.

Original: The error scaling and sample complexity ofTheorem 4.
Revised: We note that the error scaling and sample complexity of

Original:  and Theorem 4.2 matches the vanilla BC with expert demonstrations (Agarwal et al., 2019; 2020).
Revised: Theorem 4.1 and Theorem 4.2 matches the vanilla BC with expert demonstrations (Agarwal et al., 2019; 2020).

Original: Remark 4.1.
Revised: Remark 4.1.

Original: The quadratic dependency on the effective horizon – 1 (1 − γ ) 2 – is  known as the compounding error or distribution shift in literature, which is due to the essential limitation of ofﬂine imitation learning setting.
Revised: The quadratic dependency on the effective horizon ( 1 (1 − γ ) 2 in the discounted setting or H 2 in the episodic setting) is widely known as the compounding error or distribution shift in literature, which is due to the essential limitation of ofﬂine imitation learning setting.

Original: Recent work (Rajaraman et al., 2020) shows that this quadratic dependency cannot be improved without any further interaction with the environment or the knowledge of transition dynamic P .
Revised: Recent work (Rajaraman et al., 2020; Xu et al., 2021) shows that this quadratic dependency cannot be improved without any further interaction with the environment or the knowledge of transition dynamic P .

Original: Hence BC is actually optimal under no-interaction setting.
Revised: Hence BC is actually optimal under no-interaction setting.

Original: Also, a line of research considers improving BC by further online interaction with the environment or even active query of the experts (Ross et al., 2011; Brantley et al., 2019).
Revised: Also, a line of research considers improving BC by further online interaction with the environment or even active query of the experts (Ross et al., 2011; Brantley et al., 2019; Ross & Bagnell, 2014).

Original: Since our work, as a robust counterpart of BC, focuses on the robustness to the corruptions in the ofﬂine demonstrations, it can be naturally used in online algorithms such as DAGGER (Ross et al., 2011) and Brantley et al.
Revised: Since our work, as a robust counterpart of BC, focuses on the robustness to the corruptions in the ofﬂine demonstrations setting, it can be naturally used in online the online setting such as DAGGER (Ross et al., 2011) and Brantley et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we study the empirical performance of our Robust Behavior Cloning.
Revised: In this section, we study the empirical performance of our Robust Behavior Cloning.

Original: We evaluate the robustness of Robust Behavior Cloning on several continuous control benchmarks simulated by PyBullet Coumans & Bai (2016) simulator: LunarLanderContinuous-v2 and HalfCheetahBulletEnvv0.
Revised: We evaluate the robustness of Robust Behavior Cloning on several continuous control benchmarks simulated by PyBullet Coumans & Bai (2016) simulator: HopperBulletEnv-v0, Walker2DBulletEnv-v0, HalfCheetahBulletEnv-v0 and AntBulletEnv-v0.

Original: Actually, these tasks have true reward function already in the simulator.
Revised: Actually, these tasks have true reward function already in the simulator.

Original: We will use only state observation and action for the imitation algorithm, and we then use the reward to evaluate the obtained policy when running in the simulator.
Revised: We will use only state observation and action for the imitation algorithm, and we then use the reward to evaluate the obtained policy when running in the simulator.

Original: For each task, we collect the presumably optimal expert trajectories using pre-trained agents from
Revised: For each task, we collect the presumably optimal expert trajectories using pre-trained agents from

Original: Standard Baselines3 2 .
Revised: Standard Baselines3 2 .

Original: In the experiment, we use TQC (Kuznetsov et al.
Revised: In the experiment, we use Soft Actor-Critic (Haarnoja et al.

Original:  in theStandard
Revised: (2018)) in the

Original:  Baselines3 pre-trained agents,which has the highest reward, and we consider it to be an expert.
Revised: Standard Baselines3 pre-trained agents, and we consider it to be an expert.

Original: For the continuous control environments, the action space are bounded.
Revised: For the continuous control environments, the action space are bounded.

Original: Hence we generate corrupted demonstration data set D as follows: we ﬁrst randomly choose (cid:15) fraction of samples, and corrupt the action to the boundary (normally − 1 or +1 ).
Revised: Hence we generate corrupted demonstration data set D as follows: we ﬁrst randomly choose (cid:15) fraction of samples, and corrupt the action to the boundary (normally − 1 or +1 ).

Revised: We note that Deﬁnition 2.1 allows for arbitrary corruptions, and we choose these outliers’ action since it has the maximum effect, and cannot be easily detected.

Original: We compare our RBC algorithm (Algorithm 1) to a number of natural baselines: the ﬁrst baseline is directly using BC on the corrupted demonstration D without any robustness consideration.
Revised: We compare our RBC algorithm (Algorithm 1) to a number of natural baselines: the ﬁrst baseline is directly using BC on the corrupted demonstration D without any robustness consideration.

Original: The second one is using BC on the expert demonstrations with the same sample size.
Revised: The second one is using BC on the expert demonstrations with the same sample size.

Original: In different settings, we ﬁx the policy network as 2 hidden layer feed-forward Neural Network of size { 400 , 300 } with ReLU activation, which is standard in the baselines.
Revised: In different settings, we ﬁx the policy network as 2 hidden layer feed-forward Neural Network of size { 500 , 500 } with ReLU activation, which is standard in the baselines.

Original: 5. C ONVERGENCE OF OUR A LGORITHM
Revised: 5. C ONVERGENCE OF OUR ALGORITHM

Original: In this subsection, we illustrate the convergence  of our algorithm to support our theoretical analysis.
Revised: We illustrate the convergence and the performance of our algorithm to support our theoretical analysis.

Original: We track the performance metric of different algorithms vs. epoch number in the whole training process.
Revised: We track the performance metric of different algorithms vs. epoch number in the whole training process.

Original: More speciﬁcally, weuse the metrics as follows: (1) we track the Mean Squared Error (cid:107) a − (cid:98) a (cid:107) 22 in the validation data set, which is additional expert demonstrations not revealed to the training process; (2) We then evaluate current policy in the simulator for 20 trials, and obtain the mean and standard deviation of cumulative reward for each epoch.
Revised: More speciﬁcally, we then evaluate current policy in the simulator for 20 trials, and obtain the mean and standard deviation of cumulative reward for each epoch.

Original: These metrics correspond to theoretical bounds in Theorem 4.1 and Theorem 4.2 respectively.
Revised: This metric corresponds to theoretical bounds in Theorem 4.2.

Original: Weﬁrst focus on the continuous control environment LunarLanderContinuous-v2, where the state space has dimension 8, and the action space hasdimension 2 with boundary [ − 1 , 1] .We ﬁx the sample size as 4000, and the corruption fraction (cid:15) = 10% . In Figure 2a, the validation error of vanilla BC on corrupted demonstrations D barely converge. Instead, the validation error of our RBC has good convergence properties compared to the “oracle ver-
Revised: We focus on four continuous control environments, where the observation space has dimensions around 30, and the action space has boundary [ − 1 , 1] .

Original: [Figure] We ﬁx the  corruption ration (cid:15) = 10% , and run algorithms with 20 epochs.
Revised: We ﬁx the sample size as 60000, and vary the corruption fraction (cid:15) to be 10% , 20% .

Original: Vanilla BC on corrupted demonstrations fails to converge to expert policy.

Original: Using the robust counterpart Algorithm 1 on corrupted demonstrations has good convergence properties.

Original: Surprisingly, our RBC on corrupted data set has nearly the same validation error in Figure 2a and reward performance in Figure 2b compared to using BC on expert data set.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: with 150 epochs.

Original: Directly running BC on corrupted demonstration data set fails drastically with no improvement over a random policy.

Original: Using the robust counterpart Algorithm 1 on corrupted demonstrations has good convergence properties.

Original: Surprisingly, in Figure 3b, our RBC on corrupted data set has nearly the same performance of using

Original: BC on expert data set.

Original: sion” which uses BC on expert demonstrations. This validates our theoretical results Theorem 4.1, which guarantees the closeness of robust policy to the expert policy π E . In Figure 2b, we track the cumulative reward for the current policy after each epoch. The BC on corrupted demonstrations cannot converge to the expert policy, yet surprisingly, our robust method has nearly the same performance compared to the BC on expert demonstrations .
Revised: Figure 2 validates our theory that our Robust Behavior

Original: Then, we move to another continuous control task HalfCheetahBulletEnv-v0, which has signiﬁcantly higher dimension, where the observation space has dimension 27 and the action space has dimension 8 with boundary [ − 1 , 1] on each dimension.

Original: The results for Half Cheetah is presented in Figure 3, and we observe similar results as in Figure 2. The vanilla BC on corrupted demonstrations is fragile indeed, and fails drastically. Instead, our robust algorithm achieves nearly  the same results as the BC on expert demonstrations.
Revised: Cloning nearly matches the performance of BC on expert demonstrations for different environments and corruption ratio.

Original: [Figure]

Original: 5. P ERFORMANCE UNDER DIFFERENT SETUPS
Revised: 5. P ERFORMANCE UNDER DIFFERENT SETUPS

Original: In this subsection, we present the effectiveness and predicted robustness of our Robust Behavior

Original: Cloning by evaluating on multiple runs under different setups.

Original: We keep using the continuous control environment LunarLanderContinuous-v2, and focus on different settings for the demonstrations D : the sample size N and the fraction of corruptions (cid:15) .

Original: Different sample size in D .

Original: In this experiment, we ﬁx the fraction of the corruption (cid:15) =

Original: 10% , and vary the sample size of the demonstration data set.

Original: It is expected in Theorem 4.2 that larger sample size of D leads to smaller suboptimality gap in value function.

Original: Figure 4 validates our theory:

Original: Our

Original: RBC on corrupted demonstrations has nearly the same reward as BC on expert demonstrations, and the sub-optimality gap gets smaller as N grows larger.

Original: However, directly using BC on corrupted demonstrations cannot improve as sample size grows larger.

Original: Different fraction of corruption.

Original: This is the experiment we have shown in Section 1.
Revised: This is the experiment we have shown in Section 1.

Original: In this experiment, we ﬁx the sample size N = 4000 as usual, and then vary the fraction of corruptions (cid:15) .
Revised: In the Lunar Lander control environment, we ﬁx the sample size N = 4000 , and then vary the fraction of corruptions (cid:15) .

Original: As expected, Figure 1 shows that our RBC is resilient to a constant-fraction of outliers in the demonstrations ranging from 0 to 20%, and it achieves nearly the same performance as the BC on expert demonstrations.
Revised: As expected, Figure 1 shows that our RBC is resilient to a constant-fraction of outliers in the demonstrations ranging from 0 to 30%, and it achieves nearly the same performance as the BC on expert demonstrations.

Original: In contrast, directly using BC on corrupted demonstrations obtains worse reward performance as the fraction of outliers grows.
Revised: In contrast, directly using BC on corrupted demonstrations obtains worse reward performance as the fraction of outliers grows.

Original: In all our experiments, BC on expert demonstrations is a strong baseline which achieves reward performance at expert level with very few transition samples in demonstrations.

Original: This suggests that compounding error or distribution shift may be less of a problem in these environments.

Original: This is consistent with the ﬁndings in Brantley et al.

Original: Since our RBC is a robust counterpart of

Original: BC, which is resilient to outliers in ofﬂine data , it is straightforward to combine RBC with online

Original: Imitation Learning algorithms (such as DAGGER (Ross et al., 2011), and ensemble based method (Brantley et al., 2019), etc.).

Original: It would be interesting to examine the performance of our algorithm in the online interaction setting, and we leave it as future works.

Original:  D ISCUSSIONS 6. R ELATED W ORK
Revised:  D ISCUSSIONS 6. R ELATED W ORK

Original: Imitation Learning.
Revised: Imitation Learning.

Original: Behavior Cloning (BC) is the most widely-used imitation learning algorithm
Revised: Behavior Cloning (BC) is the most widely-used imitation learning algorithm

Original: (Pomerleau, 1988; Osa et al., 2018) due to its simplicity, effectiveness and scalability, and has been widely used in practice.
Revised: (Pomerleau, 1988; Osa et al., 2018) due to its simplicity, effectiveness and scalability, and has been widely used in practice.

Original: From a theoretical viewpoint, it has been showed that BC achieves informational optimality in the ofﬂine setting (Rajaraman et al., 2020) with no further online interactions  .
Revised: From a theoretical viewpoint, it has been showed that BC achieves informational optimality in the ofﬂine setting (Rajaraman et al., 2020) with no further online interactions or the knowledge of the transition dynamic P .

Original: With online interaction, there’s a line of research focusing on improving BC  – for example, Ross et al.
Revised: With online interaction, there’s a line of research focusing on improving BC in different scenarios – for example, Ross et al.

Original: (2011) proposed DAGGER (Data Aggregation) by querying the expert policy in the online setting.
Revised: (2011) proposed DAgger (Data Aggregation) by querying the expert policy in the online setting.

Original: Brantley et al.
Revised: Brantley et al.

Original: (2019) proposed using an ensemble of BC as uncertainty measure and interacts with the environment to improve BC by taking the uncertainty into account, without the need to query the expert.
Revised: (2019) proposed using an ensemble of BC as uncertainty measure and interacts with the environment to improve BC by taking the uncertainty into account, without the need to query the expert.

Revised: Very recently, (Xu et al., 2021; Rajaraman et al., 2021) leveraged the knowledge of the transition dynamic P to eliminate compounding error/distribution shift issue in

Revised: BC.

Original: Besides BC, there are other imitation learning algorithms: Ho & Ermon (2016) uses generative adversarial networks for distribution matching to learn a reward function; Reddy et al.
Revised: Besides BC, there are other imitation learning algorithms: Ho & Ermon (2016) used generative adversarial networks for distribution matching to learn a reward function; Reddy et al.

Original: (2019) provides a reinforcement learning framework to deal with imitation learning by artiﬁcially setting the reward; Ghasemipour et al.
Revised: (2019) provided a reinforcement learning framework to deal with imitation learning by artiﬁcially setting the reward; Ghasemipour et al.

Original: (2020) uniﬁed several existing imitation learning algorithm as minimizing distribution divergence between learned policy and expert demonstration, just to name a few.
Revised: (2020) uniﬁed several existing imitation learning algorithm as minimizing distribution divergence between learned policy and expert demonstration, just to name a few.

Original: Ofﬂine RL.
Revised: Ofﬂine RL.

Original: Reinforcement learning leverages the signal from reward function to train the policy.
Revised: Reinforcement learning leverages the signal from reward function to train the policy.

Original: Different from IL, ofﬂine RL often does not require the demonstration to be expert demonstration (e.g. Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2020) (interested readers are referred to (Levine et al., 2020)), and even expects the ofﬂine data with higher coverage for different suboptimal policies (Buckman et al., 2020;
Revised: Different from IL, ofﬂine RL often does not require the demonstration to be expert demonstration (e.g. Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2020) (interested readers are referred to (Levine et al., 2020)), and even expects the ofﬂine data with higher coverage for different suboptimal policies (Buckman et al., 2020;

Original: Jin et al., 2021; Rashidinejad et al., 2021).
Revised: Jin et al., 2021; Rashidinejad et al., 2021).

Original: Behavior-agnostic setting (Nachum et al., 2019; Mousavi et al., 2020) even does not require the collected data from a single policy.
Revised: Behavior-agnostic setting (Nachum et al., 2019; Mousavi et al., 2020) even does not require the collected data from a single policy.

Original: The closest relation between ofﬂine RL and IL is the learning of stationary visitation distribution, where learning such visitation distribution does not involve with reward signal, similar to IL.
Revised: The closest relation between ofﬂine RL and IL is the learning of stationary visitation distribution, where learning such visitation distribution does not involve with reward signal, similar to IL.

Original: A line of recent research especially for off-policy evaluation tries to learn the stationary visitation distribution of a given target policy (e.g. Liu et al., 2018; Nachum et al., 2019; Tang et al., 2020;
Revised: A line of recent research especially for off-policy evaluation tries to learn the stationary visitation distribution of a given target policy (e.g. Liu et al., 2018; Nachum et al., 2019; Tang et al., 2020;

Original: Mousavi et al., 2020; Dai et al., 2020).
Revised: Mousavi et al., 2020; Dai et al., 2020).

Original: Especially Kostrikov et al.
Revised: Especially Kostrikov et al.

Original: (2020) leverages the off-policy evaluation idea to IL area.
Revised: (2020) leverages the off-policy evaluation idea to IL area.

Original: Robustness in IL and RL.
Revised: Robustness in IL and RL.

Original: There are several recent papers consider corruption-robust in either RL or IL.
Revised: There are several recent papers consider corruption-robust in either RL or IL.

Original: In RL, Zhang et al.
Revised: In RL, Zhang et al.

Original: (2021b) considers that the adversarial corruption may corrupt the whole episode in the online RL setting while a more recent one (Zhang et al., 2021a) considers ofﬂine RL where (cid:15) -fraction of the whole data set can be replaced by the outliers. 
Revised: (2021b) considers that the adversarial corruption may corrupt the whole episode in the online RL setting while a more recent one (Zhang et al., 2021a) considers ofﬂine RL where (cid:15) -fraction of the whole data set can be replaced by the outliers. However, the (cid:15) dependency scales with the dimension in Zhang et al. (2021a), yet (cid:15) can be a constant in this paper for robust ofﬂine IL.

Original: Many other papers consider perturbations, heavy tails, or corruptions in either reward function (Bubeck et al., 2013) or in transition dynamic
Revised: Many other papers consider perturbations, heavy tails, or corruptions in either reward function (Bubeck et al., 2013) or in transition dynamic

Original: (Xu & Mannor, 2012; Tamar et al., 2014; Roy et al., 2017).
Revised: (Xu & Mannor, 2012; Tamar et al., 2014; Roy et al., 2017).

Original: The most related papers follow a similar setting of robust IL are (Wu et al., 2019; Tangkaratt et al., 2020; 2021), where they consider imperfect or noisy observations in imitation learning.
Revised: The most related papers follow a similar setting of robust IL are (Wu et al., 2019; Tangkaratt et al., 2020; 2021; Brown et al., 2019; Sasaki & Yamashina, 2020), where they consider imperfect or noisy observations in imitation learning.

Original: However, their algorithmsrequire online interactions with the environment, and cannot handle outliers in the demonstrations.
Revised: However, their algorithms cannot handle outliers in the demonstrations, and (Wu et al., 2019; Tangkaratt et al., 2020; 2021) require additional online interactions with the environment.

Original: Our algorithm achieves robustness guarantee from purely ofﬂine demonstration, without the potentially costly or risky interaction with the real world environment.
Revised: Our algorithm achieves robustness guarantee from purely ofﬂine demonstration, without the potentially costly or risky interaction with the real world environment.

Original: 6. S UMMARY AND F UTURE W ORKS
Revised: 6. S UMMARY AND F UTURE W ORKS

Original: In this paper, we considered the corrupted demonstrations issues in imitation learning, and proposed a novel robust algorithm, Robust Behavior Cloning, to deal with the corruptions in ofﬂine demonstration data set.
Revised: In this paper, we considered the corrupted demonstrations issues in imitation learning, and proposed a novel robust algorithm, Robust Behavior Cloning, to deal with the corruptions in ofﬂine demonstration data set.

Original: The core technique is replacing the vanilla Maximum Likelihood Estimation with a Median-of-Means (MOM) objective which guarantees the policy estimation and reward performance in the presence of constant fraction of outliers.
Revised: The core technique is replacing the vanilla Maximum Likelihood Estimation with a Median-of-Means (MOM) objective which guarantees the policy estimation and reward performance in the presence of constant fraction of outliers.

Original: Our algorithm has strong robustness guarantees and works well in practice.
Revised: Our algorithm has strong robustness guarantees and works well in practice.

Original: There are several avenues for future work: since our work focuses on the corruption in ofﬂine data set, any improvement in online imitation learning which utilizes Behavior Cloning would beneﬁt from the corruption-robustness guarantees by our ofﬂine Robust Behavior Cloning.
Revised: There are several avenues for future work: since our work focuses on the corruption in ofﬂine data set, any improvement in online imitation learning which utilizes Behavior Cloning would beneﬁt from the corruption-robustness guarantees by our ofﬂine Robust Behavior Cloning.

Original: Also, it would also be of interest to apply our algorithm for real-world environment, such as automated medical diagnosis and autonomous driving. 
Revised: Also, it would also be of interest to apply our algorithm for real-world environment, such as automated medical diagnosis and autonomous driving. [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: epochs, we evaluate the current policy in the environment for 20 trials, and the shaded region represents the standard deviation.

Revised: Vanilla BC on corrupted demonstrations fails to converge to expert policy.

Revised: Using the robust counterpart Algorithm 1 on corrupted demonstrations has good convergence properties.

Revised: Surprisingly, our RBC on corrupted demonstrations has nearly the same reward performance of using

Revised: BC on expert demonstrations.

Revised: A P ROOFS

Revised: The analysis of maximum likelihood estimation is standard in i.i.d.

Revised: setting for the supervised learning setting (van de Geer, 2000).

Revised: In our proofs of the robust ofﬂine imitation learning algorithm, the analysis for the sequential decision making leverages the martingale analysis technique from (Zhang, 2006; Agarwal et al., 2020).

Revised: Our Robust Behavior Cloning (Deﬁnition 3.1) solves the following optimization

Revised: [Equation]

Revised: where the loss function (cid:96) j

Revised: ( π ) is the average Negative Log-Likelihood in the batch B j :

Revised: [Equation]

Revised: This can be understood as a robust counterpart for the maximum likelihood estimation in sequential decision process.

Revised: With a slight abuse of notation, we use x i and y i to denote the observation and action, and the underlying unknown expert distribution is y

Revised: i ∼ p ( ·| x i ) and p ( y | x ) =

Revised: f ∗ ( x, y ) .

Revised: Following Assumption 4.1, we have the realizable f ∗ ∈ F , and the discrete function class satisﬁes |F| < ∞ .

Revised: Let D denote the data set and let D (cid:48) denote a tangent sequence { x (cid:48)

Revised: i , y (cid:48) i } |D| i =1 .

Revised: The tangent sequence is deﬁned as x (cid:48)

Revised: i ∼ D

Revised: i ( x 1: i − 1 , y 1: i − 1 ) and y (cid:48)

Revised: i ∼ p ( ·| x (cid:48) i ) .

Revised: Note here that x (cid:48)

Revised: i follows from the distribution

Revised: D i , and depends on the original sequence, hence the tangent sequence is independent conditional on

Revised: D .

Revised: For this martingale process, we ﬁrst introduce a decoupling Lemma from Agarwal et al.

Revised: Lemma A.1.

Revised: [Lemma 24 in Agarwal et al.

Revised: Let D be a dataset, and let D (cid:48) be a tangent sequence.

Revised: Let Γ( f, D ) =

Revised: (cid:80) ( x,y )

Revised: ∈D φ ( f, ( x, y )) be any function which can be decomposed additively across samples in D .

Revised: Here, φ is any function of f and sample ( x, y ) .

Revised: Let (cid:98) f

Revised: = (cid:98) f

Revised: ( D ) be any estimator taking the dataset D as input and with range F .

Revised: Then we have

Revised: [Equation]

Revised: Then we present a Lemma which upper bounds the TV distance via a loss function closely related to KL divergence.

Revised: Such bounds for probabilistic distributions are discussed extensively in literature such as Tsybakov (2009).

Revised: Lemma A.2.

Revised: [Lemma 25 in Agarwal et al.

Revised: For any two conditional probability densities f 1 , f 2 and any state distribution D ∈ ∆( X ) we have

Revised: [Equation]

Revised: A. P ROOF OF T HEOREM 4.

Revised: With these Lemmas in hand, we are now equipped to prove our main theorem (Theorem 4.1), which guarantees the solution (cid:98) π RBC of eq.

Revised: (3) is close to the optimal policy π E in TV distance.

Revised: Theorem A.1 (Theorem 4.1) .

Revised: Suppose we have corrupted demonstration data set D with sample size N from Deﬁnition 2.1, and there exists a constant corruption ratio (cid:15) < 0 .

Revised: Under Assump- tion 4.1, let τ to be the output objective value with (cid:98) π RBC in the optimization eq.

Revised: (3) with the batch size b ≤ 13 (cid:15) , then with probability at least 1 − c 1 δ , we have

Revised: [Equation]

Revised: Proof of Theorem 4.1.

Revised: En route to the proof of Theorem 4.1, we keep using the notations in

Revised: Lemma A.1 and Lemma A.2, where the state observation is x , the action is y , and the discrete function class is F .

Revised: Similar to Agarwal et al.

Revised: (2020), we ﬁrst note that Lemma A.1 can be combined with a simple

Revised: Chernoff bound to obtain an exponential tail bound.

Revised: With probability at least 1 − c 1 δ , we have

Revised: [Equation]

Revised: Our proof technique relies on lower bounding the LHS of eq.

Revised: (9), and upper bounding the RHS eq.

Revised: Let the batch size b ≤ 13 (cid:15) , which is a constant in Deﬁnition 3.1, then the number of batches M ≥ 3 (cid:15)N such that there exists at least 66% batches without corruptions.

Revised: In the deﬁnition of RBC (Deﬁnition 3.1), we solve

Revised: [Equation]

Revised: [Equation]

Revised: Notice that since π E is one feasible solution of the inner maximization step eq.

Revised: (10), we can choose π (cid:48)

Revised: =

Revised: π E .

Revised: Now we consider the objective function which is the difference of Negative

Revised: Log-Likelihood between f and f ∗ , i.e., (cid:96) j

Revised: ( f ) − (cid:96) j

Revised: ( f ∗ ) , deﬁned in eq.

Revised: (4) where

Revised: [Equation]

Revised: Hence, we choose Γ( f, D ) in Lemma A.1 as

Revised: [Equation]

Revised: which is the difference of Negative Log-Likelihood N ( (cid:96) j ( f ∗ ) − (cid:96) j ( f ))

Revised: / 2 evaluated on a single batch

Revised: B j , j ∈ [ M ] .

Revised: This is actually the objective function on a single batch appeared in eq.

Revised: Lower bound for the LHS of eq.

Revised: We apply the concentration bound eq.

Revised: (9) for such uncorrupted batches, hence the majority of all batches satisﬁes eq.

Revised: For those batches, the LHS of eq.

Revised: (9) can be lower bounded by the TV distance according to Lemma A.2.

Revised: [Equation]

Revised: where (i) follows from the independence between (cid:98) f and D (cid:48) due to the decoupling technique, and (ii) follows from Lemma A.2, which is an upper bound of the Total Variation distance.

Revised: Upper bound for the RHS of eq.

Revised: Note that the objective is the median of means of each batches and f ∗ is one feasible solution of the inner maximization step eq.

Revised: Since τ is the output objective value with (cid:98) π RBC in the optimization eq.

Revised: (3), this implies that (cid:96) Med ( π ) − (cid:96)

Revised: Med ( π (cid:48) ) ≤ τ for the median batch B Med , which is equivalent to − Γ Med ( f, D ) ≤ Nτ/ 2 .

Revised: Hence for the median batch B Med , the RHS of eq.

Revised: (9) can be upper bounded by

Revised: [Equation]

Revised: Putting together the pieces eq.

Revised: (11) and eq.

Revised: (12) for B Med , we have

Revised: [Equation]

Revised: with probability at least 1 − c 1 δ .

Revised: A. P ROOF OF T HEOREM 4.

Revised: With the supervised learning guarantees Theorem 4.1 in hand, which provides an upper bound for

Revised: E s ∼ ρ π E (cid:13)(cid:13)(cid:98) π RBC

Revised: [Equation]

Revised: ·| s ) − π E ( ·| s ) (cid:13)(cid:13) 2 TV , we are now able to present the suboptimality guarantee of the reward for (cid:98) π RBC .

Revised: This bound directly corresponds to the reward performance of a policy.

Revised: Theorem A.2 (Theorem 4.2) .

Revised: Under the same setting as Theorem 4.1, we have

Revised: [Equation]

Revised: with probability at least 1 − c 1 δ .

Revised: Proof of Theorem 4.2.

Revised: This part is similar to Agarwal et al.

Revised: and we have

Revised: [Equation]

Revised: where we use the fact that sup s,a,π | A π ( s , a ) | ≤ 11 − γ for the advantage function and the reward is always bounded between 0 and 1.

Revised: Combining Theorem 4.1, we have

Revised: [Equation]

Revised: with probability at least 1 − c 1 δ .