Original: D ISAGREEMENT -R EGULARIZED I MITATION L EARNING
Revised: D ISAGREEMENT -R EGULARIZED I MITATION L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning.
Revised: We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning.

Original: It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost.
Revised: It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost.

Original: Unlike adversarial imitation methods, it uses a ﬁxed reward function which is easy to optimize.
Revised: Unlike adversarial imitation methods, it uses a ﬁxed reward function which is easy to optimize.

Original: We prove a regret bound for the algorithmin the tabular setting which is linear in the time horizon multiplied by a coefﬁcient which we show to be low for certain problems in which behavioral cloning fails.
Revised: We prove a regret bound for the algorithm which is linear in the time horizon multiplied by a coefﬁcient which we show to be low for certain problems on which behavioral cloning fails.

Original: We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or signiﬁcantly outperforms behavioral cloning and generative adversarial imitation learning.
Revised: We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks and show that it matches or signiﬁcantly outperforms behavioral cloning and generative adversarial imitation learning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training artiﬁcial agents to perform complex tasks is essential for many applications in robotics, video games and dialogue.
Revised: Training artiﬁcial agents to perform complex tasks is essential for many applications in robotics, video games and dialogue.

Original: If success on the task can be accurately described using a reward or cost function, reinforcement learning (RL) methods offer an approach to learning policies which has been shown to be successful in a wide variety of applications (Mnih et al., 2015; 2016; Lillicrap et al., 2016; Hessel et al., 2018)
Revised: If success on the task can be accurately described using a reward or cost function, reinforcement learning (RL) methods offer an approach to learning policies which has proven to be successful in a wide variety of applications (Mnih et al., 2015; 2016; Lillicrap et al., 2016; Hessel et al., 2018).

Original: However, in other cases the desired behavior may only be roughly speciﬁed and it is unclear how to design a reward function to characterize it.
Revised: However, in other cases the desired behavior may only be roughly speciﬁed and it is unclear how to design a reward function to characterize it.

Original: For example, training a video game agent to adopt more human-like behavior using RL would require designing a reward function which characterizes behaviors as more or less human-like, which is difﬁcult.
Revised: For example, training a video game agent to adopt more human-like behavior using RL would require designing a reward function which characterizes behaviors as more or less human-like, which is difﬁcult.

Original: Imitation learning (IL) offers an elegant approach whereby agents are trained to mimic the demonstrations of an expert rather than optimizing a reward function.
Revised: Imitation learning (IL) offers an elegant approach whereby agents are trained to mimic the demonstrations of an expert rather than optimizing a reward function.

Original: Its simplest form consists of training a policy to predict the expert’s actions from states in the demonstration data using supervised learning.
Revised: Its simplest form consists of training a policy to predict the expert’s actions from states in the demonstration data using supervised learning.

Original: While appealingly simple, this approach suffers from the fact that the distribution over states observed at execution time can differ from the distribution observed during training.
Revised: While appealingly simple, this approach suffers from the fact that the distribution over states observed at execution time can differ from the distribution observed during training.

Original: Minor errors which initially produce small deviationsfrom the expert trajectories become magniﬁed as the policy encounters states further and further from its training distribution.
Revised: Minor errors which initially produce small deviations become magniﬁed as the policy encounters states further and further from its training distribution.

Original: This phenomenon, initially noted in the early work of (Pomerleau, 1989), was formalized in the work of (Ross & Bagnell, 2010) who proved a quadratic O ( (cid:15)T 2 ) bound on the regret and showed that this bound is tight.
Revised: This phenomenon, initially noted in the early work of (Pomerleau, 1989), was formalized in the work of (Ross & Bagnell, 2010) who proved a quadratic O ( (cid:15)T 2 ) bound on the regret and showed that this bound is tight.

Original: The subsequent work of (Ross et al., 2011) showed that if the policy is allowed to further interact with the environment and make queries to the expert policy, it is possible to obtain a linear bound on the regret.
Revised: The subsequent work of (Ross et al., 2011) showed that if the policy is allowed to further interact with the environment and make queries to the expert policy, it is possible to obtain a linear bound on the regret.

Original: However, the ability to query an expert can often be a strong assumption.
Revised: However, the ability to query an expert can often be a strong assumption.

Original: In this work, we propose a new and simple algorithm called DRIL (Disagreement-Regularized Imitation Learning) to address the covariate shift problem in imitation learning, in the setting where the agent is allowed to interact with its environment.
Revised: In this work, we propose a new and simple algorithm called DRIL (Disagreement-Regularized Imitation Learning) to address the covariate shift problem in imitation learning, in the setting where the agent is allowed to interact with its environment.

Original: Importantly, the algorithm does not require any additional interaction with the expert.
Revised: Importantly, the algorithm does not require any additional interaction with the expert.

Original: It operates by training an ensemble of policies on the demonstration data, and using the disagreement in their predictions as a cost which is optimized through RL together with a supervised behavioral cloning cost.
Revised: It operates by training an ensemble of policies on the demonstration data, and using the disagreement in their predictions as a cost which is optimized through RL together with a supervised behavioral cloning cost.

Original: The motivation is that the policies in the ensemble will tend to agree on the set of states covered by the expert, leading to low cost, but are more likely to disagree on states not covered by the expert, leading to high cost.
Revised: The motivation is that the policies in the ensemble will tend to agree on the set of states covered by the expert, leading to low cost, but are more likely to disagree on states not covered by the expert, leading to high cost.

Original: The RL cost thus pushes the agent back towards the distribution of the expert, while the supervised cost ensures that it mimics the expert within the expert’s distribution.
Revised: The RL cost thus guides the agent back towards the distribution of the expert, while the supervised cost ensures that it mimics the expert within the expert’s distribution.

Original: Our theoretical results show that, subject to realizability and optimization oracle assumptions, our algorithm obtains a O ( (cid:15)κ (cid:63) T ) regret bound for tabular MDPs, where κ(cid:63) is a measure which quantiﬁes a tradeoff between the concentration of the demonstration data and the diversity of the ensemble outside the demonstration data.
Revised: Our theoretical results show that, subject to realizability and optimization oracle assumptions 1 , our algorithm obtains a O ( (cid:15)κT ) regret bound, where κ is a measure which quantiﬁes a tradeoff between the concentration of the demonstration data and the diversity of the ensemble outside the demonstration data.

Original: We evaluate DRIL empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or signiﬁcantly outperforms behavioral cloning and generative adversarial imitation learning, often recovering expert performance with only a few trajectories.
Revised: We evaluate DRIL empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or signiﬁcantly outperforms behavioral cloning and generative adversarial imitation learning, often recovering expert performance with only a few trajectories.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Revised: We consider episodic ﬁnite horizon MDP in this work.

Original: Denote by S the state space, A the action space, and Π the class of policies the learner is considering.
Revised: Denote by S the state space, A the action space, and Π the class of policies the learner is considering.

Original: Let T denote the task horizon and π (cid:63) the expert policy whose behavior the learner is trying to mimic.
Revised: Let T denote the task horizon and π (cid:63) the expert policy whose behavior the learner is trying to mimic.

Original: For any policy π , let d π denote the distribution over states induced by following π .
Revised: For any policy π , let d π denote the distribution over states induced by following π .

Original: Denote C ( s, a ) the expected immediate cost of performing action a in state s , which we assume is bounded in [0 , 1] .
Revised: Denote C ( s, a ) the expected immediate cost of performing action a in state s , which we assume is bounded in [0 , 1] .

Original: In the imitation learning setting, we do not necessarily know the true costs C ( s, a ) ,  instead we observe expert demonstrations.
Revised: In the imitation learning setting, we do not necessarily know the true costs C ( s, a ) , and instead we observe expert demonstrations.

Original: Our goal is to ﬁnd a policy π which minimizes an observed surrogate loss (cid:96) between its actions and the actions of the expert under the induced distribution of states, i.e.
Revised: Our goal is to ﬁnd a policy π which minimizes an observed surrogate loss (cid:96) between its actions and the actions of the expert under its induced distribution of states, i.e.

Original: [Equation]
Revised: [Equation]

Original: For the following, we will assume (cid:96) is the total variation distance (denoted by (cid:107) · (cid:107) ), which is an upper bound on the 0 − 1 loss.
Revised: For the following, we will assume (cid:96) is the total variation distance (denoted by (cid:107) · (cid:107) ), which is an upper bound on the 0 − 1 loss.

Original: Our goal is thus to minimize the following quantity, which represents the distance between the actions taken by our policy π and the expert policy π (cid:63) :
Revised: Our goal is thus to minimize the following quantity, which represents the distance between the actions taken by our policy π and the expert policy π (cid:63) :

Original: [Equation]
Revised: [Equation]

Revised: Denote Q πt ( s, a ) as the standard Q-function of the policy π , which is deﬁned as Q πt ( s, a ) =

Revised: E (cid:104)(cid:80) Tτ =

Original:  The following result shows that if (cid:96) represents an upper bound on the 0 − 1 loss and C satisﬁes certain smoothness conditions, then minimizing this loss within (cid:15) translates into an O ( (cid:15)T ) regret bound on the  task cost J C ( π ) =
Revised: t C ( s τ , a τ ) | ( s t , a t ) = ( s, a ) , a τ ∼ π (cid:105) . The following result shows that if (cid:96) is an upper bound on the 0 − 1 loss and C satisﬁes certain smoothness conditions, then minimizing this loss within (cid:15) translates into an O ( (cid:15)T ) regret bound on the true task cost J C ( π ) =

Original: E s,a ∼  π [ C ( s, a )] .
Revised: E s,a ∼ d π [ C ( s, a )] :

Original: Theorem 1.
Revised: Theorem 1.

Original: (Ross et al., 2011)Let π be such that J exp ( π )
Revised: (Ross et al., 2011)

Original:  = (cid:15) , and Q π (cid:63) T − t +1 ( s, a ) − Q π (cid:63) T − t +1 ( s, π (cid:63) ) ≤ u for all a ∈ A , t∈ { 1 , 2 , ..., T } , d tπ ( s ) > 0 . Then J C ( π ) ≤ J C ( π (cid:63) ) + uT(cid:15) .
Revised: If π satisﬁes J exp ( π ) = (cid:15) , and Q π (cid:63) T − t +1 ( s, a ) − Q π (cid:63) T − t +1 ( s, π (cid:63) ) ≤ u for all time steps t , actions a and states s reachable by π , then J C ( π ) ≤ J C ( π (cid:63) ) + uT(cid:15) .

Original: Unfortunately, it is often not possible to optimize J exp directly, since it requires evaluating the expert policy on the states induced by following the current policy.
Revised: Unfortunately, it is often not possible to optimize J exp directly, since it requires evaluating the expert policy on the states induced by following the current policy.

Original: The supervised behavioral cloning cost
Revised: The supervised behavioral cloning cost

Original: J BC , which is computed on states induced by the expert, is often used instead:
Revised: J BC , which is computed on states induced by the expert, is often used instead:

Original: [Equation]
Revised: [Equation]

Original: Minimizing this loss within (cid:15) yields a quadratic regret bound on regret:
Revised: Minimizing this loss within (cid:15) yields a quadratic regret bound on regret:

Original: Theorem 2.
Revised: Theorem 2.

Original: (Ross & Bagnell, 2010) Let J BC ( π ) =
Revised: (Ross & Bagnell, 2010) Let J BC ( π ) =

Original: (cid:15) , then J C ( π ) ≤ J C ( π (cid:63) )
Revised: (cid:15) , then J C ( π ) ≤ J C ( π (cid:63) )

Original: + T 2 (cid:15) .
Revised: + T 2 (cid:15) .

Original: Furthermore, this bound is tight: as we will discuss later, there exist simple problems which match the worst-case lower bound.
Revised: Furthermore, this bound is tight: as we will discuss later, there exist simple problems which match the worst-case lower bound.

Original:  A LGORITHM
Revised:  A LGORITHM

Original: Our algorithm is motivated by two criteria: i) the policy should perform similarly to the expert on the expert’s data distribution, and ii) the policy should move towards the expert’s data distribution if it is away from it.
Revised: Our algorithm is motivated by two criteria: i) the policy should act similarly to the expert within the expert’s data distribution, and ii) the policy should move towards the expert’s data distribution if it is outside of it.

Original: These two criteria are addressed by combining two losses: a standard behavior cloning loss, and an additional loss which represents the variance overactions induced by sampling different policies from the posterior given the demonstration data D .
Revised: These two criteria are addressed by combining two losses: a standard behavior cloning loss, and an additional loss which represents the variance over the outputs of an ensemble Π E = { π 1 , ..., π E } of policies trained on the demonstration data D .

Original: We call this the uncertainty cost, which is deﬁned as:
Revised: We call this the uncertainty cost, which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The motivation is that the variance over plausible policies is high outside the expert’s distribution, since the data is sparse, but it is low inside the expert’s distribution [Equation] since the data there is dense.
Revised: The motivation is that the variance over plausible policies is high outside the expert’s distribution, since the data is sparse, but it is low inside the expert’s distribution, since the data there is dense.

Original: Minimizing this cost encourages the policy to return to regions of dense coverage by the expert.
Revised: Minimizing this cost encourages the policy to return to regions of dense coverage by the expert.

Original: Intuitively [Equation] this is what we would expect the expert policy π (cid:63) to do as well.
Revised: Intuitively, this is what we would expect the expert policy π (cid:63) to do as well.

Original: The total cost which the algorithm optimizes is given by:
Revised: The total cost which the algorithm optimizes is given by:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is a behavior cloning loss and is computed over states generated by the expert policy, of which the demonstration data
Revised: The ﬁrst term is a behavior cloning loss and is computed over states generated by the expert policy, of which the demonstration data

Original: D is a representative sample.
Revised: D is a representative sample.

Original: The second term is computed over the distribution of states generated by the current policy and can be optimized using policy gradient.
Revised: The second term is computed over the distribution of states generated by the current policy and can be optimized using policy gradient.

Original: A number of methods can be used to approximate the posterior p ( π |D ) .

Original: In our experiments, we use an ensemble E = { π e } | E | e =1 of models with different initializations which are trained on different bootstrap samples of the demonstration data.

Original: Note that the demonstration data is ﬁxed, and this ensemble can be trained once ofﬂine.
Revised: Note that the demonstration data is ﬁxed, and this ensemble can be trained once ofﬂine.

Original: We then interleave the supervised behavioral cloning updates and the policy gradient updates which minimize the variance of the posterior.
Revised: We then interleave the supervised behavioral cloning updates and the policy gradient updates which minimize the variance of the ensemble.

Original: The full algorithm is shown in Algorithm 1.
Revised: The full algorithm is shown in Algorithm 1.

Original: Other methods for approximating the posterior include Bayesian neural networks (MacKay, 1992) using diagonal Gaussian approximations (Blundell et al., 2015) or MCdropout (Gal & Ghahramani, 2016), which we also found to work well (see Appendix D.2).
Revised: We also found that dropout (Srivastava et al., 2014), which has been proposed as an approximate form of ensembling, worked well (see Appendix D).

Original: In practice, for the supervised loss we optimize the KL divergence between the actions predicted by the policy and the expert actions, which is an upper bound on the total variation distance due to
Revised: In practice, for the supervised loss we optimize the KL divergence between the actions predicted by the policy and the expert actions, which is an upper bound on the total variation distance due to

Original: Pinsker’s inequality.
Revised: Pinsker’s inequality.

Original: We also found it helpful to use a clipped uncertainty cost:
Revised: We also found it helpful to use a clipped uncertainty cost:

Original: [Equation]
Revised: [Equation]

Original: where the threshold q is a top quantile of the raw uncertainty costs computed over the demonstration data.
Revised: where the threshold q is a top quantile of the raw uncertainty costs computed over the demonstration data.

Original: The threshold q deﬁnes a normal range of uncertainty based on the demonstration data, and values above this range incur a positive cost (or negative reward).
Revised: The threshold q deﬁnes a normal range of uncertainty based on the demonstration data, and values above this range incur a positive cost (or negative reward).

Original: The RL cost can be optimized using any policy gradient method.
Revised: The RL cost can be optimized using any policy gradient method.

Original: In our experiments we used advantage actor-critic (A2C)
Revised: In our experiments we used advantage actor-critic (A2C)

Original: (Mnih et al., 2016), which estimates the expected cost using rollouts from multiple parallel actors all sharing the same policy (see Appendix C for details).
Revised: (Mnih et al., 2016) or PPO (Schulman et al., 2017), which estimate the expected cost using rollouts from multiple parallel actors all sharing the same policy (see Appendix C for details).

Original: We note that model-based RL methods could in principle be used as well if sample efﬁciency is a constraint.
Revised: We note that model-based RL methods could in principle be used as well if sample efﬁciency is a constraint.

Original:  A NALYSIS 4. C OVERAGE C OEFFICIENT
Revised:  A NALYSIS 4. C OVERAGE C OEFFICIENT

Original: We now analyze DRIL for tabular MDPs.
Revised: We now analyze DRIL for MDPs with discrete action spaces and potentially large or inﬁnite state spaces.

Original: We will show that, subject to assumptions that the policy class contains an optimal policy and that we are able to optimize costs within (cid:15) of their global minimum, our algorithm obtains a regret bound which is linear in κ (cid:63) T , where κ(cid:63) is a quantity speciﬁc to the environment and d (cid:63)π .
Revised: We will show that, subject to assumptions that the policy class contains an optimal policy and that we are able to optimize costs within (cid:15) of their global minimum, our algorithm obtains a regret bound which is linear in κT , where κ is a quantity which depends on the environment dynamics, the expert distribution d (cid:63)π , and our learned ensemble.

Original: Intuitively, κ(cid:63) represents a tradeoff between how concentrated the demonstration data is and how high the variance of the posterior is outside the expert distribution.
Revised: Intuitively, κ represents a tradeoff between how concentrated the demonstration data is and how high the variance of the ensemble is outside the expert distribution.

Original: Assumption 1.
Revised: Assumption 1.

Original: (Realizability) π (cid:63) ∈ Π .
Revised: (Realizability [Equation] ∈ Π .

Original: Assumption 2.
Revised: Assumption 2.

Original: (Optimization Oracle)
Revised: (Optimization Oracle)

Original: For any given cost function J , our minimization procedure returns a policy ˆ π ∈ Π such that J (ˆ π ) ≤ arg min π ∈ Π J
Revised: For any given cost function J , our minimization procedure returns a policy ˆ π ∈ Π such that J (ˆ π ) ≤ arg min π ∈ Π J

Original: ( π ) + (cid:15) .
Revised: ( π ) + (cid:15) .

Original: The motivation behind our algorithm is that the policies in the ensemble agree inside the expert’s distribution and disagree outside of it.
Revised: The motivation behind our algorithm is that the policies in the ensemble agree inside the expert’s distribution and disagree outside of it.

Original: This deﬁnes a reward function which pushes the learner back towards the expert’s distribution if it strays away.
Revised: This deﬁnes a reward function which pushes the learner back towards the expert’s distribution if it strays away.

Original: However, what constitutes inside and outside the distribution, or sufﬁcient agreement or disagreement, is ambiguous.
Revised: However, what constitutes inside and outside the distribution, or sufﬁcient agreement or disagreement, is ambiguous.

Original: Below we deﬁne quantities which makes these ideas precise.
Revised: Below we introduce quantities which makes these ideas precise.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: For any set U ⊆ S , deﬁne the maximum probability ratio between the state distributions induced by the expert policy and by policies in the policy class inside of U as α ( U ) =
Revised: For any set U ⊆ S , deﬁne the concentrability inside of U as α ( U ) =

Original: [Equation]
Revised: [Equation]

Original: Note that α ( U ) ≤ [Equation] (cid:63)π ( s ) .
Revised: The notion of concentrability has been previously used to give bounds on the performance of value iteration (Munos & Szepesv ´ ari, 2008).

Original: For a set U , α ( U ) will be low if the expert distribution has high mass inside of U , and the states in U is reachable by policies in the policy class.
Revised: For a set U , α ( U ) will be low if the expert distribution has high mass at the states in U that are reachable by policies in the policy class.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Deﬁne the minimum variance of the posterior outside of U as β ( U ) =
Revised: Deﬁne the minimum variance of the ensemble outside of U as β ( U ) =

Original: min [Equation] [Equation] [Equation]
Revised: min s/ ∈U ,a ∈A Var π ∼ Π E

Original: |D )

Original: [ π ( a | s )] .
Revised: [ π ( a | s )] .

Original: We now deﬁne the κ(cid:63) coefﬁcient as the minimum ratio of these two quantities over all possible subsets of S .
Revised: We now deﬁne the κ coefﬁcient as the minimum ratio of these two quantities over all possible subsets of S .

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: We deﬁne κ( U ) =
Revised: We deﬁne κ =

Original: α ( U ) β ( U ) , and κ (cid:63) =
Revised: min

Original: min U⊆S κ ( U ) .
Revised: U⊆S α ( U ) β ( U ) .

Original: We can viewminimizing κ ( U ) over different  U⊆ S as minimizing a tradeoff between coverage by the expert policy inside of U , and variance of the posterior outside of U .
Revised: We can view κ as the quantity which minimizes the tradeoff over different subsets U between coverage by the expert policy inside of U , and variance of the ensemble outside of U .

Original: Note that by making U very small, it may be easy to make α ( U ) small, but doing so may also make β ( U ) small and κ ( U ) large.

Original: Conversely, making U large may make β ( U ) large but may also make α ( U ) large as a result.

Original: 4. R EGRET B OUND
Revised: 4. R EGRET B OUND

Original: We now establish a relationship between the κ(cid:63) coefﬁcient just deﬁned, the cost our algorithm optimizes, and J exp deﬁned in Equation (2) which we would ideally like to minimize and which translates into a regret bound.
Revised: We now establish a relationship between the κ coefﬁcient just deﬁned, the cost our algorithm optimizes, and J exp deﬁned in Equation (2) which we would ideally like to minimize and which translates into a regret bound.

Original: All proofs can be found in Appendix A.
Revised: All proofs can be found in Appendix A.

Original: Lemma 1.
Revised: Lemma 1.

Original: For any π ∈ Π , we have J exp ( π ) ≤ κ (cid:63) J alg ( π ) .
Revised: For any π ∈ Π , we have J exp ( π ) ≤ κJ alg ( π ) .

Original: This result shows that if κ(cid:63) is not too large, and we are able to make our cost function J alg ( π ) small, then we can ensure J exp ( π ) is alsobe small.
Revised: This result shows that if κ is not too large, and we are able to make our cost function J alg ( π ) small, then we can ensure J exp ( π ) is also small.

Original: This result is only useful if our cost function can indeed achieve a small minimum.
Revised: This result is only useful if our cost function can indeed achieve a small minimum.

Original: The next lemma shows that this is the case.
Revised: The next lemma shows that this is the case.

Original: Lemma 2.
Revised: Lemma 2.

Original: min π ∈ Π J alg ( π ) ≤ 2 (cid:15) .
Revised: min π ∈ Π J alg ( π ) ≤ 2 (cid:15) .

Original: Here (cid:15) is the threshold speciﬁed in Assumption 2.
Revised: Here (cid:15) is the threshold speciﬁed in Assumption 2.

Original: Combining these two lemmas with the previous result of Ross et al.
Revised: Combining these two lemmas with the previous result of Ross et al.

Original: (2011), we get a regret bound which is linear in κ (cid:63) T .
Revised: (2011), we get a regret bound which is linear in κT .

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ˆ π be the result of minimizing J alg using our optimization oracle, and assume that
Revised: Let ˆ π be the result of minimizing J alg using our optimization oracle, and assume that

Original: [Equation] a [Equation] +
Revised: Q π (cid:63) T − t +1 ( s, a ) − Q π (cid:63) T − t +1 ( s, π (cid:63) ) ≤ u for all actions

Original: [Equation] ( s, π [Equation] ) ≤ u for all a∈ A ,  t ∈ { 1 , 2 , ..., T } , d tπ ( s ) > 0 .
Revised: a , time steps t and states s reachable by π .

Original: Then ˆ π satisﬁes J 
Revised: Then ˆ π satisﬁes J C (ˆ π ) ≤ J C

Original: [Equation] Our bound is an improvement over that of behavior cloning if κ(cid:63) is less than O ( T ) .
Revised: ( π (cid:63) ) + 3 uκ(cid:15)T . Our bound is an improvement over that of behavior cloning if κ is less than O ( T ) .

Original: Note that
Revised: Note that

Original: DRIL does not require knowledge of κ(cid:63) .
Revised: DRIL does not require knowledge of κ .

Original: The quantity κ(cid:63) is problem-dependent and depends on the environment dynamics, the expert policy and theclass of policies available to the learner.
Revised: The quantity κ is problem-dependent and depends on the environment dynamics, the expert policy and the policies in the learned ensemble.

Original: We next compute κ(cid:63) exactly for a problem for which behavior cloning is known to perform poorly, and show that it is independent of T .
Revised: We next compute κ exactly for a problem for which behavior cloning is known to perform poorly, and show that it is independent of T .

Original: Example 1.
Revised: Example 1.

Original: Consider the tabular MDP given in (Ross & Bagnell, 2010) as an example of a problem where behavioral cloning incurs quadratic regret, shown in Figure 1.
Revised: Consider the tabular MDP given in (Ross & Bagnell, 2010) as an example of a problem where behavioral cloning incurs quadratic regret, shown in Figure 1.

Original: There are 3 states S =
Revised: There are 3 states S =

Original: ( s 0 , s 1 , s 2 ) and two actions ( a 1 , a 2 ) .
Revised: ( s 0 , s 1 , s 2 ) and two actions ( a 1 , a 2 ) .

Original: Each policy π can be represented as a set of probabilities π ( a 1 | s ) for each state s ∈ S 1 .
Revised: Each policy π can be represented as a set of probabilities π ( a 1 | s ) for each state s ∈ S 2 .

Original: The posterior p ( π ( a 1 | s ) |D )is given by a Beta distribution with parameters Beta ( n 1 + 1 , n 2 + 1) where n 1 , n 2 are the number of times the pairs ( s, a 1 ) and ( s, a 2 ) occur, respectively, in the demonstration data D .
Revised: Assume the models in our ensemble are drawn from a posterior p ( π ( a 1 | s ) |D ) given by a Beta distribution with parameters Beta ( n 1 + 1 , n 2 + 1) where n 1 , n 2 are the number of times the pairs ( s, a 1 ) and ( s, a 2 ) occur, respectively, in the demonstration data D .

Original: The agent always starts in s 0 and the expert’s policy is given by π (cid:63) ( a 1 | s 0 )
Revised: The agent always starts in s 0 and the expert’s policy is given by π (cid:63) ( a 1 | s 0 )

Original: = 1 , π (cid:63) ( a 1 | s 1 )
Revised: = 1 , π (cid:63) ( a 1 | s 1 )

Original: = 0 , π (cid:63) ( a 1 | s 2 )
Revised: = 0 , π (cid:63) ( a 1 | s 2 )

Original: = 1 .
Revised: = 1 .

Original: {+For any ( s, a ) pair, the task cost is C ( s, a ) = 
+}
Revised: {+For any ( s, a ) pair, the task cost is C ( s, a ) = 
+}

Original:  Here d (cid:63)π = ( 1 T , T − 1 T , 0) .Writing out κ ( { s 0 , s 1 } ) yields:
Revised: if a = π (cid:63) ( s ) and 1 otherwise. Here d (cid:63)π = ( 1 T , T − 1 T , 0) .

Original: [Equation] For any π , d π ( s 0 ) = 1 T and d π ( s 1 ) ≤ T − 1 T due to the dynamics of the MDP, so d π ( s ) d (cid:63)π ( s ) ≤ 1 for s ∈ { s 0 , s 1 } . 
Revised: For any π , d π ( s 0 ) = 1 T and d π ( s 1 ) ≤ T − 1 T due to the dynamics of the MDP, so d π ( s ) d (cid:63)π ( s ) ≤ 1 for s ∈ { s 0 , s 1 } . Writing out α ( { s 0 , s 1 } ) , we get: α ( { s 0 , s 1 } ) = max π ∈ Π sup s ∈{ s 0 ,s 1 } d π ( s ) d (cid:63)π ( s ) ≤ 1 .

Original: Furthermore, since s 2 is never visited in the demonstration data, p ( π ( a 1 | s 2 ) |D ) =
Revised: Furthermore, since s 2 is never visited in the demonstration data, for each policy

Original: Beta (1 , 1) = Uniform (0 , 1) , which also implies that p ( π  ( a 1 | s 2 ) |D )  =
Revised: π i in the ensemble we have π i ( a 1 | s 2 ) , π i ( a 2 | s 2 ) ∼ Beta (1 , 1) =

Original: Uniform (0 , 1) .
Revised: Uniform (0 , 1) .

Original: It follows that Var π ∼ p ( π |D ) ( π ( a | s 2 )) is  equal  to the variance of a uniform distribution over [0 , 1] , i.e. 112 .
Revised: It follows that Var π ∼ Π E ( π ( a | s 2 )) is approximately equal 3 to the variance of a uniform distribution over [0 , 1] , i.e. 112 .

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: Applying our result from Theorem 3, we see that our algorithm obtains an O ( (cid:15)T ) regret bound on this problem, in contrast to the O ( (cid:15)T 2 ) regret of behavioral cloning 2 .
Revised: Applying our result from Theorem 3, we see that our algorithm obtains an O ( (cid:15)T ) regret bound on this problem, in contrast to the O ( (cid:15)T 2 ) regret of behavioral cloning 4 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The idea of learning through imitation dates back at least to the work of (Pomerleau, 1989), who trained a neural network to imitate the steering actions of a human driver using images as input.
Revised: The idea of learning through imitation dates back at least to the work of (Pomerleau, 1989), who trained a neural network to imitate the steering actions of a human driver using images as input.

Original: The problem of covariate shift was already observed, as the author notes: “when driving for itself, the networkmay occasionally stray from the center of the road and so must  be prepared to recover by steering the vehicle back to the center of the road”.
Revised: The problem of covariate shift was already observed, as the author notes: “the network must not solely be shown examples of accurate driving, but also how to recover once a mistake has been made”.

Original: This issue was formalized in the work of (Ross & Bagnell, 2010), who on one hand proved an
Revised: This issue was formalized in the work of (Ross & Bagnell, 2010), who on one hand proved an

Original: O ( (cid:15)T 2 ) regret bound, and on the other hand provided an example showing this bound is tight.
Revised: O ( (cid:15)T 2 ) regret bound, and on the other hand provided an example showing this bound is tight.

Original: The subsequent work (Ross et al., 2011) proposed the DAGGER algorithm which obtains linear regret, provided the agent can both interact with the environment, and query the expert policy.
Revised: The subsequent work (Ross et al., 2011) proposed the DAGGER algorithm which obtains linear regret, provided the agent can both interact with the environment, and query the expert policy.

Original: Our approach also requires environment interaction, but importantly does not require the ability to query the expert.
Revised: Our approach also requires environment interaction, but importantly does not need to query the expert.

Original: Also of note is the work of (Venkatraman et al., 2015), which extended DAGGER to time series prediction problems by using the true targets as expert corrections.
Revised: Also of note is the work of (Venkatraman et al., 2015), which extended DAGGER to time series prediction problems by using the true targets as expert corrections.

Original: Imitation learning has been used within the context of modern RL to help improve sample efﬁciency (Hester et al.,  2018) or overcome exploration (Nair et al., 2017).
Revised: Imitation learning has been used within the context of modern RL to help improve sample efﬁciency (Chang et al., 2015; Ross & Bagnell, 2014; Sun et al., 2017; Hester et al., 2018; Le et al., [Figure] Cheng & Boots, 2018) or overcome exploration (Nair et al., 2017).

Original: These settings assume the reward is known and that the policies can then be ﬁne-tuned with reinforcement learning.
Revised: These settings assume the reward is known and that the policies can then be ﬁne-tuned with reinforcement learning.

Original: In this case, covariate shift is less of an issue since it can be corrected using the reinforcement signal.
Revised: In this case, covariate shift is less of an issue since it can be corrected using the reinforcement signal.

Original: The work of (Luo et al., 2019) also proposed a method to address the covariate shift problem when learning from demonstrations when the reward is known, by conservatively extrapolating the value function outside the training distribution using negative sampling.
Revised: The work of (Luo et al., 2019) also proposed a method to address the covariate shift problem when learning from demonstrations when the reward is known, by conservatively extrapolating the value function outside the training distribution using negative sampling.

Original: This addresses a different setting from ours, and requires generating plausible states which are off the manifold of training data, which may be challenging when the states are high dimensional such as images.
Revised: This addresses a different setting from ours, and requires generating plausible states which are off the manifold of training data, which may be challenging when the states are high dimensional such as images.

Original: The work of Reddy et al. (2019) proposed to treat imitation learning within the Q-learning framework, setting a positive reward for all transitions inside the demonstration data and zero reward for all other transitions in the replay buffer.
Revised: The work of (Reddy et al., 2019) proposed to treat imitation learning within the Q-learning framework, setting a positive reward for all transitions inside the demonstration data and zero reward for all other transitions in the replay buffer.

Original: This rewards the agent for repeating (or returning to) the expert’s transitions.
Revised: This rewards the agent for repeating (or returning to) the expert’s transitions.

Original: The work of (Sasaki et al., 2019) also incorporates a mechanism for reducing covariate shift by ﬁtting a Q-function that classiﬁes whether the demonstration states are reachable from the current state.
Revised: The work of (Sasaki et al., 2019) also incorporates a mechanism for reducing covariate shift by ﬁtting a Q-function that classiﬁes whether the demonstration states are reachable from the current state.

Original: Random Expert Distillation (Wang et al., 2019) uses Random Network Distillation (RND)Burda et al.
Revised: Random Expert Distillation (Wang et al., 2019) uses Random Network Distillation (RND)

Original: (2019) to estimate the support of the expert’s distribution in state-action space, and minimizes an RL cost designed to guide the agent towards the expert’s support.
Revised: (Burda et al., 2019) to estimate the support of the expert’s distribution in state-action space, and minimizes an RL cost designed to guide the agent towards the expert’s support.

Original: This is related to our method, but differs in that it minimizes the RND prediction error rather than the posterior variance and does not include a behavior cloning cost.
Revised: This is related to our method, but differs in that it minimizes the RND prediction error rather than the ensemble variance and does not include a behavior cloning cost.

Original: The behavior cloning cost is essential to our theoretical results and avoids certain failure modes, see Appendix B for more discusion.
Revised: The behavior cloning cost is essential to our theoretical results and avoids certain failure modes, see Appendix B for more discusion.

Original: Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) is a state-of-the-art algorithm which addresses the same setting as ours.
Revised: Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) is a state-of-the-art algorithm which addresses the same setting as ours.

Original: It operates by training a discriminator network to distinguish expert states from states generated by the current policy, and the negative output of the discriminator is used as a reward signal to train the policy.
Revised: It operates by training a discriminator network to distinguish expert states from states generated by the current policy, and the negative output of the discriminator is used as a reward signal to train the policy.

Original: The motivation is that states which are outside the training distribution will be assigned a low reward while states which are close to it will be assigned a high reward.
Revised: The motivation is that states which are outside the training distribution will be assigned a low reward while states which are close to it will be assigned a high reward.

Original: This encourages the policy to return to the expert distribution if is strays away from it.
Revised: This encourages the policy to return to the expert distribution if it strays away from it.

Original: However, the adversarial training procedure means that the reward function is changing over time, which can make the algorithm unstable or difﬁcult to tune.
Revised: However, the adversarial training procedure means that the reward function is changing over time, which can make the algorithm unstable or difﬁcult to tune.

Original: In contrast, our approach uses a simple ﬁxed reward function.
Revised: In contrast, our approach uses a simple ﬁxed reward function.

Original: We include comparisons to GAIL in our experiments.
Revised: We include comparisons to GAIL in our experiments.

Original: Using disagreement between models in an ensemble to represent uncertainty has recently been explored in several contexts.
Revised: Using disagreement between models in an ensemble to represent uncertainty has recently been explored in several contexts.

Original: The works of (Shyam et al., 2018; Pathak et al.,  2019) used disagreement between different dynamics models to drive exploration in the context of model-based RL.
Revised: The works of (Shyam et al., 2018; Pathak et al., 2019; Henaff, 2019) used disagreement between different dynamics models to drive exploration in the context of modelbased RL.

Original: Conversely, (Henaff et al., 2019) used variance across different dropout masks to prevent policies from exploiting error in dynamics models.
Revised: Conversely, (Henaff et al., 2019) used variance across different dropout masks to prevent policies from exploiting error in dynamics models.

Original: Ensembles have also been used to represent uncertainty over Q-values in model-free RL in order to encourage exploration (Osband et al., 2016).
Revised: Ensembles have also been used to represent uncertainty over Q-values in model-free RL in order to encourage exploration (Osband et al., 2016).

Original: Within the context of imitation learning, the work of (Menda et al., 2018) used the variance of the ensemble together with the DAGGER algorithm to decide when to query the expert demonstrator to minimize unsafe situations.
Revised: Within the context of imitation learning, the work of (Menda et al., 2018) used the variance of the ensemble together with the DAGGER algorithm to decide when to query the expert demonstrator to minimize unsafe situations.

Original: Here, we use disagreement between different policies sampled from the posterior to address covariate shift in the context of imitation learning.
Revised: Here, we use disagreement between different policies trained on demonstration data to address covariate shift in the context of imitation learning.

Original:  E XPERIMENTS 6. T ABULAR MDP S
Revised:  E XPERIMENTS 6. T ABULAR MDP S

Original: As a ﬁrst experiment, we applied DRIL to the tabular MDP of (Ross & Bagnell, 2010) shown in
Revised: As a ﬁrst experiment, we applied DRIL to the tabular MDP of (Ross & Bagnell, 2010) shown in

Original: Figure 1.
Revised: Figure 1.

Original: We computed the posterior over the policy parameters given the demonstration data using[Figure] a separate Beta distribution for each state s with parameters determined by the number of times each action was performed in s .
Revised: We computed the posterior over the policy parameters given the demonstration data using a separate Beta distribution for each state s with parameters determined by the number of times each action was performed in s .

Original: For behavior cloning, we sampled a single policy from this posterior.
Revised: For behavior cloning, we sampled a single policy from this posterior.

Revised: For

Original: For our method, we sampled  5 policies and used their negative variance to deﬁne an additional reward function.
Revised: DRIL, we sampled an ensemble of 5 policies and used their negative variance to deﬁne an additional reward function.

Original: We combined this with a reward which was the probability density function of a given state-action pair under the posterior distribution, which corresponds to the supervised learning loss, and used tabular Q-learning to optimize the sum of these two reward functions.
Revised: We combined this with a reward which was the probability density function of a given state-action pair under the posterior distribution, which corresponds to the supervised learning loss, and used tabular Q-learning to optimize the sum of these two reward functions.

Original: This experiment was repeated 500 times for time horizon lengths up to 500 and N = 1 , 5 , 10 expert demonstration trajectories.
Revised: This experiment was repeated 500 times for time horizon lengths up to 500 and N = 1 , 5 , 10 expert demonstration trajectories.

Original: Figure 2 shows plots of the regret over the 500 different trials across different time horizons.
Revised: Figure 2 shows plots of the regret over the 500 different trials across different time horizons.

Original: Althoughthe average performance of BC improves with more expert demonstrations, it exhibits poor worst-case performance with some trials incurring very high regret, especially when using fewer demonstrations.
Revised: Although BC achieves good average performance, it exhibits poor worst-case performance with some trials incurring very high regret, especially when using fewer demonstrations.

Original: Our method has low regret across all trials, which stays close to constant independantly of the time horizon, even with a single demonstration.
Revised: Our method has low regret across all trials, which stays close to constant independantly of the time horizon, even with a single demonstration.

Original: This performance is better than that suggested by our analysis, which showed a worst-case linear bound with respect to time horizon.
Revised: This performance is better than that suggested by our analysis, which showed a worst-case linear bound with respect to time horizon.

Original: 6. A TARI E NVIRONMENTS
Revised: 6. A TARI E NVIRONMENTS

Original: We next evaluated our approach on six different Atari environments.
Revised: We next evaluated our approach on six different Atari environments.

Original: We used pretrained PPO (Schulman et al., 2017) agents from the stable baselines repository (Hill et al., 2018) to generate
Revised: We used pretrained PPO (Schulman et al., 2017) agents from the stable baselines repository (Hill et al., 2018) to generate

Original: N = { 1 , 3 , 5 , 10 , 15 , 20 } expert trajectories.
Revised: N = { 1 , 3 , 5 , 10 , 15 , 20 } expert trajectories.

Original: We compared against two other methods: standard behavioral cloning (BC) and Generative Adversarial Imitation Learning (GAIL).
Revised: We compared against two other methods: standard behavioral cloning (BC) and Generative Adversarial Imitation Learning (GAIL).

Original: Results are shown in
Revised: Results are shown in

Original: Figure 3a.
Revised: Figure 3a.

Original: DRIL outperforms behavioral cloning across most environments and numbers of demonstrations, often by a substantial margin.
Revised: DRIL outperforms behavioral cloning across most environments and numbers of demonstrations, often by a substantial margin.

Original: In the worst case its performance matches that of behavior cloning.

Original: In many cases, our method is able to match the expert’s performance using a small number of trajectories.
Revised: In many cases, our method is able to match the expert’s performance using a small number of trajectories.

Original: Figure 3b shows the evolution of the uncertainty cost and the policy reward throughout training.
Revised: Figure 3b shows the evolution of the uncertainty cost and the policy reward throughout training.

Original: In all cases, thetest reward improves while the uncertainty cost decreases.
Revised: In all cases, the reward improves while the uncertainty cost decreases.

Original: Interestingly, there is correspondence between the change in the uncertainty cost during training and the gap in performance between behavior cloning and DRIL.

Original: For example, in MsPacman there is both a small improvement in uncertainty cost over time and a small gap between behavior cloning and our method, whereas in

Original: Breakout there is a large improvement in uncertainty cost and a large gap between behavior cloning and our method.

Original: This suggests that the gains from our method comes from redirecting the policy back towards the expert manifold, which is manifested as a decrease in the uncertainty cost.

Original: We were not able to obtain meaningful performance for GAIL on these domains, despite performing a hyperparameter search across learning rates for the policy and discriminator, and across different numbers of discriminator updates.
Revised: We were not able to obtain meaningful performance for GAIL on these domains, despite performing a hyperparameter search across learning rates for the policy and discriminator, and across different numbers of discriminator updates.

Original: We additionally experimented with clipping rewards in an effort to stabilize performance.
Revised: We additionally experimented with clipping rewards in an effort to stabilize performance.

Original: These results are consistent with those of (Reddy et al., 2019), who also reported negative results when running GAIL on images.
Revised: These results are consistent with those of (Reddy et al., 2019), who also reported negative results when running GAIL on images.

Original: While improved performance might be possible with more sophisticated adversarial training techniques, we note that this contrasts with our method which uses a ﬁxed reward function obtained through simple supervised learning.
Revised: While improved performance might be possible with more sophisticated adversarial training techniques, we note that this contrasts with our method which uses a ﬁxed reward function obtained through simple supervised learning.

Original: In Appendix D we provide ablation experiments examining the effects of different cost function choices and the role of the BC loss.
Revised: In Appendix D we provide ablation experiments examining the effects of the cost function clipping and the role of the BC loss.

Original: We also compare the ensemble approach to a dropout-basedapproach for posterior approximation and show that DRIL works well in both cases.[Figure]
Revised: We also compare the ensemble approach to a dropout-based approximation and show that DRIL works well in both cases.

Original: 6. C ONTINUOUS C ONTROL
Revised: 6. C ONTINUOUS C ONTROL

Original: We next report results of running our method ona 6 different continuous control tasks from the 
Revised: We next report results of running our method on 6 different continuous control tasks from the PyBullet 5 and

Original: PyBullet 3 and OpenAI Gym (Brockman et al., 2016) environments.
Revised: OpenAI Gym (Brockman et al., 2016) environments.

Original: We again used pretrained agents to generate expert demonstrations.
Revised: We again used pretrained agents to generate expert demonstrations, and compared to Behavior Cloning and GAIL.

Original: Results  are shown in Figure 4.
Revised: Results for all methods are shown in Figure 4.

Original: In these environments we found behavior cloning to be a much stronger baseline than for the Atari environments, and in many tasks it was able to match expert performance using as little as 3 trajectories.
Revised: In these environments we found Behavior Cloning to be a much stronger baseline than for the Atari environments: in several tasks it was able to match expert performance using as little as 3 trajectories, suggesting that covariate shift may be less of an issue.

Original: Our method exhibits a modest improvement on Walker2D and BipedalWalkerHardcore when a single trajectory is used, and otherwise has similar performance to behavior cloning. The fact that our method does not perform worse than behavior cloning on tasks where covariate shift is likely less of an issue provides evidence of its robustness.
Revised: Our method performs similarly to Behavior Cloning on most tasks, except on Walker2D, where it yields improved performance for N = 1 , 3 , 5 trajectories. GAIL performs somewhat better than DRIL on HalfCheetah and Walker2D, but performs worse than both DRIL

Original: [Figure]
Revised: [Figure]

Revised: and BC on LunarLander and BipedalWalkerHardcore.

Revised: The fact that DRIL is competitive across all tasks provides evidence of its robustness.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Addressing covariate shift has been a long-standing challenge in imitation learning.
Revised: Addressing covariate shift has been a long-standing challenge in imitation learning.

Original: In this work, we have proposed a new method to address this problem by penalizing the disagreement between an ensemble of different policies sampled from the posterior.
Revised: In this work, we have proposed a new method to address this problem by penalizing the disagreement between an ensemble of different policies trained on the demonstration data.

Original: Importantly, our method requires no additional labeling by an expert.
Revised: Importantly, our method requires no additional labeling by an expert.

Original: Our experimental results demonstrate that DRIL can often match expert performance while using only a small number of trajectories across a wide array of tasks, ranging from tabular MDPs to pixel-based Atari games and continuous control tasks.
Revised: Our experimental results demonstrate that DRIL can often match expert performance while using only a small number of trajectories across a wide array of tasks, ranging from tabular MDPs to pixel-based Atari games and continuous control tasks.

Original: On the theoretical side, we have shown that our algorithm can provably obtain a low regret bound fortabular problems in which the κ(cid:63) parameter is low. 
Revised: On the theoretical side, we have shown that our algorithm can provably obtain a low regret bound for problems in which the κ parameter is low. [Figure]

Original: There are multiple directions for future work.
Revised: There are multiple directions for future work.

Original: On the theoretical side,extending our analysis to continuous state spaces and characterizing the κ(cid:63) parameter on a larger array of problems would help to better understand the settings where our method can expect to do well.
Revised: On the theoretical side, characterizing the κ parameter on a larger array of problems would help to better understand the settings where our method can expect to do well.

Original: Empirically, there are many other settings in structured prediction (Daum ´ e et al., 2009) where covariate shift is an issue and where our method could be applied.
Revised: Empirically, there are many other settings in structured prediction (Daum ´ e et al., 2009) where covariate shift is an issue and where our method could be applied.

Original: For example, in dialogue and language modeling it is common for generated text to become progressively less coherent as errors push the model off the manifold it was trained on.
Revised: For example, in dialogue and language modeling it is common for generated text to become progressively less coherent as errors push the model off the manifold it was trained on.

Original: Our method could potentially be used to ﬁne-tune language or translation models (Cho et al., 2014; Welleck et al., 2019) after training by applying our uncertainty-based cost function to the generated text.
Revised: Our method could potentially be used to ﬁne-tune language or translation models (Cho et al., 2014; Welleck et al., 2019) after training by applying our uncertainty-based cost function to the generated text [Figure]

Original: A P ROOFS
Revised: A P ROOFS

Original: Lemma 1.
Revised: Lemma 1.

Original: For any π ∈ Π we have J exp ( π ) ≤ κJ alg ( π )
Revised: For any π ∈ Π we have J exp ( π ) ≤ κJ alg ( π )

Original: Proof.
Revised: Proof.

Original: We will ﬁrst show that for any π ∈ Π and U ⊆ S , we have J exp ( π ) ≤ α ( U ) β
Revised: We will ﬁrst show that for any π ∈ Π and U ⊆ S , we have J exp ( π ) ≤ α ( U ) β

Original: ( U ) J alg ( π ) .
Revised: ( U ) J alg ( π ) .

Original: We can rewrite this as:
Revised: We can rewrite this as:

Original: [Equation]
Revised: [Equation]

Original: We begin by bounding the ﬁrst term:
Revised: We begin by bounding the ﬁrst term:

Original: [Equation]
Revised: [Equation]

Original: We next bound the second term:
Revised: We next bound the second term:

Original: [Equation]
Revised: [Equation]

Original: Now observe we can decompose the RL cost as follows:
Revised: Now observe we can decompose the RL cost as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Putting these together
Revised: Putting these together

Original: [Equation]
Revised: [Equation]

Original: we get the following:
Revised: we get the following:

Original: [Equation]
Revised: [Equation]

Original: Here we have used the fact that β ( U ) ≤ 1 since 0 ≤ π ( a | s ) ≤ 1 and α ( U ) ≥ (cid:80) s ∈U d (cid:63)π ( s ) d (cid:63)π ( s ) ≥hence 1 α ( U ) ≤ 1 .
Revised: Here we have used the fact that β ( U ) ≤ 1 since 0 ≤ π ( a | s ) ≤ 1 and α ( U ) ≥ sup s ∈U d (cid:63)π ( s ) d (cid:63)π ( s ) = hence 1 α ( U ) ≤ 1 .

Original: Taking the minimum over subsets U ⊆ S , we get J exp ( π ) ≤ κJ alg ( π ) .
Revised: Taking the minimum over subsets U ⊆ S , we get J exp ( π ) ≤ κJ alg ( π ) .

Original: Lemma 2.
Revised: Lemma 2.

Original: min π ∈ Π J alg ( π ) ≤ 2 (cid:15)
Revised: min π ∈ Π J alg ( π ) ≤ 2 (cid:15)

Original: Proof.
Revised: Proof.

Original: Plugging the optimal policy into J alg , we get:
Revised: Plugging the optimal policy into J alg , we get:

Original: [Equation]
Revised: [Equation]

Original: Term
Revised: Term

Original: We will ﬁrst bound Term 1:
Revised: We will ﬁrst bound Term 1:

Original: [Table]
Revised: [Table]

Original: We will next bound Term 2:
Revised: We will next bound Term 2:

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: | | E | (cid:88) i =
Revised: [Table] [Equation] [Table] [Equation]

Original: E s ∼ d π(cid:63) ,a ∼ π (cid:63) ( ·| s ) (cid:104)(cid:12)(cid:12)(cid:12) π (cid:63) ( a
Revised: [Table] [Equation] [Table] − Schwarz

Original: [Equation]
Revised: [Equation]

Original: s ) − π i ( a | s ) (cid:12)(cid:12)(cid:12)(cid:105)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]
Revised: [Table]

Original:  ∼ 
Revised: ≤| E | | E | (cid:88) i = E s ∼ d π(cid:63) ,a ∼ π (cid:63) ( ·| s ) (cid:104)(cid:12)(cid:12)(cid:12) π (cid:63) ( a | s ) − π i ( a | s ) (cid:12)(cid:12)(cid:12)(cid:105) ≤| E | | E | (cid:88) i = E s ∼ d π(cid:63) (cid:104) (cid:107) π (cid:63) ( ·| s ) − π i ( ·| s ) (cid:107) (cid:105) =|

Original: [Equation]
Revised: E | | E | (cid:88) i = J BC ( π i ) ≤ (cid:15)

Original: The last step follows from our optimization oracle assumption:
Revised: The last step follows from our optimization oracle assumption:

Original: 0 ≤ min π ∈ Π J BC ( π ) ≤ J BC ( π (cid:63) ) = 0 , hence J BC ( π i ) ≤ 0 + (cid:15) =
Revised: 0 ≤ min π ∈ Π J BC ( π ) ≤ J BC ( π (cid:63) ) = 0 , hence J BC ( π i ) ≤ 0 + (cid:15) =

Original: (cid:15) .
Revised: (cid:15) .

Original: Combining the bounds on the two terms, we get J alg ( π (cid:63) ) ≤ 2 (cid:15) .
Revised: Combining the bounds on the two terms, we get J alg ( π (cid:63) ) ≤ 2 (cid:15) .

Original: Since π (cid:63) ∈ Π , the result follows.
Revised: Since π (cid:63) ∈ Π , the result follows.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ˆ π be the result of minimizing J alg using our optimization oracle, and assume that
Revised: Let ˆ π be the result of minimizing J alg using our optimization oracle, and assume that

Original: Q π (cid:63) T − t +1 ( s, a ) − Q π (cid:63) T − t +1 ( s, π (cid:63) ) ≤ u for all a ∈ A , t ∈ { 1 , 2 , ..., T } , d tπ ( s ) > 0 .
Revised: Q π (cid:63) T − t +1 ( s, a ) − Q π (cid:63) T − t +1 ( s, π (cid:63) ) ≤ u for all a ∈ A , t ∈ { 1 , 2 , ..., T } , d tπ ( s ) > 0 .

Original: Then ˆ π satisﬁes
Revised: Then ˆ π satisﬁes

Original: J (ˆ π ) ≤ J ( π (cid:63) ) + 3 uκ(cid:15)T .
Revised: J (ˆ π ) ≤ J ( π (cid:63) ) + 3 uκ(cid:15)T .

Original: Proof.
Revised: Proof.

Original: By our optimization oracle and Lemma 2, we have
Revised: By our optimization oracle and Lemma 2, we have

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma 1, we get:
Revised: Combining with Lemma 1, we get:

Original: [Equation]
Revised: [Equation]

Original: Applying Theorem 1 from (Ross et al., 2011), we get J (ˆ π ) ≤ J ( π (cid:63) ) + 3 uκ(cid:15)T .
Revised: Applying Theorem 1 from (Ross et al., 2011), we get J (ˆ π ) ≤ J ( π (cid:63) ) + 3 uκ(cid:15)T .

Original: B I MPORTANCE OF B EHAVIOR C LONING C OST
Revised: B I MPORTANCE OF B EHAVIOR C LONING C OST

Original: The following example shows how minimizing the uncertainty cost alone without the BC cost can lead to highly sub-optimal policies if the demonstration data is generated by a stochastic policy which is only slightly suboptimal.
Revised: The following example shows how minimizing the uncertainty cost alone without the BC cost can lead to highly sub-optimal policies if the demonstration data is generated by a stochastic policy which is only slightly suboptimal.

Original: Consider the following deterministic chain MDP:
Revised: Consider the following deterministic chain MDP:

Original: The agent always starts in s 1 , and gets a reward of 1 in s 3 and 0 elsewhere.
Revised: The agent always starts in s 1 , and gets a reward of 1 in s 3 and 0 elsewhere.

Original: The optimal policy is given by:
Revised: The optimal policy is given by:

Original: [Equation]
Revised: [Equation]

Original: Assume the demonstration data is generated by the following policy, which is only slightly suboptimal:
Revised: Assume the demonstration data is generated by the following policy, which is only slightly suboptimal:

Original: [Equation]
Revised: [Equation]

Original: Let us assume realizability and perfect optimization for simplicity.
Revised: Let us assume realizability and perfect optimization for simplicity.

Original: If both transitions ( s 2 , a 0 ) and ( s 2 , a 1 ) appear in the demonstration data, then Random Expert Distillation (RED) will assign zero cost to both transitions.
Revised: If both transitions ( s 2 , a 0 ) and ( s 2 , a 1 ) appear in the demonstration data, then Random Expert Distillation (RED) will assign zero cost to both transitions.

Original: If we do not use bootstrapped samples to train the ensemble, then DRIL without the BC cost (we will call this UO-DRIL for Uncertainty-Only DRIL) will also assign zero cost to both transitions since all models in the ensemble would recover the Bayes optimal solution given the demonstration data.
Revised: If we do not use bootstrapped samples to train the ensemble, then DRIL without the BC cost (we will call this UO-DRIL for Uncertainty-Only DRIL) will also assign zero cost to both transitions since all models in the ensemble would recover the Bayes optimal solution given the demonstration data.

Original: If we are using bootstrapped samples, then the Bayes optimal solution for each bootstrapped sample may differ and thus the different policies in the ensemble might disagree in their predictions, although given enough demonstration data we would expect these differences (and thus the uncertainty cost) to be small.
Revised: If we are using bootstrapped samples, then the Bayes optimal solution for each bootstrapped sample may differ and thus the different policies in the ensemble might disagree in their predictions, although given enough demonstration data we would expect these differences (and thus the uncertainty cost) to be small.

Original: Note also that since no samples at the state s 0 occur in the demonstration data, both RED and UODRIL will likely assign high uncertainty costs to state-action pairs at ( s 0 , a 0 ) , ( s 0 , a 1 ) and thus avoid highly suboptimal policies which get stuck at s 0 .
Revised: Note also that since no samples at the state s 0 occur in the demonstration data, both RED and UODRIL will likely assign high uncertainty costs to state-action pairs at ( s 0 , a 0 ) , ( s 0 , a 1 ) and thus avoid highly suboptimal policies which get stuck at s 0 .

Original: Now consider policies ˆ π 1 , ˆ π 2 given by:
Revised: Now consider policies ˆ π 1 , ˆ π 2 given by:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Both of these policies only visit state-action pairs which are visited by the demonstration policy.
Revised: Both of these policies only visit state-action pairs which are visited by the demonstration policy.

Original: In the case described above, both RED and UO-DRIL will assign ˆ π 1 and ˆ π 2 similarly low costs.
Revised: In the case described above, both RED and UO-DRIL will assign ˆ π 1 and ˆ π 2 similarly low costs.

Original: However, ˆ π 1 will cycle forever between s 1 and s 2 , never collecting reward, while ˆ π 2 will with high probability reach s 3 and stay there, thus achieving high reward.
Revised: However, ˆ π 1 will cycle forever between s 1 and s 2 , never collecting reward, while ˆ π 2 will with high probability reach s 3 and stay there, thus achieving high reward.

Original: This shows that minimizing the uncertainty cost alone does not necessarily distinguish between good and bad policies.
Revised: This shows that minimizing the uncertainty cost alone does not necessarily distinguish between good and bad policies.

Original: However, ˆ πwill incur a higher BC cost than ˆ π 2 , since ˆ π 2 more closely matches the demonstration data at s 2 .
Revised: However, ˆ πwill incur a higher BC cost than ˆ π 2 , since ˆ π 2 more closely matches the demonstration data at s 2 .

Original: This shows that including the BC cost can be important for further disambiguating between policies which all stay within the distribution of the demonstration data, but have different behavior within that distribution.
Revised: This shows that including the BC cost can be important for further disambiguating between policies which all stay within the distribution of the demonstration data, but have different behavior within that distribution.

Original: C E XPERIMENTAL D ETAILS C. A TARI E NVIRONMENTS
Revised: C E XPERIMENTAL D ETAILS C. A TARI E NVIRONMENTS

Original: All behavior cloningand ensemble models were trained to minimize the negative log-likelihood classiﬁcation loss on the demonstration data for 500 epochs using Adam (Kingma & Ba, 2014) and a learning rate of 2 . 5 · 10 − 4 .
Revised: All behavior cloning models were trained to minimize the negative log-likelihood classiﬁcation loss on the demonstration data for 500 epochs using Adam (Kingma & Ba, 2014) and a learning rate of 2 . 5 · 10 − 4 .

Revised: We stopped training once the validation error did not improve for 20 epochs.

Original: For our method, we initially performed a hyperparameter search on Space Invaders over the following values:
Revised: For our method, we initially performed a hyperparameter search on Space Invaders over the values shown in Table

Revised: [Table]

Original: We then chose the best values and kept those hyperparameters ﬁxed for all other environments.
Revised: We then chose the best values and kept those hyperparameters ﬁxed for all other environments.

Original: All other A2C hyperparameters follow the default values in the repo (Kostrikov, 2018): policy networks consisted of 3-layer convolutional networks with 8 − 32 − 64 feature maps followed by a single-layer
Revised: All other A2C hyperparameters follow the default values in the repo (Kostrikov, 2018): policy networks consisted of 3-layer convolutional networks with 8 − 32 − 64 feature maps followed by a single-layer

Original: MLP with 512 hidden units.
Revised: MLP with 512 hidden units.

Original: For GAIL, we used the implementation in (Kostrikov, 2018) and replaced the MLP discriminator by a CNN discriminator with the same architecture as the policy network.
Revised: For GAIL [Table] we used the implementation in (Kostrikov, 2018) and replaced the MLP discriminator by a CNN discriminator with the same architecture as the policy network.

Original: We initially performed a hyperparameter search on Breakout with 10 demonstrations over the values shown in Table 2.
Revised: We initially performed a hyperparameter search on Breakout with 10 demonstrations over the values shown in Table 2.

Original: However, we did not ﬁnd any hyperparameter conﬁguration which performed better than behavioral cloning.[Table]
Revised: However, we did not ﬁnd any hyperparameter conﬁguration which performed better than behavioral cloning.

Original: [Table]
Revised: [Table]

Original: C. C ONTINUOUS C ONTROL
Revised: . C ONTINUOUS C ONTROL

Original: All behavior cloning and ensemble models were trained to minimize the mean-squared error regression loss on the demonstration data for 500 epochs using Adam (Kingma & Ba, 2014) and a learning rate of 2 . 5 · 10 − 4 .
Revised: All behavior cloning and ensemble models were trained to minimize the mean-squared error regression loss on the demonstration data for 500 epochs using Adam (Kingma & Ba, 2014) and a learning rate of 2 . 5 · 10 − 4 .

Original: Policy networks were 2-layer fully-connected MLPs with tanh activations andhidden units.
Revised: Policy networks were 2-layer fully-connected MLPs with tanh activations andhidden units.

Original: [Table]
Revised: [Table]

Original: D A BLATION E XPERIMENTSD. C OST T ERMS
Revised: D A BLATION E XPERIMENTS

Original: Here we provide ablation experiments showing the effect of different cost function choices. We compare the following variants:
Revised: In this section we provide ablation experiments examining the effects of the cost function clipping and the role of the BC loss.

Original: • DRIL This is the regular DRIL agent, which optimizes both the BC cost and the clipped cost: [Equation] • DRIL (clipped cost 0/1) This is the same as the regular DRIL agent, except that we use the following clipped cost function:
Revised: We also compare the ensemble approach to a dropout-based approximation and show that DRIL works well in both cases.

Original: [Figure]
Revised: [Table]

Original: [Equation]

Original: • DRIL (raw cost)

Original: This is the same as the regular DRIL agent, except that we use the raw cost C U ( s, a ) rather than the clipped cost.

Original: • DRIL (no BC cost)

Original: This is the same as the regular DRIL agent, except that we remove the BC updates and only optimize C clipU ( s, a ) .

Original: Results are shown in Figure 5.
Revised: Results are shown in Figure 4.

Original: First, switching from the clipped cost in {− 1 , +1 } to theclipped cost in { 0 , 1 } or the raw cost causes alarge drop in performance across most environments.
Revised: First, switching from the clipped cost in {− 1 , +1 } to the the raw cost causes a drop in performance.

Original: One explanation may be that since the  costs are always positivefor both variants (which corresponds to a reward which is always negative), the agent may learn to terminate the episode early in order to minimize the total cost incurred.
Revised: One explanation may be that since the raw costs are always positive (which corresponds to a reward which is always negative), the agent may learn to terminate the episode early in order to minimize the total cost incurred.

Original: Using a cost/reward which has both positive and negative values avoids this behavior.
Revised: Using a cost/reward which has both positive and negative values avoids this behavior.

Original: Second, optimizing the pure BC cost performs better than the pure uncertainty cost for some environments (MsPacman, SpaceInvaders, BeamRider) while optimizing the pure uncertainty cost performs better than BC for others (Breakout, Qbert).
Revised: Second, optimizing the pure BC cost performs better than the pure uncertainty cost for some environments (SpaceInvaders, BeamRider) while optimizing the pure uncertainty cost performs better than BC in Breakout.

Original: DRIL, which optimizes both, has robust performance and performs the bestout of the different variants over most environments and numbers of trajectories.
Revised: DRIL, which optimizes both, has robust performance and performs the best over all environments.

Original: D. U NCERTAINTY E STIMATION C OMPARISON

Original: We provide additional results comparing the ensembling and MC-dropout (Gal & Ghahramani, 2016) approaches to posterior estimation.

Original: For MC-Dropout we trained a single policy network with a dropout rate of 0 .
Revised: For the dropout approximation we trained a single policy network with a dropout rate of 0 .

Original: 1 applied to all layers except the last, and estimated the variance for each state-action pair using 10 different dropout masks.
Revised: 1 applied to all layers except the last, and estimated the variance for each state-action pair using 5 different dropout masks.

Original: Similarly to the ensemble approach, we computed the 98 th quantile of the variance on the demonstration data and used this value in our clipped cost.
Revised: Similarly to the ensemble approach, we computed the 98 th quantile of the variance on the demonstration data and used this value in our clipped cost.

Original: Results are shown for three environments in Figure 6.

Original: MC-dropout performs similarly to the ensembling approach, which shows that our method can be paired with different approaches to posterior estimation.[Figure]
Revised: MC-dropout performs similarly to the ensembling approach, which shows that our method can be paired with different approaches to posterior estimation.