Original: Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Revised: Robust Imitation via Mirror Descent Inverse Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recently, adversarial imitation learning has shown a scalable reward acquisitionmethod for inverse reinforcement learning (IRL) problems.
Revised: Recently, adversarial imitation learning has shown a scalable reward acquisitionmethod for inverse reinforcement learning (IRL) problems.

Original: However, estimatedreward signals often become uncertain and fail to train a reliable statistical modelsince the existing methods tend to solve hard optimization problems directly.
Revised: However, estimatedreward signals often become uncertain and fail to train a reliable statistical modelsince the existing methods tend to solve hard optimization problems directly.

Original: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions fora constrained convex problem.
Revised: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions fora constrained convex problem.

Original: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount ofreward learning is regulated with respect to local geometric constraints.
Revised: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount ofreward learning is regulated with respect to local geometric constraints.

Original: We provethat the proposed mirror descent update rule ensures robust minimization of a
Revised: We provethat the proposed mirror descent update rule ensures robust minimization of a

Original: Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes{ η t } Tt =1 .
Revised: Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes{ η t } Tt =1 .

Original: Our IRL method was applied on top of an adversarial framework, and itoutperformed existing adversarial methods in an extensive suite of benchmarks.
Revised: Our IRL method was applied on top of an adversarial framework, and itoutperformed existing adversarial methods in an extensive suite of benchmarks.

Original:  Introduction
Revised:  Introduction

Original: One crucial requirement of practical imitation learning algorithms is robustness , often describedas learning expert behavior fro ﬁnite number of demonstrations, overcoming various realistic challenges [1].
Revised: One crucial requirement of practical imitation learning algorithms is robustness , often describedas learning expert behavior for ﬁnite number of demonstrations, overcoming various realistic chalNEW [ Ptuv ]lenges [1].

Original: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations canbe noisy or suboptimal to solve the problem [3].
Revised: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations canbe noisy or suboptimal to solve the problem [3].

Original: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling theuncertainty of estimated cost functions dramatically affects the imitation performance.
Revised: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling theuncertainty of estimated cost functions dramatically affects the imitation performance.

Original: Therefore, athorough analysis of addressing these issues is required to construct a robust algorithm.
Revised: Therefore, athorough analysis of addressing these issues is required to construct a robust algorithm.

Original: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expertdemonstrations where the expert acts optimally with respect to an unknown reward function [4].
Revised: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expertdemonstrations where the expert acts optimally with respect to an unknown reward function [4].

Original: Traditional IRL methods solve the imitation problem based on iterative algorithms [5, 6], alternatingbetween the reward estimation process and a reinforcement learning (RL) [7] algorithm.
Revised: Traditional IRL methods solve the imitation problem based on iterative algorithms [5, 6], alternatingbetween the reward estimation process and a reinforcement learning (RL) [7] algorithm.

Original: Newerstudies of adversarial imitation learning (AIL) [8, 9] rather suggest learning reward functions of a
Revised: Newerstudies of adversarial imitation learning (AIL) [8, 9] rather suggest learning reward functions of a

Original: FIX [ Mged ]certain form “directly,” by using adversarial learning objectives [10] and nonlinear discriminativeneural networks [9].
Revised: FIX [ Mged ]certain form “directly,” by using adversarial learning objectives [10] and nonlinear discriminativeneural networks [9].

Original: Compared to classical approaches, the AIL methods have shown great successon control benchmarks in terms of scalability for challenging control tasks [11].
Revised: Compared to classical approaches, the AIL methods have shown great successon control benchmarks in terms of scalability for challenging control tasks [11].

Original: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [12].
Revised: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [12].

Original: Throughthe lens of differential geometries, the limitation of AIL naturally comes from the implication thatminimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [13].
Revised: Throughthe lens of differential geometries, the limitation of AIL naturally comes from the implication thatminimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [13].

Original: In order to ensure stability, we argue that an IRL algorithm’s progress needs to beregulated, yielding gradual updates with respect to local geometries of policy distributions
Revised: In order to ensure stability, we argue that an IRL algorithm’s progress needs to beregulated, yielding gradual updates with respect to local geometries of policy distributions

Original: We claim that there are two issues leading to unconstrained policy updates: (1) a statistical divergence often cannot be accurately obtained for challenging problems and (2) an immediate divergencebetween agent and expert densities does not guarantee unbiased learning directions.
Revised: We claim that there are two issues leading to unconstrained policy updates: (1) a statistical divergence often cannot be accurately obtained for challenging problems and (2) an immediate divergencebetween agent and expert densities does not guarantee unbiased learning directions.

Original: Our approach isconnected to a collection of optimization processes called mirror descent (MD) [14].
Revised: Our approach isconnected to a collection of optimization processes called mirror descent (MD) [14].

Original: For a sequenceof parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as
Revised: For a sequenceof parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as

Original: [Equation]
Revised: [Equation]

Original: In the equation, the gradient ∇ Ω ( · ) creates a transformation that links a parametric space to its dualspace.
Revised: In the equation, the gradient ∇ Ω ( · ) creates a transformation that links a parametric space to its dualspace.

Original: Theoretically, MD is a ﬁrst-order method for solving constrained problems which enjoysrigorous regret bounds for various geometries [15, 16], including probability spaces.
Revised: Theoretically, MD is a ﬁrst-order method for solving constrained problems which enjoysrigorous regret bounds for various geometries [15, 16], including probability spaces.

Original: Thus, applying
Revised: Thus, applying

Original: MD to the reward estimation can be efﬁcient in terms of the number of learning phases.
Revised: MD to the reward estimation can be efﬁcient in terms of the number of learning phases.

Original: In this paper, we derive an MD update rule in IRL with a postulate of nonstationary estimations ofthe expert density, resulting to convergent reward acquisition even for challenging problems.
Revised: In this paper, we derive an MD update rule in IRL with a postulate of nonstationary estimations ofthe expert density, resulting to convergent reward acquisition even for challenging problems.

Original: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions onan alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).
Revised: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions onan alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).

Original: To this end, we propose an AILalgorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).
Revised: To this end, we propose an AILalgorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).

Original: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)
Revised: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)

Original: [17]methods.
Revised: [17]methods.

Original: For example, MD-AIRL showed higher performance in 30 distinct cases among 32 different conﬁgurations in challenging MuJoCo [18] benchmarks, and it clearly showed higher toleranceto suboptimal data.
Revised: For example, MD-AIRL showed higher performance in 30 distinct cases among 32 different conﬁgurations in challenging MuJoCo [18] benchmarks, and it clearly showed higher toleranceto suboptimal data.

Original: All of these results are strongly aligned with our theoretical results.
Revised: All of these results are strongly aligned with our theoretical results.

Original: [Table]
Revised: [Table]

Original: Rewards Bregman divergence Iterative
Revised: Rewards Bregman divergence Iterative

Original: [Table]
Revised: [Table]

Original: Our contributions.
Revised: Our contributions.

Original: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithmsas a combined optimization process with dual aspects.
Revised: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithmsas a combined optimization process with dual aspects.

Original: Comparing MD-AIRL with RAIRL, both arehighly generalized algorithms in terms of a variety of choices of divergence functions.
Revised: Comparing MD-AIRL with RAIRL, both arehighly generalized algorithms in terms of a variety of choices of divergence functions.

Original: Tab. 1 showsthat MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since ourapproach is more aligned with earlier theoretical IRL studies providing formalized reward learningschemes and convergence guarantees.
Revised: Tab. 1 showsthat MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since ourapproach is more aligned with earlier theoretical IRL studies providing formalized reward learningschemes and convergence guarantees.

Original: In summary, we list our main contributions below:
Revised: In summary, we list our main contributions below:

Original: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence ofreward functions that provides iterative local objectives (Section 4).• We formally prove that rewards derived by an MD update rule guarantee robust performance ofdivergence minimization along with a rigorous regret bound (Section 5).•
Revised: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence ofreward functions that provides iterative local objectives (Section 4).• We formally prove that rewards derived by an MD update rule guarantee robust performance ofdivergence minimization along with a rigorous regret bound (Section 5).•

Original: We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant tounreliable discriminative signals of the AIL framework (Sections 6 and 7).
Revised: We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant tounreliable discriminative signals of the AIL framework (Sections 6 and 7).

Original:  Related Works
Revised:  Related Works

Original: Mirror descent.
Revised: Mirror descent.

Original: We are interested in a family of statistical divergence called the Bregman divergence [20].
Revised: We are interested in a family of statistical divergence called the Bregman divergence [20].

Original: The divergence generalizes constrained optimization problems such as least squares[21, 22], and it also has been applied in various subﬁelds of machine learning [23, 24].
Revised: The divergence generalizes constrained optimization problems such as least squares[21, 22], and it also has been applied in various subﬁelds of machine learning [23, 24].

Original: In differentialgeometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁesmetric-like properties [13, 25].
Revised: In differentialgeometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁesmetric-like properties [13, 25].

Original: MD is also closely related to optimization methods regarding nonFIX [ Mged ]
Revised: MD is also closely related to optimization methods regarding nonFIX [ Mged ]

Original: Euclidean geometries with a discretization of steps such as natural gradients [26, 27].
Revised: Euclidean geometries with a discretization of steps such as natural gradients [26, 27].

Original: In the primalspace, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow[28, 29].
Revised: In the primalspace, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow[28, 29].

Original: In the RL domain, MD has been recently studied for policy optimization [30–32].
Revised: In the RL domain, MD has been recently studied for policy optimization [30–32].

Original: In thispaper, we focus on learning with suboptimal representations of policy, and our distinct goal is todraw a robust reward learning scheme based on MD for the IRL problem.
Revised: In thispaper, we focus on learning with suboptimal representations of policy, and our distinct goal is todraw a robust reward learning scheme based on MD for the IRL problem.

Original: Imitation learning.
Revised: Imitation learning.

Original: As a statistical model for the information geometry [33], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient IRL, and maximum likelihood IRL [34–36] for modeling expert distribution to parameterized functions.
Revised: As a statistical model for the information geometry [33], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient IRL, and maximum likelihood IRL [34–36] for modeling expert distribution to parameterized functions.

Original: Notably, MaxEnt IRL [6, 37] is one of the representative classical IRL algorithmsbased on an information-theoretic perspective toward IRL solutions.
Revised: Notably, MaxEnt IRL [6, 37] is one of the representative classical IRL algorithmsbased on an information-theoretic perspective toward IRL solutions.

Original: Also, discriminators of AIL aretrained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy functionthat approximates the truth data density for the expert distribution [38].
Revised: Also, discriminators of AIL aretrained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy functionthat approximates the truth data density for the expert distribution [38].

Original: Other statistical entropieshave also been applied to AIL, such as the Tsallis entropy [39].
Revised: Other statistical entropieshave also been applied to AIL, such as the Tsallis entropy [39].

Original: On the one hand, our approach isclosely related to RAIRL [17], which deﬁned its objective as minimizing the Bregman divergence.
Revised: On the one hand, our approach isclosely related to RAIRL [17], which deﬁned its objective as minimizing the Bregman divergence.

Original: On the other hand, this work further employs the divergence to derive iterative MD updates for reward functions, resulting to theoretically pleasing properties while retaining the scalability of AIL.
Revised: On the other hand, this work further employs the divergence to derive iterative MD updates for reward functions, resulting to theoretically pleasing properties while retaining the scalability of AIL.

Original: Learning theory.
Revised: Learning theory.

Original: There have been considerable achievements in dealing with temporal costs{ F t } ∞ t =1 , often referred to as online learning [40].
Revised: There have been considerable achievements in dealing with temporal costs{ F t } ∞ t =1 , often referred to as online learning [40].

Original: The most ordinary approach is stochastic gradi- 91 ent descent (SGD): w t +1 = w t − η t ∇ F t ( w t ) .
Revised: The most ordinary approach is stochastic gradi- 91 ent descent (SGD): w t +1 = w t − η t ∇ F t ( w t ) .

Original: In particular, SGD is a desirable algorithm when theparameter w t resides in the Euclidean space since it ensures unbiased minimization of the expectedcost.
Revised: In particular, SGD is a desirable algorithm when theparameter w t resides in the Euclidean space since it ensures unbiased minimization of the expectedcost.

Original: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may notbe the direction of the steepest descent due to geometric constraints [26, 33].
Revised: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may notbe the direction of the steepest descent due to geometric constraints [26, 33].

Original: An online form of MD(Eq.
Revised: An online form of MD(Eq.

Original: (1)) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a
Revised: (1)) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a

Original: Bregman divergence [29].
Revised: Bregman divergence [29].

Original: Our theoretical ﬁndings and proofs follow the results of online mirrordescent (OMD) that appeared in previous literature for general aspects [14, 15, 41, 27, 16, 29].
Revised: Our theoretical ﬁndings and proofs follow the results of online mirrordescent (OMD) that appeared in previous literature for general aspects [14, 15, 41, 27, 16, 29].

Original: Ouranalyses extend existing theoretical results to IRL; at the same time, they are also highly general tocover various online imitation learning problems which require making decision sequentially.
Revised: Ouranalyses extend existing theoretical results to IRL; at the same time, they are also highly general tocover various online imitation learning problems which require making decision sequentially.

Original:  Background
Revised:  Background

Original: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional)probabilities over X (conditioned on Y ).
Revised: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional)probabilities over X (conditioned on Y ).

Original: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ )with the state space S , the action space A ,
Revised: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ )with the state space S , the action space A ,

Original: the Markovian transition kernel P ∈ ∆ S × A S , the rewardfunction r ∈ R S
Revised: the Markovian transition kernel P ∈ ∆ S × A S , the rewardfunction r ∈ R S

Original: × A and the discount factor γ ∈ [0 , 1) .
Revised: × A and the discount factor γ ∈ [0 , 1) .

Original: Let a function Ω : ∆ A → R be stronglyconvex.
Revised: Let a function Ω : ∆ A → R be stronglyconvex.

Original: Using Ω , the Bregman divergence is deﬁned as
Revised: Using Ω , the Bregman divergence is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where π s and ˆ π denote arbitrary policies for a given state s .
Revised: where π s and ˆ π denote arbitrary policies for a given state s .

Original: For a representative divergence, onecan consider the popular KL divergence.
Revised: For a representative divergence, onecan consider the popular KL divergence.

Original: The KL divergence is a Bregman divergence when Ω isspeciﬁed as the negative Shannon entropy: Ω ( π s
Revised: The KL divergence is a Bregman divergence when Ω isspeciﬁed as the negative Shannon entropy: Ω ( π s

Original: [Equation]
Revised: [Equation]

Original: Regularization of policy distribution with respect to convex Ω brings distinct properties for learningagent [42, 43].
Revised: Regularization of policy distribution with respect to convex Ω brings distinct properties for learningagent [42, 43].

Original: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected valueof discounted cumulative returns along with a causal convex regularizer Ω : ∆ A → R , i.e.
Revised: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected valueof discounted cumulative returns along with a causal convex regularizer Ω : ∆ A → R , i.e.

Original: [Equation]
Revised: [Equation]

Original: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for thegiven MDP.
Revised: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for thegiven MDP.

Original: In this setup, a trainable policy space is a subset of the entire conditional probabilityspace denoted as Π :=[ Π s ] s ∈ S ⊂ ∆ S A , which is constrained by the parameterization of a policy.
Revised: In this setup, a trainable policy space is a subset of the entire conditional probabilityspace denoted as Π :=[ Π s ] s ∈ S ⊂ ∆ S A , which is constrained by the parameterization of a policy.

Original: The objective of IRL is to ﬁnd a function r E that rationalizes the expert behavior π E .
Revised: The objective of IRL is to ﬁnd a function r E that rationalizes the expert behavior π E .

Original: For an innerproduct 〈 · , · 〉 A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :
Revised: For an innerproduct 〈 · , · 〉 A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :

Original: [Equation]
Revised: [Equation]

Original: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .
Revised: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .

Original: Differentiating both sides withrespect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.
Revised: Differentiating both sides withrespect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.

Original: One fundamentalproperty in regularized IRL [42] is that π E is the maximizing argument of Ω ∗ for q E , where q E isthe regularized state-action value function q E ( s, a ) =
Revised: One fundamentalproperty in regularized IRL [42] is that π E is the maximizing argument of Ω ∗ for q E , where q E isthe regularized state-action value function q E ( s, a ) =

Original: E π E [ %
Revised: E π E [ %

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: { r E ( s, a ) − Ω ( π s E ) }| s 

Revised: { r E ( s, a ) − Ω ( π s E ) }| s

Original: = s, a 0 = 121 a ] .
Revised: = s, a 0 = 121 a ] .

Original: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E =∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.
Revised: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E =∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.

Original: Addressing this issue, Jeon et al.
Revised: Addressing this issue, Jeon et al.

Original: [17] proposed a rewardoperator Ψ Ω : ∆ SA → R S × A , providing a unique IRL solution by Ψ Ω ( π E ) .
Revised: [17] proposed a rewardoperator Ψ Ω : ∆ SA → R S × A , providing a unique IRL solution by Ψ Ω ( π E ) .

Original: Deﬁnition 1 (Regularized reward operators) .
Revised: Deﬁnition 1 (Regularized reward operators) .

Original: Deﬁne the regularized reward operator Ψ Ω asψ π ( s, a ) :
Revised: Deﬁne the regularized reward operator Ψ Ω asψ π ( s, a ) :

Original: =
Revised: =

Original: Ω ′ ( s, a ; π ) − # π s , ∇ Ω ( π s ) $
Revised: Ω ′ ( s, a ; π ) − # π s , ∇ Ω ( π s ) $

Original: A + Ω ( π s ) , for Ω ′ ( s, · ; π ) :
Revised: A + Ω ( π s ) , for Ω ′ ( s, · ; π ) :

Original: = ∇ Ω ( π s ) =
Revised: = ∇ Ω ( π s ) =

Original: + ∇ p Ω ( p ) , p = π ( ·| s ) .
Revised: + ∇ p Ω ( p ) , p = π ( ·| s ) .

Original: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite Bregman divergences derived from Eq.
Revised: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite Bregman divergences derived from Eq.

Original: (2) allows reward learning in a greedy manner [17].
Revised: (2) allows reward learning in a greedy manner [17].

Original: P o li c y S p ace Π π t π t +˜ π t +
Revised: P o li c y S p ace Π π t π t +˜ π t +

Original: P Ω
Revised: P Ω

Original: R e w a r d S p ace ψ t ψ t +
Revised: R e w a r d S p ace ψ t ψ t +

Original: R S ×
Revised: R S ×

Original: A Ψ Ω ∇ Ω ∗
Revised: A Ψ Ω ∇ Ω ∗

Original: Associated reward functions.
Revised: Associated reward functions.

Original: We consider the
Revised: We consider the

Original: RL-IRL processes as a sequential algorithm withlocal constraints and deﬁne sequences { π t } ∞ t =1 132 and { ψ t } ∞ t =1 that denote policies and associated 133 reward functions, respectively.
Revised: RL-IRL processes as a sequential algorithm withlocal constraints and deﬁne sequences { π t } ∞ t =1 132 and { ψ t } ∞ t =1 that denote policies and associated 133 reward functions, respectively.

Original: The associated reward functions is in a space Ψ Ω ( Π ) , which is analternative space of the dual space, deﬁned by theregularized reward operator Ψ Ω .
Revised: The associated reward functions is in a space Ψ Ω ( Π ) , which is analternative space of the dual space, deﬁned by theregularized reward operator Ψ Ω .

Original: Formally, we provide Lemma 1 which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .
Revised: Formally, we provide Lemma 1 which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .

Original: Theproof is in Appendix A.
Revised: Theproof is in Appendix A.

Original: Lemma 1 (Natural isomorphism) .
Revised: Lemma 1 (Natural isomorphism) .

Original: Let ψ ∈ Ψ Ω ( Π )for Ψ Ω ( X ) :
Revised: Let ψ ∈ Ψ Ω ( Π )for Ψ Ω ( X ) :

Original: = { ψ | ψ ( s, a ) =
Revised: = { ψ | ψ ( s, a ) =

Original: ψ π ( s, a ) , ∀ s ∈
Revised: ψ π ( s, a ) , ∀ s ∈

Original: S , a ∈ A , π ∈ X } .
Revised: S , a ∈ A , π ∈ X } .

Original: Then, ∇ Ω ∗ ( ψ ) is unique and 143 for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .
Revised: Then, ∇ Ω ∗ ( ψ ) is unique and 143 for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .

Original: Fig.
Revised: Fig.

Original: 1 illustrates that there is unique ψ t for π t in every time step.
Revised: 1 illustrates that there is unique ψ t for π t in every time step.

Original: Note that Ψ Ω ( π t ) is differentfrom ∇ Ω ( π t ) ; it is shifted by a vector 1 c with a constant c = Ω ( π st ) − 〈 π st , ∇ Ω ( π st ) 〉
Revised: Note that Ψ Ω ( π t ) is differentfrom ∇ Ω ( π t ) ; it is shifted by a vector 1 c with a constant c = Ω ( π st ) − 〈 π st , ∇ Ω ( π st ) 〉

Original: A .
Revised: A .

Original: Since theunderlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for bothΨ Ω and ∇ Ω , as the distributivity [42]
Revised: Since theunderlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for bothΨ Ω and ∇ Ω , as the distributivity [42]

Original: Ω ∗ ( y + 1 c ) =
Revised: Ω ∗ ( y + 1 c ) =

Original: Ω ∗ ( y )+ c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).
Revised: Ω ∗ ( y )+ c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).

Original: Analternative interpretation is of considering a projection (gray dashed line in Fig. 1).
Revised: Analternative interpretation is of considering a projection (gray dashed line in Fig. 1).

Original: Suppose that apolicy π t is updated to ˜ π t +1 ∈ R S × A .
Revised: Suppose that apolicy π t is updated to ˜ π t +1 ∈ R S × A .

Original: The Bregman projection operator P Ω is applied that locates the 150 subsequent update π t
Revised: The Bregman projection operator P Ω is applied that locates the 150 subsequent update π t

Original: [Equation]
Revised: [Equation]

Original: 1 to the “feasible” region, i.e.
Revised: 1 to the “feasible” region, i.e.

Original: [Equation]
Revised: [Equation]

Original: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .
Revised: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .

Original: For instance, the parameters of ψ t can construct asoftmax policy for a discrete space, or a Gaussian policy for a continuous space.
Revised: For instance, the parameters of ψ t can construct asoftmax policy for a discrete space, or a Gaussian policy for a continuous space.

Original: Using the rewardfunction ψ t +1 , an arbitrary regularized RL process maximizing Eq.
Revised: Using the rewardfunction ψ t +1 , an arbitrary regularized RL process maximizing Eq.

Original: (2) at the t -th step [17]
Revised: (2) at the t -th step [17]

Original: [Equation]
Revised: [Equation]

Original: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the RL objective.
Revised: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the RL objective.

Original: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.
Revised: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.

Original: Online imitation learning.
Revised: Online imitation learning.

Original: Our setup comes from an observation that imitation learning do notretain the global target π E during training.
Revised: Our setup comes from an observation that imitation learning do notretain the global target π E during training.

Original: That is, it is fundamentally uncertain to model globalobjectives (such as J Ω ( π , ψ E ) ) which are not attainable for both RL and IRL.
Revised: That is, it is fundamentally uncertain to model globalobjectives (such as J Ω ( π , ψ E ) ) which are not attainable for both RL and IRL.

Original: Instead, we hypothesize on existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex 162 neighborhood of π
Revised: Instead, we hypothesize on existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex 162 neighborhood of π

Original: E , generated by an arbitrary estimation algorithm.
Revised: E , generated by an arbitrary estimation algorithm.

Original: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) , J Ω ( π , ψ ¯ π E,t ) forms a temporal costs F t :
Revised: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) , J Ω ( π , ψ ¯ π E,t ) forms a temporal costs F t :

Original: [Equation]
Revised: [Equation]

Original: For a better understanding, we consider an actual experiment in Fig.
Revised: For a better understanding, we consider an actual experiment in Fig.

Original: Suppose that policiesof the learning agent and the expert follow multivariate Gaussian distributions at N ( [0 , 0] T , I) and
Revised: Suppose that policiesof the learning agent and the expert follow multivariate Gaussian distributions at N ( [0 , 0] T , I) and

Original: FIX [ Mged ]
Revised: FIX [ Mged ]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: N ( [5 , 3] T , Σ E ) with | Σ E | < 1 .
Revised: N ( [5 , 3] T , Σ E ) with | Σ E | < 1 .

Original: Suppose a (suboptimal) reference policy ¯ π E ,t , independently ﬁtted witha maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .
Revised: Suppose a (suboptimal) reference policy ¯ π E ,t , independently ﬁtted witha maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .

Original: Thepolicy π t was trained by a cost function D Ω ( · % ¯ π E ,t ) using the MD update rule in Eq.
Revised: Thepolicy π t was trained by a cost function D Ω ( · % ¯ π E ,t ) using the MD update rule in Eq.

Original: In Fig.
Revised: In Fig.

Original: 2,we
Revised: 2,we

Original: ﬁrst observed that choosing high step size constant η accelerates the training speed mostly in theearly phase.
Revised: ﬁrst observed that choosing high step size constant η accelerates the training speed mostly in theearly phase.

Original: The results also showed that the performance of MD ( D Ω ( π t % π E ) ) outperformed thatof referenced maximum likelihood estimation ( D Ω (¯ π E ,
Revised: The results also showed that the performance of MD ( D Ω ( π t % π E ) ) outperformed thatof referenced maximum likelihood estimation ( D Ω (¯ π E ,

Original: t % π E ) ) by choosing an effectively low stepsize.
Revised: t % π E ) ) by choosing an effectively low stepsize.

Original: This suggests that there are clear advantages in scheduling step sizes by time steps especiallyfor unreliable ¯ π E ,t .
Revised: This suggests that there are clear advantages in scheduling step sizes by time steps especiallyfor unreliable ¯ π E ,t .

Original: MD update rules.
Revised: MD update rules.

Original: As a result, we formulate subsequent MD steps for a policy dstistribution.
Revised: As a result, we formulate subsequent MD steps for a policy dstistribution.

Original: Letw t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions
Revised: Letw t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions

Original: F at the t -th step.
Revised: F at the t -th step.

Original: Replacing the L2 proximity term of proximal gradient descent with a Bregmandivergence, the proximal form of the MD update for Eq.
Revised: Replacing the L2 proximity term of proximal gradient descent with a Bregmandivergence, the proximal form of the MD update for Eq.

Original: (1) is written as [44]
Revised: (1) is written as [44]

Original: [Equation]
Revised: [Equation]

Original: where α t := 1 / η t denotes an inverse of the current step size η t [45].
Revised: where α t := 1 / η t denotes an inverse of the current step size η t [45].

Original: Plugging each divergences of thecumulative cost F t to Eq.
Revised: Plugging each divergences of thecumulative cost F t to Eq.

Original: (6), the MD update for the subsequent reward function ψ t +1 =
Revised: (6), the MD update for the subsequent reward function ψ t +1 =

Original: Ψ Ω ( π t +1 )is derived by solving a problemminimize
Revised: Ψ Ω ( π t +1 )is derived by solving a problemminimize

Original: [Equation]
Revised: [Equation]

Original: + α t D Ω !
Revised: + α t D Ω !

Original: π s ..
Revised: π s ..

Original: π s t " ⇐⇒ minimize π s ∈ Π s
Revised: π s t " ⇐⇒ minimize π s ∈ Π s

Original: D Ω !
Revised: D Ω !

Original: π s ..
Revised: π s ..

Original: ¯ π s E ,t " − D Ω !
Revised: ¯ π s E ,t " − D Ω !

Original: π s ..
Revised: π s ..

Original: π st " + α t D Ω !
Revised: π st " + α t D Ω !

Original: π s ..
Revised: π s ..

Original: π st " ⇐⇒ minimize π s ∈ Π s η t D Ω !
Revised: π st " ⇐⇒ minimize π s ∈ Π s η t D Ω !

Original: π s ..
Revised: π s ..

Original: ¯ π s E ,t "   estimated expert + (1 − η t ) D Ω !
Revised: ¯ π s E ,t "   estimated expert + (1 − η t ) D Ω !

Original: π s ..
Revised: π s ..

Original: π st "   learning agent
Revised: π st "   learning agent

Original: ∀ s ∈ S , (7) where the gradient of D Ω is taken with respect to its ﬁrst argument π st .
Revised: ∀ s ∈ S , (7) where the gradient of D Ω is taken with respect to its ﬁrst argument π st .

Original: Note that solving theoptimization Eq.
Revised: Note that solving theoptimization Eq.

Original: (7) requires interaction between π t and the dynamics of the given environmentin order to minimize F t ; thus, the corresponding RL process plays an essential role in sequentiallearning by the induced the value measures.
Revised: (7) requires interaction between π t and the dynamics of the given environmentin order to minimize F t ; thus, the corresponding RL process plays an essential role in sequentiallearning by the induced the value measures.

Original: At a glance the objective is analogous to ﬁnding aninterpolation at each iteration where the point is controlled η t .
Revised: At a glance the objective is analogous to ﬁnding aninterpolation at each iteration where the point is controlled η t .

Original: Fig.
Revised: Fig.

Original: 3 shows that the uncertainty ofπ t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region).
Revised: 3 shows that the uncertainty ofπ t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region).

Original: [Figure]
Revised: [Figure]

Original:  Convergence Analyses
Revised:  Convergence Analyses

Original: In this section, we introduce our theoretical results.
Revised: In this section, we introduce our theoretical results.

Original: The main goals are to address (1) the convergence of MD updates for various cases and (2) the necessity of scheduling the amount of learning.
Revised: The main goals are to address (1) the convergence of MD updates for various cases and (2) the necessity of scheduling the amount of learning.

Original: NEW [ Mged ]
Revised: NEW [ Mged ]

Original: Suppose that states at the t -th step s ( t ) i ∈
Revised: Suppose that states at the t -th step s ( t ) i ∈

Original: τ t covers S by executing the policy π t in an inﬁnite horizon.
Revised: τ t covers S by executing the policy π t in an inﬁnite horizon.

Original: From this assumption, we deﬁne a temporal cost function at the time step t as
Revised: From this assumption, we deﬁne a temporal cost function at the time step t as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: that additionally involves a trajectory τ t as an input.
Revised: that additionally involves a trajectory τ t as an input.

Original: We refer to the global objective as ﬁnding aunique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :
Revised: We refer to the global objective as ﬁnding aunique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :

Original: =
Revised: =

Original: E [ f ( π , τ t )] , where the expectation istaken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π , τ t )] .
Revised: E [ f ( π , τ t )] , where the expectation istaken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π , τ t )] .

Original: Taking the (stepwise) gradient foreach π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω ( π ∗ ( ·| s ))
Revised: Taking the (stepwise) gradient foreach π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω ( π ∗ ( ·| s ))

Original: − ∇ Ω (¯ π E ,t ( ·| s ))]
Revised: − ∇ Ω (¯ π E ,t ( ·| s ))]

Original: = 0 when t → ∞ ; 196 hence, ∇ Ω ( π ∗ ( ·| s )) =
Revised: = 0 when t → ∞ ; 196 hence, ∇ Ω ( π ∗ ( ·| s )) =

Original: lim t →∞ E [ ∇ Ω (¯ π E ,
Revised: lim t →∞ E [ ∇ Ω (¯ π E ,

Original: t ( ·| s ))] .
Revised: t ( ·| s ))] .

Original: Introducing the optimal policy π ∗ allows not only 197 the speciﬁc situation when (1) π
Revised: Introducing the optimal policy π ∗ allows not only 197 the speciﬁc situation when (1) π

Original: E = π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually conver-
Revised: E = π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually conver-

Original: 198 gent with t → ∞ , but also more general situations where (2) π E / ∈ Π or the estimated expert policy¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.
Revised: 198 gent with t → ∞ , but also more general situations where (2) π E / ∈ Π or the estimated expert policy¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.

Original: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.
Revised: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.

Original: • Convergent sequence & divergent series:
Revised: • Convergent sequence & divergent series:

Original: [Equation]
Revised: [Equation]

Original: • Divergent series & convergent series of squared terms
Revised: • Divergent series & convergent series of squared terms

Original: [Equation]
Revised: [Equation]

Original: We assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.
Revised: We assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.

Original: In some Ω , thesetwo assumptions do not necessarily hold for extreme cases in ∆ SA , e.g., a distribution that π ( a | s ) =
Revised: In some Ω , thesetwo assumptions do not necessarily hold for extreme cases in ∆ SA , e.g., a distribution that π ( a | s ) =

Original: 0for some entries.
Revised: 0for some entries.

Original: Nevertheless, these outliers can be leaved out if the parametrization is constrainedto satisfy the assumptions.
Revised: Nevertheless, these outliers can be leaved out if the parametrization is constrainedto satisfy the assumptions.

Original: For example, one can either (1) prevent a policy to have non-zero entriesof probabilities for a discrete policy by a constraint on its parameters, or (2) prevent having too lowvalue of entropy for a continuous policy, typically by a constraint on the covariance functions.
Revised: For example, one can either (1) prevent a policy to have non-zero entriesof probabilities for a discrete policy by a constraint on its parameters, or (2) prevent having too lowvalue of entropy for a continuous policy, typically by a constraint on the covariance functions.

Original: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.
Revised: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: Theorem 1 (Stepsize considerations) .
Revised: Theorem 1 (Stepsize considerations) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, andthe associated Bregman divergence D Ω is bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, andthe associated Bregman divergence D Ω is bounded.

Original: Assume a general condition of the problem thatinf π
Revised: Assume a general condition of the problem thatinf π

Original: [Equation]
Revised: [Equation]

Original: Π E + f
Revised: Π E + f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , > 0 .
Revised: , > 0 .

Original: Then we get lim T →∞ E τ
Revised: Then we get lim T →∞ E τ

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: = 

Revised: =

Original: if and only if
Revised: if and only if

Original: Eq.
Revised: Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T
Revised: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T

Original: [Equation]
Revised: [Equation]

Original: (b) If the step size takes the form
Revised: (b) If the step size takes the form

Original: of η t = 4 t +

Revised: of η t = 4 t +

Original: then E τ 1: T +% ∞
Revised: then E τ 1: T +% ∞

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 0 D Ω !
Revised: 0 D Ω !

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: ( ·| s i ) ..
Revised: ( ·| s i ) ..

Original: π T ( ·| s i )) , = O (1 /T ) .
Revised: π T ( ·| s i )) , = O (1 /T ) .

Original: Next, we provide Theorem 2, which addresses the convergence in a speciﬁc case when π E resides inΠ .
Revised: Next, we provide Theorem 2, which addresses the convergence in a speciﬁc case when π E resides inΠ .

Original: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .
Revised: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .

Original: Theorem 2 (Optimal cases) .
Revised: Theorem 2 (Optimal cases) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.

Original: Assume π 1 ∕ = π E and inf π ∈ Π E [ f ( π , τ t )]
Revised: Assume π 1 ∕ = π E and inf π ∈ Π E [ f ( π , τ t )]

Original: = 0 .
Revised: = 0 .

Original: Then,
Revised: Then,

Original: E + f ( π t , τ t ) , = 

Revised: E + f ( π t , τ t ) , =

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: = ∞ .
Revised: = ∞ .

Original: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that
Revised: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that

Original: [Equation]
Revised: [Equation]

Original: , for
Revised: , for

Original: [Equation]
Revised: [Equation]

Original: sup
Revised: sup

Original: [Equation]
Revised: [Equation]

Original: FIX [ Ptuv ]
Revised: FIX [ Ptuv ]

Original: Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of thealgorithm by imposing the stronger condition of step size in Eq.
Revised: Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of thealgorithm by imposing the stronger condition of step size in Eq.

Original: The proofs are in the appendix.
Revised: The proofs are in the appendix.

Original: Proposition 1 (General cases) .
Revised: Proposition 1 (General cases) .

Original: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π , τ t )] > 0 .
Revised: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π , τ t )] > 0 .

Original: If the step sizessatisﬁes Eq.
Revised: If the step sizessatisﬁes Eq.

Original: (10), then lim t →∞
Revised: (10), then lim t →∞

Original: [Equation]
Revised: [Equation]

Original: " converges to 0 almost surely.
Revised: " converges to 0 almost surely.

Original: Regrets.
Revised: Regrets.

Original: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as
Revised: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as

Original: [Equation]
Revised: [Equation]

Original: 1 t % tj =
Revised: 1 t % tj =

Original: [Equation]
Revised: [Equation]

Original: f (
Revised: f (

Original: [Equation]
Revised: [Equation]

Original: In the optimal case of inf π ∈ Π E [ f ( π , τ t )] = 0 , the cost f inherits the property of Bregman divergenceso that the inﬁmum is achieved by 0 at π E .
Revised: In the optimal case of inf π ∈ Π E [ f ( π , τ t )] = 0 , the cost f inherits the property of Bregman divergenceso that the inﬁmum is achieved by 0 at π E .

Original: In this case, the regret is bounded to O (1 /T ) by thetheorems.
Revised: In this case, the regret is bounded to O (1 /T ) by thetheorems.

Original: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π , τ t )] > 0 whenthe step sizes abide by Eq.
Revised: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π , τ t )] > 0 whenthe step sizes abide by Eq.

Original: Thus, the regret is bounded to O (1 /T ) even for the general case.
Revised: Thus, the regret is bounded to O (1 /T ) even for the general case.

Original:  Algorithm: MD-IRL on an Adversarial Framework
Revised:  Algorithm: MD-IRL on an Adversarial Framework

Original: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized rewardfunction with adversarial learning and the MD update rule.
Revised: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized rewardfunction with adversarial learning and the MD update rule.

Original: Neural network parameters θ , φ , and ν
Revised: Neural network parameters θ , φ , and ν

Original: FIXare newly presented representing agent policy, reward, and expert policy functions respectively.
Revised: FIXare newly presented representing agent policy, reward, and expert policy functions respectively.

Original: Dual discriminators.
Revised: Dual discriminators.

Original: In order to bridge the gap between theory and practice, we propose a novel
Revised: In order to bridge the gap between theory and practice, we propose a novel

Original: NEW [ 4Ddj , Mged ]discriminative architecture, motivated by GAN studies regarding multiple discriminators [46, 47].
Revised: NEW [ 4Ddj , Mged ]discriminative architecture, motivated by GAN studies regarding multiple discriminators [46, 47].

Original: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densitiesand imitating speciﬁc behavior.
Revised: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densitiesand imitating speciﬁc behavior.

Original: Given a learning agent policy π θ , an estimation policy π ν , and adiscriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as
Revised: Given a learning agent policy π θ , an estimation policy π ν , and adiscriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as

Original: D ν
Revised: D ν

Original: ( s, a ; θ , ξ ) =
Revised: ( s, a ; θ , ξ ) =

Original: σ !
Revised: σ !

Original: [Equation]
Revised: [Equation]

Original: ( a | s
Revised: ( a | s

Original: [Equation]
Revised: [Equation]

Original: + d
Revised: + d

Original: [Equation]
Revised: [Equation]

Original: ( s ) " and D ξ ( s ) =
Revised: ( s ) " and D ξ ( s ) =

Original: σ !
Revised: σ !

Original: d ξ ( s
Revised: d ξ ( s

Original: [Equation]
Revised: [Equation]

Original: " , ∀ s
Revised: " , ∀ s

Original: [Equation]
Revised: [Equation]

Original: S , a ∈ A , where σ ( · ) denotes the sigmoid function.
Revised: S , a ∈ A , where σ ( · ) denotes the sigmoid function.

Original: The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:maximize J d ξ = E π E + log D ξ ( s ) , + E π φ + log(1 − D ξ ( s )) , , (12) maximize J π ν =
Revised: The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:maximize J d ξ = E π E + log D ξ ( s ) , + E π φ + log(1 − D ξ ( s )) , , (12) maximize J π ν =

Original: E π E
Revised: E π E

Original: + log D ν
Revised: + log D ν

Original: ( s, a ) , + E π φ + log(1 − D ν
Revised: ( s, a ) , + E π φ + log(1 − D ν

Original: ( s, a )) , , (13)
Revised: ( s, a )) , , (13)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: α t η t− η t t αη example of ( α t , η t ) .where
Revised: α t η t− η t t αη example of ( α t , η t ) .where

Original: d ξ and π θ are not trained for learning D ν .
Revised: d ξ and π θ are not trained for learning D ν .

Original: Let ρ π ∈ ∆ S denote the state visitation density ofπ , which is deﬁned as ρ π ( s ) :
Revised: Let ρ π ∈ ∆ S denote the state visitation density ofπ , which is deﬁned as ρ π ( s ) :

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E π
Revised: E π

Original: [Equation]
Revised: [Equation]

Original: where I {·} is an indicator function.
Revised: where I {·} is an indicator function.

Original: The convergence of functions in an ideal case is found at π ν = π
Revised: The convergence of functions in an ideal case is found at π ν = π

Original: [Equation]
Revised: [Equation]

Original: and d
Revised: and d

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: Learning with MD-based rewards.
Revised: Learning with MD-based rewards.

Original: Based on the MD solution for a regularized reward function,
Revised: Based on the MD solution for a regularized reward function,

Original: NEW [ 4Ddj , Mged ]we focus on developing a learning objective.
Revised: NEW [ 4Ddj , Mged ]we focus on developing a learning objective.

Original: Let ψ φ ∈ Ψ ( Π ) denote a parameterized regularized reward function and π φ denotes a corresponding policy from φ .
Revised: Let ψ φ ∈ Ψ ( Π ) denote a parameterized regularized reward function and π φ denotes a corresponding policy from φ .

Original: Note that the transformation between
Revised: Note that the transformation between

Original: NEW [ 4Ddj ]ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁcparameterizations [17].
Revised: NEW [ 4Ddj ]ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁcparameterizations [17].

Original: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν ,we deﬁne the objective of φ as direct interpretation of the update rule of Eq.
Revised: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν ,we deﬁne the objective of φ as direct interpretation of the update rule of Eq.

Original: [Equation]
Revised: [Equation]

Original: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.
Revised: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.

Original: As shown in
Revised: As shown in

Original: Fig. 4, η t is adjusted by linearly increasing α t , originated from the analyses in Section 5.
Revised: Fig. 4, η t is adjusted by linearly increasing α t , originated from the analyses in Section 5.

Original: One drawback of Eq.
Revised: One drawback of Eq.

Original: (14) isthat the trajectory training with ¯ τ t might not cover the entire state space,and it is likely that state densities between the expert and the agent are misaligned.
Revised: (14) is the issue of covariance shifts, and it is likely that state densities between the expert and the agent are misaligned.

Original: Thus, we deﬁne FIX [ 4Ddj , Mged ]the IRL reward function as linear combinations of ψ φ and the state density discriminative signal:
Revised: Thus, we deﬁne the IRL reward function as linear FIX [ 4Ddj , Mged ]combinations of ψ φ and the state density discriminative signal:

Original: [Equation]
Revised: [Equation]

Original: with a coefﬁcient λ ∈ R + .
Revised: with a coefﬁcient λ ∈ R + .

Original: Utilizing an arbitrary regularized RL algorithm a regularizer λ Ω ( · ) , thereward learning regarding agent policy π θ is decomposed into the following:
Revised: Utilizing an arbitrary regularized RL algorithm a regularizer λ Ω ( · ) , thereward learning regarding agent policy π θ is decomposed into the following:

Original: [Equation]
Revised: [Equation]

Original: Minimizing the ﬁrst term of E π θ+
Revised: Minimizing the ﬁrst term of E π θ

Revised: [Equation]

Original: D Ω ( π s θ% π s φ ) , represents learning with the MD formulation.
Revised: Ω [Equation] s θ π s φ ) , represents learning with the MD formulation.

Original: Meanimizing the second term D KL ( ρ π θ % ρ π E ) plays an auxiliary role in facilitating the supports of statevisitation densities to be properly matched.
Revised: Meanimizing the second term D KL ( ρ π [Equation] [Equation] π E ) plays an auxiliary role in facilitating the supports of statevisitation densities to be properly matched.

Original: With the hyperparameter λ , we report that learningthe second term is helpful when the state densities are heavily misaligned in certain benchmarks.
Revised: With the hyperparameter λ , we report that learningthe second term is helpful when the state densities are heavily misaligned in certain benchmarks.

Original: NEW [ 4Ddj , Mged ]
Revised: NEW [ 4Ddj , Mged ]

Original: Algorithm 1 summarizes the entire procedure.
Revised: Algorithm 1 summarizes the entire procedure.

Original: We defer additional details to Appendix B and C.
Revised: We defer additional details to Appendix B and C.

Original:  Experimental Results
Revised:  Experimental Results

Original: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ωwhile retaining scalability of AIL.
Revised: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ωwhile retaining scalability of AIL.

Original: The comparative method was RAIRL with density based models(RAIRL-DBM) which contained comparable expressiveness as MD-AIRL For RL, we used RAC algorithm [43] that is a generalization of the SAC algorithm [48].
Revised: The comparative method was RAIRL with density based models(RAIRL-DBM) which contained comparable expressiveness as MD-AIRL For RL, we used RAC algorithm [43] that is a generalization of the SAC algorithm [48].

Original: We considered a class of regularizersΩ ( p ) =
Revised: We considered a class of regularizersΩ ( p ) =

Original: − E x ∼ p
Revised: − E x ∼ p

Original: [ ϕ ( p ( x ))] with (1) Shannon ( ϕ ( x ) = log( x ) ), (2) Tsallis ( ϕ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) ,q = 2 by default), (3) exp ( ϕ ( x )=
Revised: [ ϕ ( p ( x ))] with (1) Shannon ( ϕ ( x ) = log( x ) ), (2) Tsallis ( ϕ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) ,q = 2 by default), (3) exp ( ϕ ( x )=

Original: e − e x ), (4) cos ( ϕ ( x )=cos( π 2 x ) ), and (5) sin ( ϕ ( x )=
Revised: e − e x ), (4) cos ( ϕ ( x )=cos( π 2 x ) ), and (5) sin ( ϕ ( x )=

Original: 1 − sin( π 2 x ) ).
Revised: 1 − sin( π 2 x ) ).

Original: 7.1 Large scale multiarmed bandits
Revised: 7.1 Large scale multiarmed bandits

Original: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the carNEW [ Mged ]dinality of action spaces varies largely.
Revised: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the carNEW [ Mged ]dinality of action spaces varies largely.

Original: Learning the optimal distribution of π E becomes challengingas the size of action space |A| increases, because the frequency of each sample becomes sparse due 275 to the curse of dimensionality.
Revised: Learning the optimal distribution of π E becomes challengingas the size of action space |A| increases, because the frequency of each sample becomes sparse due 275 to the curse of dimensionality.

Original: The stateless expert distribution π E was generated by the parametersof softmax distribution π E ( a ) =
Revised: The stateless expert distribution π E was generated by the parametersof softmax distribution π E ( a ) =

Original: exp( z a ) / !
Revised: exp( z a ) / !

Original: i exp( z i )
Revised: i exp( z i )

Original: where the logits z
Revised: where the logits z

Original: i were randomly initialized.
Revised: i were randomly initialized.

Original: Weset the action size to |A| =
Revised: Weset the action size to |A| =

Original: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.
Revised: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.

Original: [Table]
Revised: [Table]

Original: Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when threedifferent cardinalities and ﬁve regularizers were considered.
Revised: Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when threedifferent cardinalities and ﬁve regularizers were considered.

Original: Fig.
Revised: Fig.

Original: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1to be greater than 1 ( α 1 = 0 . 5 ).
Revised: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1to be greater than 1 ( α 1 = 0 . 5 ).

Original: MD-AIRL exceeded the discriminative performance of RAIRLafter certain steps, while the progression of RAIRL mostly stopped at local minima.
Revised: MD-AIRL exceeded the discriminative performance of RAIRLafter certain steps, while the progression of RAIRL mostly stopped at local minima.

Original: MD-AIRLoutperformed RAIRL in four cases by an choosing effectively low step size at η T to be less than1 ( α T = 2 ).
Revised: MD-AIRLoutperformed RAIRL in four cases by an choosing effectively low step size at η T to be less than1 ( α T = 2 ).

Original: These results match the properties of MD algorithms and our convergence analyses.
Revised: These results match the properties of MD algorithms and our convergence analyses.

Original: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robustreward acquisition and imitation for situations when the total number of data samples is limited.a b
Revised: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robustreward acquisition and imitation for situations when the total number of data samples is limited.a b

Original: G oa l en t r op y
Revised: G oa l en t r op y

Original: Reaching multigoals q fromTsallisentropies c r eg .
Revised: Reaching multigoals q fromTsallisentropies c r eg .

Original: r e w a r d
Revised: r e w a r d

Original: M D r e w a r d π ( ·| s )
Revised: M D r e w a r d π ( ·| s )

Original: Shannon regularizer
Revised: Shannon regularizer

Original: Step5K
Revised: Step5K

Original: Step300K r eg .
Revised: Step300K r eg .

Original: r e w a r d
Revised: r e w a r d

Original: M D r e w a r d π ( ·| s )
Revised: M D r e w a r d π ( ·| s )

Original: Tsallis regularizer
Revised: Tsallis regularizer

Original: Step5K
Revised: Step5K

Original: Step300K
Revised: Step300K

Original: 7.2 A continuous multigoal environment
Revised: 7.2 A continuous multigoal environment

Original: We then considered a multigoal environment.
Revised: We then considered a multigoal environment.

Original: In this environment, an agent is a two-dimensionalpoint mass initialized at the origin, and the four goals are located in the four cardinal directions.
Revised: In this environment, an agent is a two-dimensionalpoint mass initialized at the origin, and the four goals are located in the four cardinal directions.

Original: Theobjective of imitation learning is to go toward each direction evenly as possible where the expertmodel was trained by an SAC algorithm.
Revised: Theobjective of imitation learning is to go toward each direction evenly as possible where the expertmodel was trained by an SAC algorithm.

Original: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with fullcovariance matrices instead of conventional diagonal Gaussian policies (see Appendices 2 and 3).
Revised: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with fullcovariance matrices instead of conventional diagonal Gaussian policies (see Appendices 2 and 3).

Original: Fig.
Revised: Fig.

Original: 6 (a) shows trajectories generated by the trained agent.
Revised: 6 (a) shows trajectories generated by the trained agent.

Original: Fig.
Revised: Fig.

Original: 6 (b) shows that MD-AIRL achievedhigher entropy for reaching the multiple goals.
Revised: 6 (b) shows that MD-AIRL achievedhigher entropy for reaching the multiple goals.

Original: Fig.
Revised: Fig.

Original: 6 (c) shows reward surfaces with regularizers,which were calculated by ψ φ ( s, a )
Revised: 6 (c) shows reward surfaces with regularizers,which were calculated by ψ φ ( s, a )

Original: + ϕ
Revised: + ϕ

Original: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .
Revised: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .

Original: Duringthe training, the MD reward was similar to the estimated ground truth using adversarial training.
Revised: Duringthe training, the MD reward was similar to the estimated ground truth using adversarial training.

Original: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t wassufﬁciently close to the expert behavior.
Revised: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t wassufﬁciently close to the expert behavior.

Original: As a result, we claim that a drastic change in the targetdistribution, which is one of the typical characteristics of adversarial frameworks, are prevented.
Revised: As a result, we claim that a drastic change in the targetdistribution, which is one of the typical characteristics of adversarial frameworks, are prevented.

Original: Weargue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.
Revised: Weargue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.

Original: 7.3 A continuous control benchmark: MuJoCo
Revised: 7.3 A continuous control benchmark: MuJoCo

Original: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.
Revised: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.

Original: We assumedfull covariance Gaussian policies for both learner’s policy π and expert policy π E .
Revised: We assumedfull covariance Gaussian policies for both learner’s policy π and expert policy π E .

Original: We used the hyperbolized environment assumption [17] where the action constraint is incorporated to the dynamicsas a part of the environment using hyperbolic tangent activation.
Revised: We used the hyperbolized environment assumption [17] where the action constraint is incorporated to the dynamicsas a part of the environment using hyperbolic tangent activation.

Original: Sample efﬁciency.
Revised: Sample efﬁciency.

Original: For each task, we considered two different numbers of episodes collected byan expert policy.
Revised: For each task, we considered two different numbers of episodes collected byan expert policy.

Original: In Fig.
Revised: In Fig.

Original: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [19] is shown with the expert and random agent performance.
Revised: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [19] is shown with the expert and random agent performance.

Original: MD-AIRL was able to achieve
Revised: MD-AIRL was able to achieve

Original: Hopper-v
Revised: Hopper-v

Original: Walker2d-v
Revised: Walker2d-v

Original: HalfCheetah-v
Revised: HalfCheetah-v

Original: Ant-v
Revised: Ant-v

Original: [Figure]
Revised: [Figure]

Original: random
Revised: random

Original: The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.
Revised: The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.

Original: [Figure]
Revised: [Figure]

Original: HalfCheetah
Revised: HalfCheetah

Original: Ant consistent performance throughout the tasks and demonstration size.
Revised: Ant consistent performance throughout the tasks and demonstration size.

Original: On the training curves, MD-
Revised: On the training curves, MD-

Original: AIRL showed high tolerance to the scarcity of data compared to RAIRL for 4 expert demonstra- tions.
Revised: AIRL showed high tolerance to the scarcity of data compared to RAIRL for 4 expert demonstra- tions.

Original: The plots in Fig.
Revised: The plots in Fig.

Original: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among 32 different conﬁgurations we have tested
Revised: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among 32 different conﬁgurations we have tested

Original: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.
Revised: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Noisy demonstrations.
Revised: Noisy demonstrations.

Original: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with two different levels of Gaussian additive noises, resulting to suboptimaldemonstrations.
Revised: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with two different levels of Gaussian additive noises, resulting to suboptimaldemonstrations.

Original: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.
Revised: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.

Original: The experiment is closely related to the general case in the theory the results suggest that thecharacteristics of MD-AIRL are in alignment with our analyses of the MD reward learning scheme
Revised: The experiment is closely related to the general case in the theory the results suggest that thecharacteristics of MD-AIRL are in alignment with our analyses of the MD reward learning scheme

Original: [Table]
Revised: [Table]

Original:  Conclusions and Discussion
Revised:  Conclusions and Discussion

Original: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitationlearning problem in the real world.
Revised: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitationlearning problem in the real world.

Original: We proved that the proposed method has clear advantages overprevious AIL methods in terms of robustness.
Revised: We proved that the proposed method has clear advantages overprevious AIL methods in terms of robustness.

Original: We veriﬁed MD-AIRL on a variety of situations,including high-dimensional spaces, limited samples, and imperfect demonstrations.
Revised: We veriﬁed MD-AIRL on a variety of situations,including high-dimensional spaces, limited samples, and imperfect demonstrations.

Original: The empiricalevidence showed that MD-AIRL outperforms previous methods on various benchmarks.
Revised: The empiricalevidence showed that MD-AIRL outperforms previous methods on various benchmarks.

Original: We conclude that the rich foundation of optimization theories shows a promising direction to AIL studies.
Revised: We conclude that the rich foundation of optimization theories shows a promising direction to AIL studies.

Original: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.
Revised: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.

Original: Although our work covers various imitation learning problems with the Bregman divergence, this doesnot include some other problems when the proximity term is of other statistical divergence familiessuch as the f-divergence [50].
Revised: Although our work covers various imitation learning problems with the Bregman divergence, this doesnot include some other problems when the proximity term is of other statistical divergence familiessuch as the f-divergence [50].

Original: If the relationship between these classes of divergences is studiedin more detail, it is expected to proceed applications to various subﬁelds of machine learning.
Revised: If the relationship between these classes of divergences is studiedin more detail, it is expected to proceed applications to various subﬁelds of machine learning.

Original: The
Revised: The

Original: NEW [ Ptuv , Mged ]assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but someoutliers might have substantial meaning for certain tasks.
Revised: NEW [ Ptuv , Mged ]assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but someoutliers might have substantial meaning for certain tasks.

Original: Therefore, extensive analyses on theseparameterizations remain as future works.
Revised: Therefore, extensive analyses on theseparameterizations remain as future works.

Original: The “impurity” of Eq.
Revised: The “impurity” of Eq.

Original: (15) compared to Ψ  ( Π ) can also
Revised: (15) compared to Ψ Ω ( Π ) can also

Original: NEW [ Ptuv , Mged ]be regarded as a limitation.
Revised: NEW [ Ptuv , Mged ]be regarded as a limitation.

Original: To fully resolve this issue, all data must be treated as on-policy samples,which might require a sophisticated sampling mechanism on top of the adversarial framework.
Revised: To fully resolve this issue, all data must be treated as on-policy samples,which might require a sophisticated sampling mechanism on top of the adversarial framework.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See
Revised: [Yes] See

Original: Section 8.
Revised: Section 8.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A.
Revised: [Yes] See Appendix A.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: Our empricalstudies can be reproduced by from the detailed information in Appendices B & C. Thecode and data will be available in the ﬁnal version.(b)
Revised: Our empricalstudies can be reproduced by from the detailed information in Appendices B & C. Thecode and data will be available in the ﬁnal version.(b)

Original: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix C.
Revised: [Yes] See Appendix C.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 7.
Revised: [Yes] See Section 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: In experiments, each algorithmwas exectued in CPU (a single thread).
Revised: In experiments, each algorithmwas exectued in CPU (a single thread).

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g
Revised: g

Original: , code, data, models) or curating/releasing new assets.
Revised: , code, data, models) or curating/releasing new assets.

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes
Revised: [Yes

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: The MuJoCo simulator used in our
Revised: The MuJoCo simulator used in our

Original: [Table]
Revised: [Table]

Original: experiments is freely available to everyone.
Revised: experiments is freely available to everyone.

Original: See the site (https://mujoco.org).
Revised: See the site (https://mujoco.org).

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: [Table]
Revised: [Table]

Original: We are preparing to release the source code and data in the ﬁnal version.
Revised: We are preparing to release the source code and data in the ﬁnal version.

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data
Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Original: [Table]
Revised: [Table]

Original: you’re using/curating?
Revised: you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁ-

Original: [Table]
Revised: [Table]

Original: able information or offensive content?
Revised: able information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]