Original: Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Revised: Robust Imitation via Mirror Descent Inverse Reinforcement Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recently, adversarial imitation learning has shown a scalable reward acquisitionmethod for inverse reinforcement learning (IRL) problems.
Revised: Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems.

Original: However, estimatedreward signals often become uncertain and fail to train a reliable statistical modelsince the existing methods tend to solve hard optimization problems directly.
Revised: However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly.

Original: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions fora constrained convex problem.
Revised: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem.

Original: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount ofreward learning is regulated with respect to local geometric constraints.
Revised: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints.

Original: We provethat the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes{ η t } Tt =1 .
Revised: We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes { η t } Tt =1 .

Original: Our IRL method was applied on top of an adversarial framework, and itoutperformed existing adversarial methods in an extensive suite of benchmarks.
Revised: Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.

Original:  Introduction
Revised:  Introduction

Original: One crucial requirement of practical imitation learning algorithms is robustness , often describedas learning expert behavior for  ﬁnite number of demonstrations, overcoming various realistic chalFIX [ Ptuv ]lenges [1].
Revised: One crucial requirement of practical imitation learning methods is robustness , often described as learning expert behavior for a ﬁnite number of demonstrations, overcoming various realistic challenges [1].

Original: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations canbe noisy or suboptimal to solve the problem [3].
Revised: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations can be noisy or suboptimal to solve the problem [3].

Original: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling theuncertainty of estimated cost functions dramatically affectsthe imitation performance.
Revised: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling the uncertainty of estimated cost functions dramatically affects imitation performance.

Original: Therefore, athorough analysis of addressing these issues is required to construct a robust algorithm.
Revised: Therefore, a thorough analysis of addressing these issues is required to construct a robust algorithm.

Original: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expertdemonstrations where the expert acts optimally with respect to an unknown reward function [4].
Revised: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expert demonstrations where the expert acts optimally with respect to an unknown reward function [4, 5].

Original: Traditional IRL methods solve the imitation problem based on iterative algorithms [5, 6], alternatingbetween the reward estimation process and a reinforcement learning (RL) [7] algorithm.
Revised: Traditional IRL studies solve the imitation problem based on iterative algorithms [6, 7], alternating between the reward estimation process and a reinforcement learning (RL) [8] algorithm.

Original: Newerstudies of adversarial imitation learning (AIL) [8, 9] rather suggest learning reward functions of a FIX [ Mged ]certain form “directly,” by using adversarial learning objectives [10] and nonlinear discriminativeneural networks [9].
Revised: In contrast, newer studies of adversarial imitation learning (AIL) [9, 10] rather suggest learning reward functions of a certain form “directly,” by using adversarial learning objectives [11] and nonlinear discriminative neural networks [10].

Original: Compared to classical approaches, the AIL methods have shown great successon control benchmarks in terms of scalability for challenging control tasks [11].
Revised: Compared to classical approaches, the AIL methods have shown great success on control benchmarks in terms of scalability for challenging control tasks [12].

Original: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [12].
Revised: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [13].

Original: Throughthe lens of differential geometries, the limitation of AIL naturally comes from the implication thatminimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [13].
Revised: Through the lens of differential geometries, the limitation of AIL naturally comes from the implication that minimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [14].

Original: In order to ensure  stability, we argue that an IRL algorithm’s progress needs to beregulated, yielding gradual updates with respect to local geometries of policy distributions
Revised: In order to ensure further stability, we argue that an IRL algorithm’s progress needs to be regulated, yielding gradual updates with respect to local geometries of policy distributions.

Original: We claim that there are two issues leading to unconstrained policy updates: (1) a statistical divergence often cannot be accurately obtained for challenging problems and (2) an immediate divergencebetween agent and expert densities does not guarantee unbiased learning directions.
Revised: We claim that there are two issues leading to unconstrained policy updates: 1 (cid:13) a statistical divergence often cannot be accurately obtained for challenging problems, and 2 (cid:13) an immediate divergence between agent and expert densities does not guarantee unbiased learning directions.

Original: Our approach isconnected to a collection of optimization processes called mirror descent (MD) [14].
Revised: Our approach is connected to a collection of optimization processes called mirror descent (MD) [15].

Original: For a sequenceof parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as
Revised: For a sequence of parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as

Original: [Equation]
Revised: [Equation]

Original: In the equation, the gradient ∇ Ω ( · ) creates a transformation that links a parametric space to its dualspace.
Revised: In the equation, the gradient ∇ Ω( · ) creates a transformation that links a parametric space to its dual space.

Original: Theoretically, MD is a ﬁrst-order method for solving constrained problems which enjoysrigorous regret bounds for various geometries [15, 16], including probability spaces.
Revised: Theoretically, MD is a ﬁrst-order method for solving constrained problems, which enjoys rigorous regret bounds for various geometries [16, 17] including probability spaces.

Original: Thus, applying
Revised: Thus, applying

Original: MD to the reward estimation  can be efﬁcient in terms of the number of learning phases.
Revised: MD to the reward estimation process can be efﬁcient in terms of the number of learning phases.

Original: In this paper, we derive an MD update rule in IRL with a postulate of nonstationary estimations ofthe expert density, resulting to convergent reward acquisition even for challenging problems.
Revised: In this paper, we derive an MD update rule in IRL upon a postulate of nonstationary estimations of the expert density, resulting in convergent reward acquisition even for challenging problems.

Original: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions onan alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).
Revised: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions on an alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).

Original: To this end, we propose an AILalgorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).
Revised: To this end, we propose an AIL algorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).

Original: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)
Revised: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)

Original: [17]methods.
Revised: [18] methods.

Original: For example, MD-AIRL showed higher performance in 30 distinct cases among 32 different conﬁgurations in challenging MuJoCo [18] benchmarks, and it  clearly showed higher toleranceto suboptimal data.
Revised: For example, MD-AIRL showed higher performance in 30 distinct cases amongdifferent conﬁgurations in challenging MuJoCo [19] benchmarks, and it also clearly showed higher tolerance to suboptimal data.

Original: All of these results are strongly aligned with our theoretical results.
Revised: All of these results are strongly aligned with our theoretical analyses.

Original: [Table]
Revised: [Table]

Original: Rewards Bregman divergence Iterative

Original: [Table]

Original: Our contributions.
Revised: Our contributions.

Original: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithmsas a combined optimization process with dual aspects.
Revised: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithms as a combined optimization process with dual aspects.

Original: Comparing MD-AIRL with RAIRL, both arehighly generalized algorithms in terms of a variety of choices of divergence functions. 
Revised: Comparing MD-AIRL and RAIRL, both are highly generalized algorithms in terms of a variety of choices of divergence functions. Tab.

Original: Tab. 1 showsthat MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since ourapproach is more aligned with earlier theoretical IRL studies providing formalized reward learningschemes and convergence guarantees.
Revised: 1 shows that MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since our approach is more aligned with earlier theoretical IRL studies providing formalized reward learning schemes and convergence guarantees.

Original: In summary, we list our main contributions below:
Revised: In summary, we list our main contributions below:

Original: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence ofreward functions that provides iterative local objectives (Section 4).• We formally prove that rewards derived by an MD update rule guarantee  robust performance ofdivergence minimization along with a rigorous regret bound (Section 5).•
Revised: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence of reward functions that provides iterative local objectives (Section 4). • We formally prove that rewards derived by an MD update rule guarantee the robust performance of divergence minimization along with a rigorous regret bound (Section 5).

Original:  We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant tounreliable discriminative signals of the AIL framework (Sections 6 and 7).
Revised: • We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant of unreliable discriminative signals of the AIL framework (Sections 6 and 7).

Original:  Related Works
Revised:  Related Works

Original: Mirror descent.
Revised: Mirror descent.

Original: We are interested in a family of statistical divergence called the Bregman divergence [20].
Revised: We are interested in a family of statistical divergences called the Bregman divergence [21].

Original: The divergence generalizes constrained optimization problems such as least squares[21, 22], and it also has been applied in various subﬁelds of machine learning [23, 24].
Revised: The divergence generalizes constrained optimization problems such as least squares [22, 23] [Table] and it also has been applied in various subﬁelds of machine learning [24, 25].

Original: In differentialgeometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁesmetric-like properties [13, 25].
Revised: In differential geometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁes metric-like properties [14, 26].

Original: MD is also closely related to optimization methods regarding nonFIX [ Mged ] Euclidean geometries with a discretization of steps such as natural gradients [26, 27].
Revised: MD is also closely related to optimization methods regarding nonEuclidean geometries with a discretization of steps such as natural gradients [27, 28].

Original: In the primalspace, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow[28, 29].
Revised: In the primal space, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow [29, 30].

Original: In the RL domain, MD has been recently studied for policy optimization [30–32].
Revised: In the RL domain, MD has been recently studied for policy optimization [31–33].

Original: In thispaper, we focus on learning with suboptimal representations of policy, and our distinct goal is todraw a robust reward learning scheme based on MD for the IRL problem.
Revised: In this paper, we focus on learning with suboptimal representations of policy, and our distinct goal is to draw a robust reward learning scheme based on MD for the IRL problem.

Original: Imitation learning.
Revised: Imitation learning.

Original: As a statistical model for the information geometry [33], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient IRL, and maximum likelihood IRL [34–36] for modeling expert distribution to parameterized functions.
Revised: As a statistical model for the information geometry [34], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient IRL, and maximum likelihood IRL [35–37] for modeling expert distribution to parameterized functions.

Original: Notably, MaxEnt IRL [6, 37] is one of the representative classical IRL algorithmsbased on an information-theoretic perspective toward IRL solutions.
Revised: Notably, MaxEnt IRL [7, 38] is one of the representative classical IRL algorithms based on an information-theoretic perspective toward IRL solutions.

Original: Also, discriminators of AIL aretrained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy functionthat approximates the truth data density for the expert distribution [38].
Revised: Also, discriminators of AIL are trained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy function that approximates the truth data density for the expert distribution [39].

Original: Other statistical entropieshave also been applied to AIL, such as the Tsallis entropy [39].
Revised: Other statistical entropies have also been applied to AIL, such as the Tsallis entropy [40].

Original: On the one hand, our approach isclosely related to RAIRL [17], which deﬁned its  objective as minimizing the Bregman divergence.
Revised: On the one hand, our approach is closely related to RAIRL [18], which deﬁned its AIL objective using the Bregman divergence.

Original: On the other hand, this work further employs the  divergence to derive iterative MD updates for reward functions, resulting to theoretically pleasing properties while retaining the scalability of AIL.
Revised: On the other hand, this work further employs the Bregman divergence to derive iterative MD updates for reward functions, resulting in theoretically pleasing properties while retaining the scalability of AIL.

Original: Learning theory.
Revised: Learning theory.

Original: There have been considerable achievements in dealing with temporal costs{ F t } ∞ t =1 , often referred to as online learning [40].
Revised: There have been considerable achievements in dealing with temporal costs { F t } ∞ t =1 , often referred to as online learning [41].

Original: The most ordinary approach is stochastic gradi- 91 ent descent (SGD): w t +1 = w t − η t ∇ F t ( w t ) .
Revised: The most ordinary approach is stochastic gradient descent (SGD): w t +1 = w t − η t ∇ F t ( w t ) .

Original: In particular, SGD is a desirable algorithm when theparameter w t resides in the Euclidean space since it ensures unbiased minimization of the expectedcost.
Revised: In particular, SGD is a desirable algorithm when the parameter w t resides in the Euclidean space since it ensures unbiased minimization of the expected cost.

Original: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may notbe the direction of the steepest descent due to geometric constraints [26, 33].
Revised: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may not be the direction of the steepest descent due to geometric constraints [27, 34].

Original: An online form of MD(Eq.
Revised: An online form of MD in Eq.

Original: (1)) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a Bregman divergence [29].
Revised: (1) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a Bregman divergence [30].

Original: Our theoretical ﬁndings and proofs follow the results of online mirrordescent (OMD) that appeared in previous literature for general aspects [14, 15, 41, 27, 16, 29].
Revised: Our theoretical ﬁndings and proofs follow the results of online mirror descent (OMD) that appeared in previous literature for general aspects [15, 16, 42, 28, 17, 30].

Original: Ouranalyses extend existing theoretical results to IRL; at the same time, they are also highly general tocover various online imitation learning problems which require making decision sequentially.
Revised: Our analyses extend existing theoretical results to IRL; at the same time, they are also highly general to cover various online imitation learning problems which require making decisions sequentially.

Original:  Background
Revised:  Background

Original: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional)probabilities over X (conditioned on Y ).
Revised: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional) probabilities over X (conditioned on Y ).

Original: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ )with the state space S , the action space A , the Markovian transition kernel P ∈ ∆ S × A S , the rewardfunction r ∈ R S 
Revised: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ ) with the state space S , the action space A , the Markovian transition kernel P ∈ ∆ S×AS , the reward function r ∈ R S × A

Original: × A and the discount factor γ ∈ [0 , 1) .
Revised: and the discount factor γ ∈ [0 , 1) .

Original: Let a function Ω : ∆ A → R be stronglyconvex.
Revised: Let a function Ω: ∆ A → R be strongly convex.

Original: Using Ω , the Bregman divergence is deﬁned as
Revised: Using Ω , the Bregman divergence is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where π s and ˆ π  denote arbitrary policies for a given state s .
Revised: where π s and ˆ π s denote arbitrary policies for a given state s .

Original: For a representative divergence, onecan consider the popular KL divergence.
Revised: For a representative divergence, one can consider the popular KL divergence.

Original: The KL divergence is a Bregman divergence when Ω isspeciﬁed as the negative Shannon entropy: Ω ( π s 
Revised: The KL divergence is a Bregman divergence when Ω is speciﬁed as the negative Shannon entropy: Ω( π s ) =

Original: [Equation]
Revised: (cid:80) a π s ( a ) ln π s ( a ) .

Original: Regularization of policy distribution with respect to convex Ω brings distinct properties for learningagent [42, 43].
Revised: Regularization of policy distribution with respect to convex Ω brings distinct properties to the learning agent [43, 44].

Original: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected valueof discounted cumulative returns along with a causal convex regularizer Ω: ∆ A → R , i.e.
Revised: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected value of discounted cumulative returns along with a causal convex regularizer Ω , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for thegiven MDP.
Revised: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for the given MDP.

Original: In this setup, a trainable policy space is a subset of theentire conditional probabilityspace denoted as Π :=[ Π s ] s ∈ S ⊂ ∆ S A , which is constrained by the parameterization of a policy.
Revised: In this setup, a regularized RL algorithm ﬁnds a unique solution in a subset of the conditional probability space denoted as Π:=[Π s ] s ∈ S ⊂ ∆ SA constrained by the parameterization of a policy.

Original: The objective of IRL is to ﬁnd a function r E that rationalizes theexpert behavior  π E .
Revised: The objective of IRL is to ﬁnd a function r E that rationalizes the behavior of an expert policy π E .

Original: For an innerproduct 〈 · , · 〉 A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :
Revised: For an inner product (cid:104)· , ·(cid:105) A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :

Original: [Equation]
Revised: [Equation]

Original: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .
Revised: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .

Original: Differentiating both sides withrespect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.
Revised: Differentiating both sides with respect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.

Original: One fundamentalproperty in regularized IRL [42] is that π E is the maximizing argument of Ω ∗ for q E , where q E isthe regularized state-action value function q E ( s, a ) =
Revised: One fundamental property in regularized IRL [43] is that π E is the maximizing argument of Ω ∗ for q E , where q E is the regularized state-action value function q E ( s, a )=

Original: E π E [ % [Equation]
Revised: E π E [ (cid:80) ∞

Original: = [Equation] γ [Equation]{+i =
+}
Revised: {+i =
+}

Original:  { r E ( s, a ) − Ω ( π s E ) }| s 

Revised: γ i { r E ( s, a ) − Ω( π s E ) }| s

Original: = s, a 0{+
+} =121 a ] .
Revised: = s, a {+
+} = a ] .

Original: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E =∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.
Revised: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E = ∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.

Original: Addressing this issue, Jeon et al.
Revised: Addressing this issue, Jeon et al.

Original: [17] proposed a rewardoperator Ψ Ω : ∆ SA → R S × A , providing a unique IRL solution by Ψ Ω ( π E ) .
Revised: [18] proposed a reward operator Ψ Ω :∆ SA → R S×A , providing a unique IRL solution by Ψ Ω ( π E ) .

Original: Deﬁnition 1 (Regularized reward operators) .
Revised: Deﬁnition 1 (Regularized reward operators) .

Original: Deﬁne the regularized reward operator Ψ Ω asψ π ( s, a ) :
Revised: Deﬁne the regularized reward operator Ψ Ω as ψ π ( s, a ) :

Original: = Ω ′ ( s, a ; π ) − # π s , ∇ Ω ( π s ) $
Revised: = Ω (cid:48) ( s, a ; π ) − (cid:10) π s , ∇ Ω( π s ) (cid:11)

Original: A + Ω ( π s ) , for Ω ′ ( s, · ; π ) :
Revised: A + Ω( π s ) , for Ω (cid:48) ( s, · ; π ) :

Original: = ∇ Ω ( π s ) = + ∇ p Ω ( p ) , p = π ( ·| s ) .
Revised: = ∇ Ω( π s ) = (cid:2) ∇ p Ω( p ) (cid:3) p = π ( ·| s ) .

Original: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite Bregman divergences derived from Eq.
Revised: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite Bregman divergences derived from Eq.

Original: (2) allows reward learning in a greedy manner [17].
Revised: (2) allows reward learning in a greedy manner [18].

Revised:  RL-IRL as a Proximal Method

Original: P o li c y S p ace Π π t π t +˜ π t +
Revised: P o li c y S p ace Π π t π t +˜ π t +

Original: P Ω
Revised: P Ω

Original: R e w a r d S p ace ψ t ψ t +
Revised: R e w a r d S p ace ψ t ψ t +

Original: R S ×
Revised: R S ×

Original: A Ψ Ω ∇ Ω ∗
Revised: A Ψ Ω ∇ Ω ∗

Original: Associated reward functions.
Revised: Associated reward functions.

Original: We consider the
Revised: We consider the

Original: RL-IRL processes as a sequential algorithm withlocal constraints and deﬁne sequences { π t } ∞ t =1132 and { ψ t } ∞ t =1{+=
+} that denote policies and associated133 reward functions, respectively.
Revised: RL-IRL processes as a sequential algorithm with local constraints and deﬁne sequences { π t } ∞ t =1 and { ψ t } ∞ t {+=
+} that denote policies and associated reward functions, respectively.

Original: The associated reward functions is in a space Ψ Ω ( Π ) , which is analternative space of the dual space, deﬁned by theregularized reward operator Ψ Ω .
Revised: The associated reward functions are in a space Ψ Ω (Π) , which is an alternative space of the dual space, deﬁned by the regularized reward operator Ψ Ω .

Original: Formally, we provide Lemma 1 which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .
Revised: Formally, we provide Lemma 1, which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .

Original: Theproof is in Appendix A.
Revised: The proof is in Appendix A.

Original: Lemma 1 (Natural isomorphism) .
Revised: Lemma 1 (Natural isomorphism) .

Original: Let ψ ∈ Ψ Ω ( Π )for Ψ Ω ( X ) :
Revised: Let ψ ∈ Ψ Ω (Π) for Ψ Ω ( X ) :

Original: = { ψ | ψ ( s, a ) =
Revised: = { ψ | ψ ( s, a ) =

Original: ψ π ( s, a ) , ∀ s ∈
Revised: ψ π ( s, a ) , ∀ s ∈

Original: S , a ∈ A , π ∈ X } .
Revised: S , a ∈A , π ∈ X } .

Original: Then, ∇ Ω ∗ ( ψ ) is unique and143 for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .
Revised: Then, ∇ Ω ∗ ( ψ ) is unique and for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .

Original: Fig.
Revised: Fig.

Original: 1 illustrates that there is  unique ψ t for π t in every time step.
Revised: 1 illustrates that there is a unique ψ t for π t in every time step.

Original: Note that Ψ Ω ( π t ) is differentfrom ∇ Ω ( π t ) ; it is shifted by a vector 1 c with a constant c = Ω ( π st ) − 〈 π st , ∇ Ω ( π st ) 〉
Revised: Note that Ψ Ω ( π t ) is different from ∇ Ω( π t ) ; it is shifted by a vector 1 c with a constant c = Ω( π st ) − (cid:104)

Original:  A .
Revised: π st , ∇ Ω( π st ) (cid:105) A .

Original: Since theunderlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for bothΨ Ω and ∇ Ω , as the distributivity [42] Ω ∗ ( y + 1 c ) =
Revised: Since the underlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for both Ψ Ω and ∇ Ω , as the distributivity [43] Ω ∗ ( y + 1 c ) =

Original: Ω ∗ ( y )+ c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).
Revised: Ω ∗ ( y ) + c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).

Original: Analternative interpretation is of considering a projection (gray dashed line in Fig. 1).
Revised: An alternative interpretation is of considering a projection (gray dashed line in Fig. 1).

Original: Suppose that apolicy π t is updated to ˜ π t +1 ∈ R S × A .
Revised: Suppose that a policy π t is updated to ˜ π t +1 ∈ R S×A .

Original: The Bregman projection operator P Ω is applied that locates the150 subsequent update π t [Equation] 1 to the “feasible” region, i.e.
Revised: The Bregman projection operator P Ω is applied that locates the subsequent update π t +1 to the “feasible” region, i.e., P Ω (˜ π t +1 ) :

Original: [Equation]
Revised: = argmin π ∈ Π [ D Ω ( π s (cid:107) ˜ π st +1 )] s ∈ S .

Original: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .
Revised: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .

Original: For instance, the parameters of ψ t can construct asoftmax policy for a discrete space, or a Gaussian policy for a continuous space.
Revised: For instance, the parameters of ψ t can construct a softmax policy for a discrete space, or a Gaussian policy for a continuous space.

Original: Using the rewardfunction ψ t +1 , an arbitrary regularized RL process maximizing Eq.
Revised: Using the reward function ψ t +1 , an arbitrary regularized RL process maximizing Eq.

Original: (2) at the t -th step [17]
Revised: (2) at the t -th step [18]

Original: [Equation]
Revised: [Equation]

Original: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the RL objective.
Revised: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the expected cumulative return.

Original: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.
Revised: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.

Original: Online imitation learning.
Revised: Online imitation learning.

Original: Our setup comes from an observation that  imitation learning do notretain the global target π E during training.
Revised: Our setup starts from the apparent yet vital premise that an imitation learning algorithm does not retain the global target π E during training.

Original: That is, it is fundamentally uncertain to model globalobjectives (such as J Ω ( π , ψ E ) ) which are not attainable for both RL and IRL.
Revised: That is, it is fundamentally uncertain to model global objectives (such as J Ω ( π, ψ E ) ), which are not attainable for both RL and IRL.

Original: Instead, we hypothesize on  existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex162 neighborhood of π E , generated by an arbitrary estimation algorithm.
Revised: Instead, we hypothesize on the existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex neighborhood of π E , generated by an arbitrary estimation algorithm.

Original: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) ,  J Ω ( π , ψ ¯ π E,t ) forms a temporal costs F t :
Revised: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) , the nonstationary objective J Ω ( π, ψ ¯ π E,t ) forms a temporal cost:

Original: [Equation]
Revised: [Equation]

Original: For a better understanding, we consider an actual experiment  in Fig.
Revised: For the sake of better understanding, we considered an actual experiment depicted in Fig.

Original: Suppose that policiesof the learning agent and the expert follow multivariate Gaussian distributions atN ( [0 , 0] T , I) and
Revised: Suppose that that policies of the learning agent and the expert follow multivariate Gaussian distributions at

Original: FIX [ Mged ]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: N (  [5 , 3] T , Σ E ) with | Σ E | < 1 .
Revised: N ( [0 , 0] T , I) and N ( [5 , 3] T , Σ E ) with | Σ E | < 1 .

Original: Suppose a (suboptimal) reference policy ¯ π E ,t , independently ﬁtted witha maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .
Revised: Let a (suboptimal) reference policy ¯ π E ,t be independently ﬁtted with a maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .

Original: Thepolicy π t was trained by a cost function D Ω ( · % ¯ π E ,t ) using the MD update rule in Eq.
Revised: The policy π t was trained by a cost function D Ω ( ·(cid:107) ¯ π E ,t ) using the MD update rule in Eq.

Original: In Fig.
Revised: In Fig.

Original: 2,we ﬁrst observed that choosing  high step size constant η accelerates the training speed mostly in theearly phase.
Revised: 2, we ﬁrst observed that choosing a high step size constant η accelerated the training speed mainly in the early phase.

Original: The results also showed that the performance of MD ( D Ω ( π t % π E ) ) outperformed thatof referenced maximum likelihood estimation ( D Ω (¯ π E , t % π E ) ) by choosing an effectively low stepsize.
Revised: The results also showed that the performance of MD ( D Ω ( π t (cid:107) π E ) ) outperformed that of referenced maximum likelihood estimation ( D Ω (¯ π E ,t (cid:107) π E ) ) by choosing an effectively low step size.

Original: This  suggests that there are clear advantages in  scheduling  step sizes by time steps especiallyfor unreliable  ¯ π E ,t .
Revised: This empirical evidence suggests that there are clear advantages in formalization of the training steps and scheduling the step sizes, especially for unreliable statistical model ¯ π E ,t .

Original: MD update rules.
Revised: MD update rules.

Original: As a result, we formulate subsequent MD steps for a policy dstistribution.
Revised: As a result of these ﬁndings, we formulate subsequent MD steps with a regularized reward function.

Original: Letw t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions F at the t -th step.
Revised: Let w t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions F at the t -th step.

Original: Replacing the L2 proximity term of proximal gradient descent with a Bregmandivergence, the proximal form of the MD update for Eq.
Revised: Replacing the L2 proximity term of proximal gradient descent with a Bregman divergence, the proximal form of the MD update for Eq.

Original: (1) is written as [44]
Revised: (1) is written as [45] minimize w ∈W

Original: [Equation]
Revised: [Equation]

Original:  where α t := 1 / η t denotes an inverse of the current step size η t [45].
Revised: (6) where α t := 1 / η t denotes an inverse of the current step size η t [46].

Original: Plugging each divergences of thecumulative cost F t to Eq.
Revised: Plugging each divergence of the cumulative cost F t to Eq.

Original: (6), the MD update for the subsequent reward function ψ t +1 =
Revised: (6), the MD-IRL update for the subsequent reward function ψ t +1 =

Original: Ψ Ω ( π t +1 )is derived by solving a problemminimize
Revised: Ψ Ω ( π t +1 ) is derived by solving a problem minimize

Original: [Equation]
Revised: [Equation]

Original: + α t D Ω ! π s .. π s t " ⇐⇒minimize π s ∈ Π s
Revised: ⇐⇒

Original: D Ω ! π s .. ¯ π s E ,t " − D Ω !
Revised: [Equation]

Original: π s .. π st " + α t D Ω ! π s .. π st " ⇐⇒minimize π s ∈ Π s η t D Ω !
Revised: ⇐⇒

Original: π s .. ¯ π s E ,t "   estimated expert + (1 − η t ) D Ω ! π s .. π st "   learning agent
Revised: [Equation]

Original: ∀ s ∈ S , (7) where the gradient of D Ω is taken with respect to its ﬁrst argument π st .
Revised: where the gradient of D Ω is taken with respect to its ﬁrst argument π st .

Original: Note that solving theoptimization Eq.
Revised: Note that solving the optimization Eq.

Original: (7) requires interaction between π t and the dynamics of the given environmentin order to minimize F t ; thus, the corresponding RL process plays an essential role in sequentiallearning by the induced the value measures.
Revised: (7) requires interaction between π t and the dynamics of the given environment in order to minimize F t ; thus, the corresponding RL process plays an essential role in sequential learning by the induced the value measures.

Original: At a glance the objective is analogous to ﬁnding aninterpolation at each iteration where the point is controlled η t .
Revised: At a glance, the objective is analogous to ﬁnding an interpolation at each iteration where the point is controlled η t .

Original: Fig.
Revised: Fig.

Original: 3 shows that the uncertainty ofπ t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region). 
Revised: 3 shows that the uncertainty of π t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region). a

Original: [Figure]
Revised: G r ound - t r u t h c o s t MDwith η t b G r ound - t r u t h c o s t MDwith η t +

Original:  Convergence Analyses
Revised:  Convergence Analyses

Original: In this section, we introduce our theoretical results.
Revised: In this section, we present our theoretical results.

Original: The main goals  are to address (1) the convergence of MD updates for various cases and (2) the necessity of scheduling the amount of learning.NEW [ Mged ]
Revised: The main goals of the following arguments are to address 1 (cid:13) the convergence of MD updates for various cases and 2 (cid:13) the necessity of scheduling the amount of learning.

Original: Suppose that states at the t -th step s ( t ) i ∈
Revised: Suppose that state instances of s ( t ) i ∈

Original: τ t covers S by executing the policy π t in an inﬁnite horizon.
Revised: τ t cover the entire S by executing the policy π t in an inﬁnite horizon.

Original: From this assumption, we deﬁne a temporal cost function at the time step t as
Revised: From this assumption, we deﬁne a temporal cost function at the time step t :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: thatadditionally involves  a trajectory τ t as an input.
Revised: that involves π t , and additionally a trajectory τ t as inputs.

Original: We refer to the global objective as ﬁnding aunique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :
Revised: We refer to the global objective as ﬁnding a unique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :

Original: =
Revised: =

Original: E [ f ( π , τ t )] , where the expectation istaken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π , τ t )] .
Revised: E [ f ( π, τ t )] , where the expectation is taken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π, τ t )] .

Original: Taking the (stepwise) gradient foreach π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω ( π ∗ ( ·| s ))
Revised: Taking the (stepwise) gradient for each π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω( π ∗ ( ·| s ))

Original: − ∇ Ω (¯ π E ,t ( ·| s ))]
Revised: − ∇ Ω(¯ π E ,t ( ·| s ))]

Original: = 0 when t → ∞ ;196 hence, ∇ Ω ( π ∗ ( ·| s )) =
Revised: = 0 when t → ∞ ; hence, ∇ Ω( π ∗ ( ·| s )) =

Original: lim t →∞ E [ ∇ Ω (¯ π E , t ( ·| s ))] .
Revised: lim t →∞ E [ ∇ Ω(¯ π E ,t ( ·| s ))] .

Original: Introducing the optimal policy π ∗ allows not only197 the speciﬁc situation when (1) π 
Revised: Introducing the optimal policy π ∗ allows not only the speciﬁc situation when 1 (cid:13) π E

Original: E = π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually conver- 198 gent with t → ∞ , but also more general situations where (2) π E / ∈ Π or the estimated expert policy¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.
Revised: = π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually convergent with t → ∞ , but also more general situations where 2 (cid:13) π E / ∈ Π or the estimated expert policy ¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.

Original: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.
Revised: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.

Original: • Convergent sequence & divergent series:
Revised: • Convergent sequence & divergent series:

Original: [Equation]
Revised: [Equation]

Original: • Divergent series & convergent series of squared terms
Revised: • Divergent series & convergent series of squared terms:

Original: [Equation]
Revised: [Equation]

Original: We assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.
Revised: Let us assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.

Original: In some Ω , thesetwo assumptions do not necessarily hold for extreme cases in ∆ SA , e.g., a distribution that π ( a | s ) = 0for some entries.
Revised: In some Ω , these two assumptions do not necessarily hold for extreme cases in ∆ S A , e.g., a distribution that π ( a | s ) = 0 for some entries.

Original: Nevertheless, these outliers can be leaved out if the parametrization is constrainedto satisfy the assumptions.
Revised: Nevertheless, these outliers can be left out if the parametrization is constrained to satisfy the assumptions.

Original: For example, one can either (1) prevent a policy to have non-zero entriesof probabilities for a discrete policyby a constraint on its parameters, or (2) prevent  having too lowvalue of entropy for a continuous policy,typically by a constraint on the covariance functions.
Revised: For example, one can either 1 (cid:13) prevent a policy from having non-zero entries of probabilities for a discrete policy or 2 (cid:13) prevent a policy from having too low entropy for a continuous policy, by enforcing certain constraints on its parametric representations.

Original: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.
Revised: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: Theorem 1 (Stepsize considerations) .
Revised: Theorem 1 (Stepsize considerations) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, andthe associated Bregman divergence D Ω is bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergence D Ω is bounded.

Original: Assume a general condition of the problem thatinf π [Equation] Π E + f
Revised: Assume a general condition of the problem that inf π ∈ Π E (cid:2) f

Original: [Equation] [Equation] , > 0 .
Revised: ( π, τ t ) (cid:3) > 0 .

Original: Then we get lim T →∞ E τ 
Revised: Then we get lim T →∞ E τ 1: T (cid:2)(cid:80) ∞

Original: [Equation]{+i =
 D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)+}
Revised: {+i =
 D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)+}

Original:  T [Equation] = 

Revised: π T ( ·| s i )) (cid:3) =

Original: if and only if
Revised: if and only if

Original: Eq.
Revised: Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T 
Revised: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T (cid:2) f T ( π T , τ T

Original: [Equation]
Revised: [Equation]

Original: (b) If the step size takes the form of η t = 4 t [-+
-] then E τ 1: T +% ∞
Revised: (b) If the step size is in the form of η t = 4 t [-+
-]+1 , then E τ 1: T (cid:2) (cid:80) ∞

Original: i {+=
+}
Revised: i {+=
+}

Original: [Equation] 0 D Ω ! π [Equation]
Revised: D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)

Original: ( ·| s i ) .. π T ( ·| s i )) , = O (1 /T ) .
Revised: π T ( ·| s i )) (cid:3) = O (1 /T ) .

Original: Next, we provide Theorem 2, which addresses the convergence in a speciﬁc case when π E resides inΠ .
Revised: Next, we present Theorem 2, which addresses the convergence in a speciﬁc case when π E resides in Π .

Original: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .
Revised: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .

Original: Theorem 2 (Optimal cases) .
Revised: Theorem 2 (Optimal cases) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.

Original: Assume π 1 ∕ = π E and inf π ∈ Π E [ f ( π , τ t )]
Revised: Assume π 1 (cid:54) = π E and inf π ∈ Π E [ f ( π, τ t )]

Original: = 0 .
Revised: = 0 .

Original: Then, E + f ( π t , τ t ) , = 

Revised: Then, E (cid:2) f ( π t , τ t ) (cid:3) =

Original: if and only if [Equation] = ∞ .
Revised: if and only if (cid:80) ∞ t =1 η t = ∞ .

Original: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that
Revised: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that

Original: [Equation]
Revised: [Equation]

Original: , for
Revised: T

Original: [Equation]
Revised: [Equation]

Original: sup [Equation] FIX [ Ptuv ] Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of thealgorithm by imposing the stronger condition of step size in Eq.
Revised: Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of the algorithm by imposing the stronger condition of step size in Eq.

Original: The proofs are in the appendix.
Revised: The proofs are in Appendix A.

Original: Proposition 1 (General cases) .
Revised: Proposition 1 (General cases) .

Original: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π , τ t )] > 0 .
Revised: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π, τ t )] > 0 .

Original: If the step sizessatisﬁes Eq.
Revised: If the step sizes satisﬁes Eq.

Original: (10), then lim t →∞ [Equation]{+(cid:80) ∞ i =
+}
Revised: (10), then lim t →∞ {+(cid:80) ∞ i =
+}

Revised: γ i D Ω (cid:0) π ∗ ( · | s i ) (cid:13)(cid:13)

Original: " converges to 0 almost surely.
Revised: π t ( · | s i ) (cid:1) converges to 0 almost surely.

Original: Regrets.
Revised: Regrets.

Original: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as
Revised: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as

Original: [Equation]
Revised: [Equation]

Original: 1 t % tj =

Original: [Equation] f ( [Equation] In the optimal case of inf π ∈ Π E [ f ( π , τ t )] = 0 , the cost f inherits the property of Bregman divergenceso that the inﬁmum is achieved by 0 at π E .
Revised: In the optimal case of inf π ∈ Π E [ f ( π, τ t )] = 0 , the cost f inherits the property of Bregman divergence so that the inﬁmum is achieved by 0 at π E .

Original: In this case, the regret is bounded to O (1 /T ) by thetheorems.
Revised: In this case, the regret is bounded to O (1 /T ) by the theorems.

Original: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π , τ t )] > 0 whenthe step sizes abide by Eq.
Revised: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π, τ t )] > 0 when the step sizes abide by Eq.

Original: Thus, the regret is bounded to O (1 /T ) even for the general case.
Revised: Thus, the regret is bounded to O (1 /T ) even for the general case.

Original:  Algorithm: MD-IRL on an Adversarial Framework
Revised:  Algorithm: MD-IRL on an Adversarial Framework

Original: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized rewardfunction with adversarial learning and the MD update rule.
Revised: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized reward function with adversarial learning and the MD update rule.

Original: Neural network parameters θ , φ , and ν FIXare newly presented representing agent policy, reward, and expert policy functions respectively.
Revised: Neural network parameters θ , φ , and ν are newly presented representing agent policy, reward, and expert policy functions respectively.

Original: Dual discriminators.
Revised: Dual discriminators.

Original: In order to bridge the gap between theory and practice, we propose a novel NEW [ 4Ddj , Mged ]discriminative architecture, motivated by GAN studies regarding multiple discriminators [46, 47].
Revised: In order to bridge the gap between theory and practice, we propose a novel discriminative architecture, motivated by GAN studies regarding multiple discriminators [47, 48].

Original: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densitiesand imitating speciﬁc behavior.
Revised: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densities and imitating speciﬁc behavior.

Original: Given a learning agent policy π θ , an estimation policy π ν , and adiscriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as
Revised: Given a learning agent policy π θ , an estimation policy π ν , and a discriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as

Original: D ν
Revised: D ν

Original: ( s, a ; θ , ξ ) =
Revised: ( s, a ; θ , ξ ) =

Original: σ ! [Equation] ( a | s [Equation]
Revised: σ (cid:0) log (cid:8) π ν ( a | s ) (cid:14) π θ ( a | s ) (cid:9) + d ξ ( s ) (cid:1) and D ξ ( s ) =

Original: + d[Equation] ( s ) " and D ξ ( s ) = σ !
Revised: σ (cid:0) d ξ ( s ) (cid:1) , ∀ s ∈ S , a ∈ A , where σ ( · ) denotes the sigmoid function.

Original: d ξ ( s
Revised: The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:

Original: [Equation]
Revised: [Equation]

Original: " , ∀ s

Original: [Equation]
Revised: [Equation]

Original: S , a ∈ A , where σ ( · ) denotes the sigmoid function. The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:maximize J d ξ = E π E + log D ξ ( s ) , + E π φ + log(1 − D ξ ( s )) , , (12) maximize J π ν =
Revised: [Table]

Original: E π E + log D ν ( s, a ) , + E π φ + log(1 − D ν ( s, a )) , , (13)
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: α t η t− η t t αη example of ( α t , η t ) .where d ξ and π θ are not trained for learning D ν .
Revised: α t η t 1 − η t t αη where d ξ and π θ are not trained for learning D ν .

Original: Let ρ π ∈ ∆ S denote the state visitation density ofπ , which is deﬁned as ρ π ( s ) :
Revised: Let ρ π ∈ ∆ S denote the state visitation density of π , which is deﬁned as ρ π ( s ) :

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E π {+(cid:2)(cid:80) ∞ i =
 γ i+}
Revised: E π {+(cid:2)(cid:80) ∞ i =
 γ i+}

Original: [Equation] where I {·} is an indicator function.
Revised: I { s i = s } (cid:3) , where I {·} is an indicator function.

Original: The convergence of functions in an ideal case is found at π ν = π [Equation] and d [Equation]
Revised: The convergence of functions in an ideal case is found at π ν = π E and d ξ ( s )=log { ρ π E ( s ) /ρ

Original: ( s

Original: [Equation]

Original: =

Original: [Equation]

Original:  s [Equation]
Revised: π ( s ) } .

Original: Learning with MD-based rewards.
Revised: Learning with MD-based rewards.

Original: Based on the MD solution for a regularized reward function, NEW [ 4Ddj , Mged ]we focus on developing a learning objective.
Revised: Based on the MD solution for a regularized reward function, we focus on developing an MD-based learning objective.

Original: Let ψ φ ∈ Ψ ( Π ) denote a parameterized regularized reward function and π φ denotes a corresponding policy from φ .
Revised: Let ψ φ ∈ Ψ(Π) denote a parameterized regularized reward function, and π φ denotes a corresponding policy from φ .

Original: Note that the transformation between NEW [ 4Ddj ]ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁcparameterizations [17].
Revised: Note that the transformation between ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁc parameterizations [18].

Original: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν ,we deﬁne the objective of φ as  direct interpretation of the update rule of Eq.
Revised: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν , we deﬁne the objective of φ as a direct interpretation of the update rule of Eq.

Original: [Equation]
Revised: [Equation]

Original: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.
Revised: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.

Original: As shown in
Revised: As shown in

Original: Fig. 4, η t is adjusted by linearly increasing α t ,  originated from the analyses in Section 5.
Revised: Fig. 4, η t is adjusted by linearly increasing α t , which originated from the analyses in Section 5.

Original: One drawback of Eq. (14) is the issue of covariance shifts, and it is likely that state densities between the expert and the agent are misaligned.
Revised: Another important consideration is the way of handling covariate shifts [49] since it is likely that state densities between the expert and the agent are misaligned.

Original: Thus, we deﬁne the IRL reward function as linear FIX [ 4Ddj , Mged ]combinations of ψ φ and the state density discriminative signal:
Revised: Thus, we deﬁne the IRL reward function as linear combinations of ψ φ and the state density discriminative signal:

Original: [Equation]
Revised: [Equation]

Original: with a coefﬁcient λ ∈ R + .
Revised: with a coefﬁcient λ ∈ R + .

Original: Utilizing an arbitrary regularized RL algorithm  a regularizer λ Ω ( · ), thereward learning regarding agent policy π θ is decomposed into the following:
Revised: Utilizing an arbitrary regularized RL algorithm with a regularizer λ Ω( · )

Original: [Equation]
Revised: [Equation]

Original: Minimizing the ﬁrst term of E π θ 
Revised: the reward learning regarding agent policy π θ is decomposed into the following:

Original: [Equation]
Revised: [Equation]

Original: Ω [Equation] s θ  π s φ ) , represents learning with the MD formulation.
Revised: Minimizing the ﬁrst term of E π θ (cid:2) D Ω ( π sθ (cid:107) π sφ ) (cid:3) represents learning with the MD formulation.

Original: Meanimizing the second term D KL ( ρ π [Equation] [Equation] π E ) plays an auxiliary role in facilitating the supports of statevisitation densities to be properly matched.
Revised: Meanimizing the second term D KL ( ρ π θ (cid:107) ρ π E ) plays an auxiliary role in facilitating the supports of state visitation densities to be correctly matched.

Original: With the hyperparameter λ , we report that learningthe second term is helpful when the state densities are heavily misaligned in certain benchmarks.
Revised: With the hyperparameter λ , we report that learning the second term is helpful when the state densities are heavily misaligned in certain benchmarks.

Original: NEW [ 4Ddj , Mged ]

Original: Algorithm 1 summarizes the entire procedure.
Revised: Algorithm 1 summarizes the entire procedure.

Original: We defer additional details to Appendix B and C.
Revised: We defer additional details to Appendices B and C.

Original:  Experimental Results
Revised:  Experimental Results

Original: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ωwhile retaining  scalability of AIL.
Revised: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ω while retaining the scalability of AIL.

Original: The comparative method was RAIRL with density based models(RAIRL-DBM) which contained comparable expressiveness as MD-AIRL For RL, we used RAC algorithm [43] that is a generalization of the SAC algorithm[48].
Revised: The comparative method was RAIRL with density-based models (RAIRL-DBM) which contained comparable expressiveness as MD-AIRL. For RL, we used RAC [44], which is a generalization of the SAC algorithm

Original: We considered a class of regularizersΩ ( p ) =
Revised: We considered a class of regularizers Ω( p ) =

Original: − E x ∼ p
Revised: − E x ∼ p

Original: [ ϕ ( p ( x ))] with (1) Shannon ( ϕ ( x ) = log( x ) ), (2) Tsallis ( ϕ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) ,q = 2 by default), (3) exp ( ϕ ( x )=
Revised: [ ϕ ( p ( x ))] with 1 (cid:13) Shannon ( ϕ ( x ) = log( x ) ), 2 (cid:13) Tsallis ( ϕ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) , q = 2 by default), 3 (cid:13) exp ( ϕ ( x )=

Original: e − e x ), (4) cos ( ϕ ( x )=cos( π 2 x ) ), and (5) sin ( ϕ ( x )= 1 − sin( π 2 x ) ).
Revised: e − e x ), 4 (cid:13) cos ( ϕ ( x )= cos( π 2 x ) ), and 5 (cid:13) sin ( ϕ ( x )=1 − sin( π 2 x ) ).

Original: 7.1 Large scale multiarmed bandits
Revised: 7.1 Large scale multiarmed bandits

Original: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the carNEW [ Mged ]dinality of action spaces varies largely.
Revised: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the cardinality of action spaces varies largely.

Original: Learning the optimal distribution of π E becomes challengingas the size of action space |A| increases, because the frequency of each sample becomes sparse due275 to the curse of dimensionality.
Revised: Learning the optimal distribution of π E becomes challenging as the cardinality of the space |A| increases, because the frequency of each sample becomes sparse due to the curse of dimensionality.

Original: The stateless expert distribution π E was generated by the parametersof softmax distribution π E ( a ) =
Revised: The stateless expert distribution π E was generated by the parameters of softmax distribution π E ( a ) =

Original: exp( z a ) / ! i exp( z i )  where the logits z i were randomly initialized.
Revised: exp( z a ) / (cid:80) i exp( z i ) , where the logits z i were randomly initialized.

Original: Weset the action size to |A| =
Revised: We set the action size to |A| =

Original: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.
Revised: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when threedifferent cardinalities and ﬁve regularizers were considered.
Revised: [Figure] [Table] [Figure] Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when three different cardinalities and ﬁve regularizers were considered.

Original: Fig.
Revised: Fig.

Original: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1to be greater than 1 ( α 1 = 0 . 5 ).
Revised: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1 to be greater than 1 ( α 1 = 0 . 5 ).

Original: MD-AIRL exceeded the discriminative performance of RAIRLafter certain steps, while the progression of RAIRL mostly stopped at local minima.
Revised: MD-AIRL exceeded the discriminative performance of RAIRL after certain steps, while the progression of RAIRL mostly stopped at local minima.

Original: MD-AIRLoutperformed RAIRL in four cases byan choosing  effectively low step size at η T to be less than1 ( α T = 2 ).
Revised: MD-AIRL outperformed RAIRL in four cases by choosing an effectively low step size at η T to be less than 1 ( α T = 2 ).

Original: These results match the properties of MD algorithms and our convergence analyses.
Revised: These results match the properties of MD algorithms and our convergence analyses.

Original: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robustreward acquisition and imitation for situations when the total number of data samples is limited.a b G oa l en t r op y Reaching multigoals q fromTsallisentropies c r eg . r e w a r d
Revised: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robust reward acquisition and imitation for situations when the total number of data samples is limited.

Original: M D r e w a r d π ( ·| s )

Original: Shannon regularizer

Original: Step5K Step300K r eg . r e w a r d M D r e w a r d π ( ·| s )
Revised: [Figure]

Original: Tsallis regularizer

Original: Step5K

Original: Step300K

Original: 7.2 A continuous multigoal environment
Revised: 7.2 A continuous multigoal environment

Original: We then considered a multigoal environment.
Revised: We then considered a multigoal environment.

Original: In this environment, an agent is a two-dimensionalpoint mass initialized at the origin, and the four goals are located in the four cardinal directions.
Revised: In this environment, an agent is a two-dimensional point mass initialized at the origin, and the four goals are located in the four cardinal directions.

Original: Theobjective of imitation learning is to go toward each direction evenly as possible where the expertmodel was trained by an SAC algorithm.
Revised: The objective of imitation learning is to go toward each direction evenly as possible where the expert model was trained by the SAC algorithm.

Original: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with fullcovariance matrices instead of conventional diagonal Gaussian policies (see Appendices 2 and 3).
Revised: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with full covariance matrices instead of conventional diagonal Gaussian policies (see Appendices B and C).

Original: Fig.
Revised: Fig.

Original: 6 (a) shows trajectories generated by the trained agent.
Revised: 6 (a) shows trajectories generated by the trained agent.

Original: Fig.
Revised: Fig.

Original: 6 (b) shows that MD-AIRL achievedhigher entropy for reaching the multiple goals.
Revised: 6 (b) shows that MD-AIRL achieved higher entropy for reaching the multiple goals.

Original: Fig.
Revised: Fig.

Original: 6 (c) shows reward surfaces with regularizers,which were calculated by ψ φ ( s, a )
Revised: 6 (c) shows reward surfaces with regularizers, which were calculated by ψ φ ( s, a )

Original: + ϕ
Revised: + ϕ

Original: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .
Revised: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .

Original: Duringthe training, the MD reward was similar to the estimated ground truth using adversarial training.
Revised: During the training, the MD reward was similar to the estimated ground truth using adversarial training.

Original: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t wassufﬁciently close to the expert behavior.
Revised: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t was sufﬁciently close to the expert behavior.

Original: As a result, we claim that a drastic change in the targetdistribution, which is one of the typical characteristics of adversarial frameworks, are prevented.
Revised: As a result, we claim that a drastic change in the target distribution, which is one of the typical characteristics of adversarial frameworks, is prevented.

Original: Weargue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.
Revised: We argue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.

Original: 7.3 A continuous control benchmark: MuJoCo
Revised: 7.3 A continuous control benchmark: MuJoCo

Original: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.
Revised: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.

Original: We assumedfull covariance Gaussian policies for both learner’s policy π and expert policy π E .
Revised: We assumed full covariance Gaussian policies for both learner’s policy π and expert policy π E .

Original: We used the hyperbolized environment assumption [17] where the action constraint is incorporated to the dynamicsas a part of the environment using hyperbolic tangent activation.
Revised: We used the hyperbolized environment assumption [18] where the action constraint is incorporated into the dynamics as a part of the environment using hyperbolic tangent activation.

Original: Sample efﬁciency.
Revised: Sample efﬁciency.

Original: For each task, we considered two different numbers of episodes collected byan expert policy.
Revised: For each task, we considered two different numbers of episodes collected by an expert policy.

Original: In Fig.
Revised: In Fig.

Original: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [19] is shown with the expert and random agent performance.
Revised: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [20] is shown with the expert and random agent performance.

Original: MD-AIRL was ableto achieve Hopper-v Walker2d-v HalfCheetah-v
Revised: MD-AIRL was able

Original: Ant-v

Original: [Figure]
Revised: [Figure]

Original: random The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.
Revised: The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.

Original: [Figure]
Revised: [Figure]

Original: HalfCheetah Ant consistent performance throughout the tasks and demonstrationsize. On the training curves, MD- AIRL showed high tolerance to the scarcity ofdata compared to RAIRL for 4 expert demonstra- tions.
Revised: to achieve consistent performance throughout the tasks and demonstration showed high tolerance to the scarcity of to RAIRL demonstrations.

Original: The plots in Fig.
Revised: The plots in Fig.

Original: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among32 different conﬁgurations we have tested
Revised: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among we have tested.

Original: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.
Revised: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Noisy demonstrations.
Revised: Noisy demonstrations.

Original: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with twodifferent levels of Gaussian additive noises, resulting to suboptimaldemonstrations.
Revised: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with two levels of Gaussian additive noises, resulting in suboptimal demonstrations.

Original: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.
Revised: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.

Original: The experiment is closely related to the general case in the theory the results suggest that thecharacteristics of MD-AIRL are in alignment with our analyses of the MD reward learning scheme
Revised: The experiment is closely related to the general case in the theory; the results suggest that the characteristics of MDAIRL are in alignment with our analyses of the MD reward learning scheme. Shannon

Original: [Table]
Revised: [Figure]

Revised: We present a detailed analysis of the noisy demonstration experiments Fig.

Revised: Let the Bregman divergence between agent and ground-truth expert policies be the error, and we measured these errors by increasing the given noise level for the expert trajectories.

Revised: Fig.

Revised: 9 shows a general tendency that MD-AIRL has lower errors than RAIRL.

Revised: With Tab. 3 and Fig.

Revised: 9, we were able to ﬁnd the evident correlation between average Bregman divergence and performance since imitation learning convergence when the divergence is zero.

Revised: Therefore, this is another piece of empirical evidence that veriﬁes our theoretical claims.

Original:  Conclusions and Discussion
Revised:  Conclusions and Discussion

Original: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitationlearning problem in the real world.
Revised: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitation learning problem in the real world.

Original: We proved that the proposed method has clear advantages overprevious AIL methods in terms of robustness.
Revised: We proved that the proposed method has clear advantages over previous AIL methods in terms of robustness.

Original: We veriﬁed MD-AIRL on a variety of situations,including high-dimensional spaces, limited samples, and imperfect demonstrations.
Revised: We veriﬁed MD-AIRL in a variety of situations, including high-dimensional spaces, limited samples, and imperfect demonstrations.

Original: The empiricalevidence showed that MD-AIRL outperforms previous methods on various benchmarks.
Revised: The empirical evidence showed that MD-AIRL outperforms previous methods on various benchmarks.

Original: We conclude that the rich foundation of optimization theories shows a promising direction to AIL studies.
Revised: We conclude that the rich foundation of optimization theories shows a promising direction for AIL studies.

Original: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.
Revised: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.

Original: Although our work covers various imitation learning problems with the Bregman divergence, this doesnot include some other problems when the proximity term is of other statistical divergence familiessuch as the f-divergence [50].
Revised: Although our work covers various imitation learning problems with the Bregman divergence, this does not include some other problems when the proximity term is of other statistical divergence families, such as the f-divergence [52].

Original: If the relationship between these classes of divergences is studiedin more detail, it is expected to proceed  applications to various subﬁelds of machine learning.
Revised: If the relationship between these classes of divergences is studied in more detail, it is expected to proceed with applications to various subﬁelds of machine learning.

Original: The

Original: NEW [ Ptuv , Mged ]assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but someoutliers might have substantial meaning for certain tasks.
Revised: The assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but some outliers might have substantial meaning for certain tasks.

Original: Therefore, extensive analyses on theseparameterizations remain as future works.
Revised: Therefore, extensive analyses on these parameterizations remain as future works.

Original: The “impurity” of Eq. (15) compared to Ψ Ω ( Π ) can also NEW [ Ptuv , Mged ]be regarded as a limitation.
Revised: The “impurity” of the MD-AIRL reward function compared to Ψ Ω (Π) can be regarded as a limitation.

Original: To fully resolve this issue, all data must be treated as on-policy samples,which might require a sophisticated sampling mechanism on top of the adversarial framework.
Revised: To fully resolve this problem, all data must be treated as on-policy samples, which might require a sophisticated sampling mechanism. Societal impacts.

Original: Checklist
Revised: The evolution of imitation learning algorithms is expected to bring a structural shift in the labor market. The negative impact could be mitigated by diversiﬁcation, uniﬁcation, and redeﬁnition of routine and manual jobs. The results of our work can be abused as a tool for analyzing individual data. Therefore, we stress that certain acts should be carefully regulated, such as collecting a substantial amount of individuals’ data and aggressively tracking personal identity.

Original: For all authors...
Revised: The authors would like to thank the anonymous reviewers, Woosuk Choi, Jaein Kim, and Min Whoo Lee for their helpful discussion and comments. This work was partly supported by the

Revised: IITP (2022-0-00951-LBA/25%, 2022-0-00953-PICA/25%, 2015-0-00310-SW.StarLab/10%, 2021-002068-AIHub/10%, 2021-0-01343-GSAI/10%, 2019-0-01371-BabyMind/10%) grant funded by the

Revised: Korean government, and the CARAI (UD190031RD/10%) grant funded by the DAPA and ADD.

Original:  (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A. 
Revised: [Yes] See Appendix A. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?[N/A]
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original:  Our empricalstudies can be reproduced by from the detailed information in Appendices B& C. Thecode and data will be available in the ﬁnal version.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] Our empirical studies can be reproduced by from the detailed information in Appendices B and C. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 7.
Revised: [Yes] See Section 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: In experiments, each algorithmwas exectued in CPU (a single thread).
Revised: In experiments, each algorithm was executed in CPU (a single thread).

Original: If you are using existing assets (e g , code, data, models) or curating/releasing new assets. (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: The MuJoCo simulator used in our[Table] experiments is freely available to everyone.
Revised: The MuJoCo simulator used in our experiments is freely available to everyone.

Original: See the site (https://mujoco.org).
Revised: See the site (https://mujoco.org).

Original: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] [Table]
Revised: [No] (d) Did you discuss whether

Original: We are preparing to release the source code and data in the ﬁnal version.

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data[Table] you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁ- [Table] able information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable?

Original: [N/A]

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]