Original: Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Revised: Robust Imitation via Mirror Descent Inverse Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems.
Revised: Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems.

Original: However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly.
Revised: However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly.

Original: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem.
Revised: Inspired by a ﬁrst-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem.

Original: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints.
Revised: IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints.

Original: We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes { η t } Tt =1 .
Revised: We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for step sizes { η t } Tt =1 .

Original: Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.
Revised: Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.

Original:  Introduction
Revised:  Introduction

Original: One crucial requirement of practical imitation learning methods is robustness , often described as learning expert behavior for a ﬁnite number of demonstrations, overcoming various realistic challenges [1].
Revised: One crucial requirement of practical imitation learning methods is robustness , often described as learning expert behavior for a ﬁnite number of demonstrations, overcoming various realistic challenges [1].

Original: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations can be noisy or suboptimal to solve the problem [3].
Revised: In real-world problems such as motor control tasks, the demonstration size can be insufﬁcient to create a precise model of an expert [2] and even in some cases, demonstrations can be noisy or suboptimal to solve the problem [3].

Original: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling the uncertainty of estimated cost functions dramatically affects imitation performance.
Revised: For such challenging scenarios, imitation learning algorithms inevitably struggle with unreliable statistical models; thus, the way of handling the uncertainty of estimated cost functions dramatically affects imitation performance.

Original: Therefore, a thorough analysis of addressing these issues is required to construct a robust algorithm.
Revised: Therefore, a thorough analysis of addressing these issues is required to construct a robust algorithm.

Original: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expert demonstrations where the expert acts optimally with respect to an unknown reward function [4, 5].
Revised: Inverse reinforcement learning (IRL) is an algorithm for learning ground-truth rewards from expert demonstrations where the expert acts optimally with respect to an unknown reward function [4, 5].

Original: Traditional IRL studies solve the imitation problem based on iterative algorithms [6, 7], alternating between the reward estimation process and a reinforcement learning (RL) [8] algorithm.
Revised: Traditional IRL studies solve the imitation problem based on iterative algorithms [6, 7], alternating between the reward estimation process and a reinforcement learning (RL) [8] algorithm.

Original: In contrast, newer studies of adversarial imitation learning (AIL) [9, 10] rather suggest learning reward functions of a certain form “directly,” by using adversarial learning objectives [11] and nonlinear discriminative neural networks [10].
Revised: In contrast, newer studies of adversarial imitation learning (AIL) [9, 10] rather suggest learning reward functions of a certain form “directly,” by using adversarial learning objectives [11] and nonlinear discriminative neural networks [10].

Original: Compared to classical approaches, the AIL methods have shown great success on control benchmarks in terms of scalability for challenging control tasks [12].
Revised: Compared to classical approaches, the AIL methods have shown great success on control benchmarks in terms of scalability for challenging control tasks [12].

Original: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [13].
Revised: Technically, it is well known that AIL formulates a divergence minimization problem with its discriminative signals, which incorporates ﬁne-tuned estimations of the target densities [13].

Original: Through the lens of differential geometries, the limitation of AIL naturally comes from the implication that minimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [14].
Revised: Through the lens of differential geometries, the limitation of AIL naturally comes from the implication that minimizing the divergence does not guarantee unbiased progression due to constraints of the underlying space [14].

Original: In order to ensure further stability, we argue that an IRL algorithm’s progress needs to be regulated, yielding gradual updates with respect to local geometries of policy distributions.
Revised: In order to ensure further stability, we argue that an IRL algorithm’s progress needs to be regulated, yielding gradual updates with respect to local geometries of policy distributions.

Original: We claim that there are two issues leading to unconstrained policy updates: 1 (cid:13) a statistical divergence often cannot be accurately obtained for challenging problems, and 2 (cid:13) an immediate divergence between agent and expert densities does not guarantee unbiased learning directions.
Revised: We claim that there are two issues leading to unconstrained policy updates: 1 (cid:13) a statistical divergence often cannot be accurately obtained for challenging problems, and 2 (cid:13) an immediate divergence between agent and expert densities does not guarantee unbiased learning directions.

Original: Our approach is connected to a collection of optimization processes called mirror descent (MD) [15].
Revised: Our approach is connected to a collection of optimization processes called mirror descent (MD) [15].

Original: For a sequence of parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as
Revised: For a sequence of parameters { w t } Tt =1 and a convex function Ω , an MD update for a cost function F t is derived as

Original: [Equation]
Revised: [Equation]

Original: In the equation, the gradient ∇ Ω( · ) creates a transformation that links a parametric space to its dual space.
Revised: In the equation, the gradient ∇ Ω( · ) creates a transformation that links a parametric space to its dual space.

Original: Theoretically, MD is a ﬁrst-order method for solving constrained problems, which enjoys rigorous regret bounds for various geometries [16, 17] including probability spaces.
Revised: Theoretically, MD is a ﬁrst-order method for solving constrained problems, which enjoys rigorous regret bounds for various geometries [16, 17] including probability spaces.

Original: Thus, applying
Revised: Thus, applying

Original: MD to the reward estimation process can be efﬁcient in terms of the number of learning phases.
Revised: MD to the reward estimation process can be efﬁcient in terms of the number of learning phases.

Original: In this paper, we derive an MD update rule in IRL upon a postulate of nonstationary estimations of the expert density, resulting in convergent reward acquisition even for challenging problems.
Revised: In this paper, we derive an MD update rule in IRL upon a postulate of nonstationary estimations of the expert density, resulting in convergent reward acquisition even for challenging problems.

Original: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions on an alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).
Revised: Compared to MD algorithms in optimization studies, our methodology draws a sequence of functions on an alternative space induced by a reward operator Ψ Ω (Deﬁnition 1).

Original: To this end, we propose an
Revised: To this end, we propose an

Original: AIL algorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).
Revised: AIL algorithm called mirror descent adversarial inverse reinforcement learning (MD-AIRL).

Original: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)
Revised: Our empirical evidence showed that MD-AIRL outperforms the regularized adversarial IRL (RAIRL)

Original: [18] methods.
Revised: [18] methods.

Original: For example, MD-AIRL showed higher performance in 30 distinct cases amongdifferent conﬁgurations in challenging MuJoCo [19] benchmarks, and it also clearly showed higher tolerance to suboptimal data.
Revised: For example, MD-AIRL showed higher performance in 30 distinct cases amongdifferent conﬁgurations in challenging MuJoCo [19] benchmarks, and it also clearly showed higher tolerance to suboptimal data.

Original: All of these results are strongly aligned with our theoretical analyses.
Revised: All of these results are strongly aligned with our theoretical analyses.

Original: [Table]
Revised: [Table]

Original: Our contributions.
Revised: Our contributions.

Original: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithms as a combined optimization process with dual aspects.
Revised: Our work is complementary to previous IRL studies; the theoretical and technical contributions are built upon a novel perspective of considering iterative RL and IRL algorithms as a combined optimization process with dual aspects.

Original: Comparing MD-AIRL and RAIRL, both are highly generalized algorithms in terms of a variety of choices of divergence functions.
Revised: Comparing MD-AIRL and RAIRL, both are highly generalized algorithms in terms of a variety of choices of divergence functions.

Original: Tab.
Revised: Tab.

Original: 1 shows that
Revised: 1 shows that

Original: MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since our approach is more aligned with earlier theoretical IRL studies providing formalized reward learning schemes and convergence guarantees.
Revised: MD-AIRL brings beneﬁcial results in realistic situations of limited time and data, since our approach is more aligned with earlier theoretical IRL studies providing formalized reward learning schemes and convergence guarantees.

Original: In summary, we list our main contributions below:
Revised: In summary, we list our main contributions below:

Original: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence of reward functions that provides iterative local objectives (Section 4).
Revised: • Instead of a monolithic estimation process of a global solution in AIL, we derive a sequence of reward functions that provides iterative local objectives (Section 4).

Original: • We formally prove that rewards derived by an MD update rule guarantee the robust performance of divergence minimization along with a rigorous regret bound (Section 5).
Revised: • We formally prove that rewards derived by an MD update rule guarantee the robust performance of divergence minimization along with a rigorous regret bound (Section 5).

Original: • We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant of unreliable discriminative signals of the AIL framework (Sections 6 and 7).
Revised: • We propose a novel adversarial algorithm that is motivated by mirror descent, which is tolerant of unreliable discriminative signals of the AIL framework (Sections 6 and 7).

Original:  Related Works
Revised:  Related Works

Original: Mirror descent.
Revised: Mirror descent.

Original: We are interested in a family of statistical divergences called the Bregman divergence [21].
Revised: We are interested in a family of statistical divergences called the Bregman divergence [21].

Original: The divergence generalizes constrained optimization problems such as least squares [22, 23] [Table] and it also has been applied in various subﬁelds of machine learning [24, 25].
Revised: The divergence generalizes constrained optimization problems such as least squares [22, 23], and it also has been applied in various subﬁelds of machine learning [24, 25].

Original: In differential geometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁes metric-like properties [14, 26].
Revised: In differential geometries, the Bregman divergence is a ﬁrst-order approximation for a metric tensor and satisﬁes metric-like properties [14, 26].

Original: MD is also closely related to optimization methods regarding nonEuclidean geometries with a discretization of steps such as natural gradients [27, 28].
Revised: MD is also closely related to optimization methods regarding nonEuclidean geometries with a discretization of steps such as natural gradients [27, 28].

Original: In the primal space, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow [29, 30].
Revised: In the primal space, training with the inﬁnitesimal limit of MD steps corresponds to a Riemannian gradient ﬂow [29, 30].

Original: In the RL domain, MD has been recently studied for policy optimization [31–33].
Revised: In the RL domain, MD has been recently studied for policy optimization [31–33].

Original: In this paper, we focus on learning with suboptimal representations of policy, and our distinct goal is to draw a robust reward learning scheme based on MD for the IRL problem.
Revised: In this paper, we focus on learning with suboptimal representations of policy, and our distinct goal is to draw a robust reward learning scheme based on MD for the IRL problem.

Original: Imitation learning.
Revised: Imitation learning.

Original: As a statistical model for the information geometry [34], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient
Revised: As a statistical model for the information geometry [34], energy-based policies (i.e., Boltzmann distributions) appeared in early IRL studies, such as Bayesian IRL, natural gradient

Original: IRL, and maximum likelihood IRL [35–37] for modeling expert distribution to parameterized functions.
Revised: IRL, and maximum likelihood IRL [35–37] for modeling expert distribution to parameterized functions.

Original: Notably, MaxEnt IRL [7, 38] is one of the representative classical IRL algorithms based on an information-theoretic perspective toward IRL solutions.
Revised: Notably, MaxEnt IRL [7, 38] is one of the representative classical IRL algorithms based on an information-theoretic perspective toward IRL solutions.

Original: Also, discriminators of AIL are trained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy function that approximates the truth data density for the expert distribution [39].
Revised: Also, discriminators of AIL are trained by logistic regression; thus, the logit score of the discriminator deﬁnes an energy function that approximates the truth data density for the expert distribution [39].

Original: Other statistical entropies have also been applied to AIL, such as the Tsallis entropy [40].
Revised: Other statistical entropies have also been applied to AIL, such as the Tsallis entropy [40].

Original: On the one hand, our approach is closely related to RAIRL [18], which deﬁned its AIL objective using the Bregman divergence.
Revised: On the one hand, our approach is closely related to RAIRL [18], which deﬁned its AIL objective using the Bregman divergence.

Original: On the other hand, this work further employs the Bregman divergence to derive iterative MD updates for reward functions, resulting in theoretically pleasing properties while retaining the scalability of AIL.
Revised: On the other hand, this work further employs the Bregman divergence to derive iterative MD updates for reward functions, resulting in theoretically pleasing properties while retaining the scalability of AIL.

Original: Learning theory.
Revised: Learning theory.

Original: There have been considerable achievements in dealing with temporal costs { F t } ∞ t =1 , often referred to as online learning [41].
Revised: There have been considerable achievements in dealing with temporal costs { F t } ∞ t =1 , often referred to as online learning [41].

Original: The most ordinary approach is stochastic gradient descent (SGD): w t +1 =
Revised: The most ordinary approach is stochastic gradient descent (SGD): w t +1 =

Original: w t − η t ∇ F t ( w t ) .
Revised: w t − η t ∇ F t ( w t ) .

Original: In particular, SGD is a desirable algorithm when the parameter w t resides in the Euclidean space since it ensures unbiased minimization of the expected cost.
Revised: In particular, SGD is a desirable algorithm when the parameter w t resides in the Euclidean space since it ensures unbiased minimization of the expected cost.

Original: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may not be the direction of the steepest descent due to geometric constraints [27, 34].
Revised: Apparently, policies appear in geometries of probabilities; thus, an incurred gradient may not be the direction of the steepest descent due to geometric constraints [27, 34].

Original: An online form of MD in Eq.
Revised: An online form of MD in Eq.

Original: (1) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a Bregman divergence [30].
Revised: (1) is analogous to SGD for non-Euclidean spaces, where each local metric is speciﬁed by a Bregman divergence [30].

Original: Our theoretical ﬁndings and proofs follow the results of online mirror descent (OMD) that appeared in previous literature for general aspects [15, 16, 42, 28, 17, 30].
Revised: Our theoretical ﬁndings and proofs follow the results of online mirror descent (OMD) that appeared in previous literature for general aspects [15, 16, 42, 28, 17, 30].

Original: Our analyses extend existing theoretical results to IRL; at the same time, they are also highly general to cover various online imitation learning problems which require making decisions sequentially.
Revised: Our analyses extend existing theoretical results to IRL; at the same time, they are also highly general to cover various online imitation learning problems which require making decisions sequentially.

Original:  Background
Revised:  Background

Original: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional) probabilities over X (conditioned on Y ).
Revised: For sets X and Y , let Y X be a set of functions from X to Y and ∆ X ( ∆ YX ) be a set of (conditional) probabilities over X (conditioned on Y ).

Original: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ ) with the state space S , the action space A , the Markovian transition kernel P ∈ ∆ S×AS , the reward function r ∈ R S × A
Revised: We consider an MDP deﬁned as a tuple ( S , A , P, r, γ ) with the state space S , the action space A , the Markovian transition kernel P ∈ ∆ S×AS , the reward function r ∈ R S × A

Original: and the discount factor γ ∈ [0 , 1) .
Revised: and the discount factor γ ∈ [0 , 1) .

Original: Let a function Ω: ∆ A → R be strongly convex.
Revised: Let a function Ω: ∆ A → R be strongly convex.

Original: Using Ω , the Bregman divergence is deﬁned as
Revised: Using Ω , the Bregman divergence is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where π s and ˆ π s denote arbitrary policies for a given state s .
Revised: where π s and ˆ π s denote arbitrary policies for a given state s .

Original: For a representative divergence, one can consider the popular KL divergence.
Revised: For a representative divergence, one can consider the popular KL divergence.

Original: The KL divergence is a Bregman divergence when Ω is speciﬁed as the negative Shannon entropy: Ω( π s ) =
Revised: The KL divergence is a Bregman divergence when Ω is speciﬁed as the negative Shannon entropy: Ω( π s ) =

Original: (cid:80) a π s ( a ) ln π s ( a ) .
Revised: (cid:80) a π s ( a ) ln π s ( a ) .

Original: Regularization of policy distribution with respect to convex Ω brings distinct properties to the learning agent [43, 44].
Revised: Regularization of policy distribution with respect to convex Ω brings distinct properties to the learning agent [43, 44].

Original: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected value of discounted cumulative returns along with a causal convex regularizer Ω , i.e.,
Revised: The objective of regularized RL is to ﬁnd π ∈ Π that maximizes the expected value of discounted cumulative returns along with a causal convex regularizer Ω , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for the given
Revised: where the subscript π on the expectation indicates that each action is sampled by π ( ·| s i ) for the given

Original: MDP.
Revised: MDP.

Original: In this setup, a regularized RL algorithm ﬁnds a unique solution in a subset of the conditional probability space denoted as Π:=[Π s ] s ∈ S ⊂ ∆ SA constrained by the parameterization of a policy.
Revised: In this setup, a regularized RL algorithm ﬁnds a unique solution in a subset of the conditional probability space denoted as Π:=[Π s ] s ∈ S ⊂ ∆ SA constrained by the parameterization of a policy.

Original: The objective of IRL is to ﬁnd a function r E that rationalizes the behavior of an expert policy π E .
Revised: The objective of IRL is to ﬁnd a function r E that rationalizes the behavior of an expert policy π E .

Original: For an inner product (cid:104)· , ·(cid:105) A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :
Revised: For an inner product (cid:104)· , ·(cid:105) A , consider Ω ∗ , the Legendre-Fenchel transform (convex conjugate) of Ω :

Original: [Equation]
Revised: [Equation]

Original: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .
Revised: where q s and π s denote the shorthand notation of q ( s, · ) and π ( ·| s ) .

Original: Differentiating both sides with respect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.
Revised: Differentiating both sides with respect to q s , the gradient of conjugate ∇ Ω ∗ maps q s to a policy distribution.

Original: One fundamental property in regularized IRL [43] is that π E is the maximizing argument of Ω ∗ for q E , where q E is the regularized state-action value function q E ( s, a )=
Revised: One fundamental property in regularized IRL [43] is that π E is the maximizing argument of Ω ∗ for q E , where q E is the regularized state-action value function q E ( s, a )=

Original: E π E [ (cid:80) ∞
Revised: E π E [ (cid:80) ∞

Original: i =

Revised: i =

Original: γ i { r E ( s, a ) − Ω( π s E ) }| s 

Revised: γ i { r E ( s, a ) − Ω( π s E ) }| s

Original: = s, a 

Revised: = s, a

Original: = a ] .
Revised: = a ] .

Original: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E = ∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.
Revised: Note that the problem is ill-posed, and every ˆ r E that makes its value function ˆ q E satisfy π s E = ∇ Ω ∗ (ˆ q s E ) ∀ s ∈ S is a valid solution.

Original: Addressing this issue, Jeon et al.
Revised: Addressing this issue, Jeon et al.

Original: [18] proposed a reward operator Ψ Ω :∆ SA → R S×A , providing a unique IRL solution by Ψ Ω ( π E ) .
Revised: [18] proposed a reward operator Ψ Ω :∆ SA → R S×A , providing a unique IRL solution by Ψ Ω ( π E ) .

Original: Deﬁnition 1 (Regularized reward operators) .
Revised: Deﬁnition 1 (Regularized reward operators) .

Original: Deﬁne the regularized reward operator Ψ Ω as ψ π ( s, a ) :
Revised: Deﬁne the regularized reward operator Ψ Ω as ψ π ( s, a ) :

Original: = Ω (cid:48) ( s, a ; π ) − (cid:10) π s , ∇ Ω( π s ) (cid:11)
Revised: = Ω (cid:48) ( s, a ; π ) − (cid:10) π s , ∇ Ω( π s ) (cid:11)

Original: A + Ω( π s ) , for Ω (cid:48)
Revised: A + Ω( π s ) , for Ω (cid:48)

Original: ( s, · ; π ) :
Revised: ( s, · ; π ) :

Original: = ∇ Ω( π s ) = (cid:2) ∇ p Ω( p ) (cid:3) p =
Revised: = ∇ Ω( π s ) = (cid:2) ∇ p Ω( p ) (cid:3) p =

Original: π ( ·| s ) .
Revised: π ( ·| s ) .

Original: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite
Revised: The reward function ψ E := Ψ Ω ( π E ) replaces its state-action value function, since the sum of composite

Original: Bregman divergences derived from Eq.
Revised: Bregman divergences derived from Eq.

Original: (2) allows reward learning in a greedy manner [18].
Revised: (2) allows reward learning in a greedy manner [18].

Original:  RL-IRL as a Proximal Method
Revised:  RL-IRL as a Proximal Method

Original: P o li c y S p ace Π π t π t +˜ π t +
Revised: P o li c y S p ace Π π t π t +˜ π t +

Original: P Ω
Revised: P Ω

Original: R e w a r d S p ace ψ t ψ t +
Revised: R e w a r d S p ace ψ t ψ t +

Original: R S ×
Revised: R S ×

Original: A Ψ Ω ∇ Ω ∗
Revised: A Ψ Ω ∇ Ω ∗

Original: Associated reward functions.
Revised: Associated reward functions.

Original: We consider the
Revised: We consider the

Original: RL-IRL processes as a sequential algorithm with local constraints and deﬁne sequences { π t } ∞ t =1 and { ψ t } ∞ t =

Revised: RL-IRL processes as a sequential algorithm with local constraints and deﬁne sequences { π t } ∞ t =1 and { ψ t } ∞ t =

Original: that denote policies and associated reward functions, respectively.
Revised: that denote policies and associated reward functions, respectively.

Original: The associated reward functions are in a space Ψ Ω (Π) , which is an alternative space of the dual space, deﬁned by the regularized reward operator Ψ Ω .
Revised: The associated reward functions are in a space Ψ Ω (Π) , which is an alternative space of the dual space, deﬁned by the regularized reward operator Ψ Ω .

Original: Formally, we provide
Revised: Formally, we provide

Original: Lemma 1, which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .
Revised: Lemma 1, which shows a bijective relation between the operators ∇ Ω ∗ and Ψ Ω in the set Π .

Original: The proof is in Appendix A.
Revised: The proof is in Appendix A.

Original: Lemma 1 (Natural isomorphism) .
Revised: Lemma 1 (Natural isomorphism) .

Original: Let ψ ∈ Ψ Ω (Π) for Ψ Ω ( X ) :
Revised: Let ψ ∈ Ψ Ω (Π) for Ψ Ω ( X ) :

Original: = { ψ | ψ ( s, a ) =
Revised: = { ψ | ψ ( s, a ) =

Original: ψ π ( s, a ) , ∀ s ∈
Revised: ψ π ( s, a ) , ∀ s ∈

Original: S , a ∈A , π ∈ X } .
Revised: S , a ∈A , π ∈ X } .

Original: Then, ∇ Ω ∗ ( ψ ) is unique and for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .
Revised: Then, ∇ Ω ∗ ( ψ ) is unique and for every π = ∇ Ω ∗ ( ψ ) , π ∈ Π .

Original: Fig.
Revised: Fig.

Original: 1 illustrates that there is a unique ψ t for π t in every time step.
Revised: 1 illustrates that there is a unique ψ t for π t in every time step.

Original: Note that Ψ Ω ( π t ) is different from ∇ Ω( π t ) ; it is shifted by a vector 1 c with a constant c = Ω( π st ) − (cid:104)
Revised: Note that Ψ Ω ( π t ) is different from ∇ Ω( π t ) ; it is shifted by a vector 1 c with a constant c = Ω( π st ) − (cid:104)

Original: π st , ∇ Ω( π st ) (cid:105) A .
Revised: π st , ∇ Ω( π st ) (cid:105) A .

Original: Since the underlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for both Ψ Ω and ∇ Ω , as the distributivity [43] Ω ∗ ( y + 1 c ) =
Revised: Since the underlying space is a probability simplex, the operator ∇ Ω ∗ reconstructs the original point for both Ψ Ω and ∇ Ω , as the distributivity [43] Ω ∗ ( y + 1 c ) =

Original: Ω ∗ ( y ) + c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).
Revised: Ω ∗ ( y ) + c holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).

Original: An alternative interpretation is of considering a projection (gray dashed line in Fig. 1).
Revised: An alternative interpretation is of considering a projection (gray dashed line in Fig. 1).

Original: Suppose that a policy π t is updated to ˜ π t +1 ∈ R S×A .
Revised: Suppose that a policy π t is updated to ˜ π t +1 ∈ R S×A .

Original: The Bregman projection operator P Ω is applied that locates the subsequent update π t +1 to the “feasible” region, i.e., P Ω (˜ π t +1 ) :
Revised: The Bregman projection operator P Ω is applied that locates the subsequent update π t +1 to the “feasible” region, i.e., P Ω (˜ π t +1 ) :

Original: = argmin π ∈ Π [ D Ω ( π s (cid:107) ˜
Revised: = argmin π ∈ Π [ D Ω ( π s (cid:107) ˜

Original: π st +1 )] s ∈ S .
Revised: π st +1 )] s ∈ S .

Original: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .
Revised: Consequently, one can consider an updated reward function ψ t +1 as a projected target of MD associated with an alternative parameterization of Π .

Original: For instance, the parameters of ψ t can construct a softmax policy for a discrete space, or a Gaussian policy for a continuous space.
Revised: For instance, the parameters of ψ t can construct a softmax policy for a discrete space, or a Gaussian policy for a continuous space.

Original: Using the reward function ψ t +1 , an arbitrary regularized RL process maximizing Eq.
Revised: Using the reward function ψ t +1 , an arbitrary regularized RL process maximizing Eq.

Original: (2) at the t -th step [18]
Revised: (2) at the t -th step [18]

Original: [Equation]
Revised: [Equation]

Original: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the expected cumulative return.
Revised: becomes ﬁnding the next iteration π t +1 = ∇ Ω ∗ ( ψ t +1 ) by maximizing the expected cumulative return.

Original: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.
Revised: The equation shows that a regularized RL algorithm with the regularizer Ω forms a cumulative sum of Bregman divergences; thus, the policy π t +1 is uniquely achieved by the property of divergence.

Original: Online imitation learning.
Revised: Online imitation learning.

Original: Our setup starts from the apparent yet vital premise that an imitation learning algorithm does not retain the global target π E during training.
Revised: Our setup starts from the apparent yet vital premise that an imitation learning algorithm does not retain the global target π E during training.

Original: That is, it is fundamentally uncertain to model global objectives (such as J Ω ( π, ψ E ) ), which are not attainable for both RL and
Revised: That is, it is fundamentally uncertain to model global objectives (such as J Ω ( π, ψ E ) ), which are not attainable for both RL and

Original: IRL.
Revised: IRL.

Original: Instead, we hypothesize on the existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex neighborhood of π E , generated by an arbitrary estimation algorithm.
Revised: Instead, we hypothesize on the existence of a random process { ¯ π E ,t } ∞ t =1 where each estimation ¯ π E ,t resides in a closed, convex neighborhood of π E , generated by an arbitrary estimation algorithm.

Original: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) , the nonstationary objective J Ω ( π, ψ ¯ π E,t ) forms a temporal cost:
Revised: Substituting ψ E to ψ ¯ π E,t =Ψ Ω (¯ π E ,t ) , the nonstationary objective J Ω ( π, ψ ¯ π E,t ) forms a temporal cost:

Original: [Equation]
Revised: [Equation]

Original: For the sake of better understanding, we considered an actual experiment depicted in Fig.
Revised: For the sake of better understanding, we considered an actual experiment depicted in Fig.

Original: Suppose that the policies of the learning agent and the expert follow multivariate Gaussian distributions at
Revised: Suppose that the policies of the learning agent and the expert follow multivariate Gaussian distributions at

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: N ( [0 , 0] T , I) and N ( [5 , 3] T , Σ E ) with | Σ E | < 1 .
Revised: N ( [0 , 0] T , I) and N ( [5 , 3] T , Σ E ) with | Σ E | < 1 .

Original: Let a (suboptimal) reference policy ¯ π E ,t be independently ﬁtted with a maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .
Revised: Let a (suboptimal) reference policy ¯ π E ,t be independently ﬁtted with a maximum likelihood estimator with a relatively high learning rate, starting from ¯ π E , 1 = π 1 .

Original: The policy π t was trained by a cost function D Ω ( ·(cid:107) ¯ π E ,t ) using the MD update rule in Eq.
Revised: The policy π t was trained by a cost function D Ω ( ·(cid:107) ¯ π E ,t ) using the MD update rule in Eq.

Original: In Fig.
Revised: In Fig.

Original: 2, we ﬁrst observed that choosing a high step size constant η accelerated the training speed mainly in the early phase.
Revised: 2, we ﬁrst observed that choosing a high step size constant η accelerated the training speed mainly in the early phase.

Original: The results also showed that the performance of MD ( D Ω ( π t (cid:107) π E ) ) outperformed that of referenced maximum likelihood estimation ( D Ω (¯ π E ,t (cid:107) π E ) ) by choosing an effectively low step size.
Revised: The results also showed that the performance of MD ( D Ω ( π t (cid:107) π E ) ) outperformed that of referenced maximum likelihood estimation ( D Ω (¯ π E ,t (cid:107) π E ) ) by choosing an effectively low step size.

Original: This empirical evidence suggests that there are clear advantages in formalization of the training steps and scheduling the step sizes, especially for unreliable statistical model ¯ π E ,t .
Revised: This empirical evidence suggests that there are clear advantages in formalization of the training steps and scheduling the step sizes, especially for unreliable statistical model ¯ π E ,t .

Original: MD update rules.
Revised: MD update rules.

Original: As a result of these ﬁndings, we formulate subsequent MD steps with a regularized reward function.
Revised: As a result of these ﬁndings, we formulate subsequent MD steps with a regularized reward function.

Original: Let w t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions F at the t -th step.
Revised: Let w t be a parameter on a set W and F t : W → R be a convex cost function from a class of functions F at the t -th step.

Original: Replacing the L2 proximity term of proximal gradient descent with a Bregman divergence, the proximal form of the MD update for Eq.
Revised: Replacing the L2 proximity term of proximal gradient descent with a Bregman divergence, the proximal form of the MD update for Eq.

Original: (1) is written as [45] minimize w ∈W
Revised: (1) is written as [45] minimize w ∈W

Original: [Equation]
Revised: [Equation]

Original: (6) where α t := 1 / η t denotes an inverse of the current step size η t [46].
Revised: (6) where α t := 1 / η t denotes an inverse of the current step size η t [46].

Original: Plugging each divergence of the cumulative cost F t to Eq.
Revised: Plugging each divergence of the cumulative cost F t to Eq.

Original: (6), the MD-IRL update for the subsequent reward function ψ t +1 =
Revised: (6), the MD-IRL update for the subsequent reward function ψ t +1 =

Original: Ψ Ω ( π t +1 ) is derived by solving a problem minimize
Revised: Ψ Ω ( π t +1 ) is derived by solving a problem minimize

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: where the gradient of D Ω is taken with respect to its ﬁrst argument π st .
Revised: where the gradient of D Ω is taken with respect to its ﬁrst argument π st .

Original: Note that solving the optimization Eq.
Revised: Note that solving the optimization Eq.

Original: (7) requires interaction between π t and the dynamics of the given environment in order to minimize F t ; thus, the corresponding RL process plays an essential role in sequential learning by the induced the value measures.
Revised: (7) requires interaction between π t and the dynamics of the given environment in order to minimize F t ; thus, the corresponding RL process plays an essential role in sequential learning by the induced the value measures.

Original: At a glance, the objective is analogous to ﬁnding an interpolation at each iteration where the point is controlled η t .
Revised: At a glance, the objective is analogous to ﬁnding an interpolation at each iteration where the point is controlled η t .

Original: Fig.
Revised: Fig.

Original: 3 shows that the uncertainty of π t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region).
Revised: 3 shows that the uncertainty of π t (blue region) gets minimal regardless of persisting uncertainty of ¯ π E ,t (red region).

Original: a
Revised: a

Original: G r ound - t r u t h c o s t
Revised: G r ound - t r u t h c o s t

Original: MDwith η t b
Revised: MDwith η t b

Original: G r ound - t r u t h c o s t
Revised: G r ound - t r u t h c o s t

Original: MDwith η t +
Revised: MDwith η t +

Original:  Convergence Analyses
Revised:  Convergence Analyses

Original: In this section, we present our theoretical results.
Revised: In this section, we present our theoretical results.

Original: The main goals of the following arguments are to address 1 (cid:13) the convergence of MD updates for various cases and 2 (cid:13) the necessity of scheduling the amount of learning.
Revised: The main goals of the following arguments are to address 1 (cid:13) the convergence of MD updates for various cases and 2 (cid:13) the necessity of scheduling the amount of learning.

Original: Suppose that state instances of s ( t ) i ∈
Revised: Suppose that state instances of s ( t ) i ∈

Original: τ t cover the entire S by executing the policy π t in an inﬁnite horizon.
Revised: τ t cover the entire S by executing the policy π t in an inﬁnite horizon.

Original: From this assumption, we deﬁne a temporal cost function at the time step t :
Revised: From this assumption, we deﬁne a temporal cost function at the time step t :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: that involves π t , and additionally a trajectory τ t as inputs.
Revised: that involves π t , and additionally a trajectory τ t as inputs.

Original: We refer to the global objective as ﬁnding a unique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :
Revised: We refer to the global objective as ﬁnding a unique ﬁxed point π ∗ ∈ Π that minimizes a total cost F ( π ) :

Original: =
Revised: =

Original: E [ f ( π, τ t )] , where the expectation is taken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π, τ t )] .
Revised: E [ f ( π, τ t )] , where the expectation is taken over trajectories of entire steps, i.e., lim t →∞ E τ 1: t [ f ( π, τ t )] .

Original: Taking the (stepwise) gradient for each π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω( π ∗ ( ·| s ))
Revised: Taking the (stepwise) gradient for each π ( ·| s ) , an optimal policy π ∗ is found by E [ ∇ Ω( π ∗ ( ·| s ))

Original: − ∇ Ω(¯ π E ,t ( ·| s ))]
Revised: − ∇ Ω(¯ π E ,t ( ·| s ))]

Original: = 0 when t → ∞ ; hence, ∇ Ω( π ∗ ( ·| s ))
Revised: = 0 when t → ∞ ; hence, ∇ Ω( π ∗ ( ·| s ))

Original: =
Revised: =

Original: lim t →∞ E [ ∇ Ω(¯ π E ,t ( ·| s ))] .
Revised: lim t →∞ E [ ∇ Ω(¯ π E ,t ( ·| s ))] .

Original: Introducing the optimal policy π ∗ allows not only the speciﬁc situation when 1 (cid:13) π E
Revised: Introducing the optimal policy π ∗ allows not only the speciﬁc situation when 1 (cid:13) π E

Original: =
Revised: =

Original: π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually convergent with t → ∞ , but also more general situations where 2 (cid:13) π E / ∈ Π or the estimated expert policy ¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.
Revised: π ∗ ∈ Π and the estimation algorithm of ¯ π E ,t is actually convergent with t → ∞ , but also more general situations where 2 (cid:13) π E / ∈ Π or the estimated expert policy ¯ π E ,t does not converge; the algorithm ﬁnds convergence to a ﬁxed point by scheduling updates.

Original: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.
Revised: We state two conditions of { η t } ∞ t =1 to guarantee convergence justiﬁed in Theorems 1 and 2.

Original: • Convergent sequence & divergent series:
Revised: • Convergent sequence & divergent series:

Original: [Equation]
Revised: [Equation]

Original: • Divergent series & convergent series of squared terms:
Revised: • Divergent series & convergent series of squared terms:

Original: [Equation]
Revised: [Equation]

Original: Let us assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.
Revised: Let us assume Lipschitz continuity of ∇ Ω and boundedness of D Ω in a Banach space.

Original: In some Ω , these two assumptions do not necessarily hold for extreme cases in ∆ S A , e.g., a distribution that π ( a | s ) = 0 for some entries.
Revised: In some Ω , these two assumptions do not necessarily hold for extreme cases in ∆ S A , e.g., a distribution that π ( a | s ) = 0 for some entries.

Original: Nevertheless, these outliers can be left out if the parametrization is constrained to satisfy the assumptions.
Revised: Nevertheless, these outliers can be left out if the parametrization is constrained to satisfy the assumptions.

Original: For example, one can either 1 (cid:13) prevent a policy from having non-zero entries of probabilities for a discrete policy or 2 (cid:13) prevent a policy from having too low entropy for a continuous policy, by enforcing certain constraints on its parametric representations.
Revised: For example, one can either 1 (cid:13) prevent a policy from having non-zero entries of probabilities for a discrete policy or 2 (cid:13) prevent a policy from having too low entropy for a continuous policy, by enforcing certain constraints on its parametric representations.

Original: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.
Revised: Theorem 1 argues that the sequence { η t } ∞ t =1 shall diverge for its series; therefore, Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: Theorem 1 (Stepsize considerations) .
Revised: Theorem 1 (Stepsize considerations) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergence D Ω is bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergence D Ω is bounded.

Original: Assume a general condition of the problem that inf π ∈ Π E (cid:2) f
Revised: Assume a general condition of the problem that inf π ∈ Π E (cid:2) f

Original: ( π, τ t ) (cid:3) > 0 .
Revised: ( π, τ t ) (cid:3) > 0 .

Original: Then we get lim T →∞ E τ 1: T (cid:2)(cid:80) ∞
Revised: Then we get lim T →∞ E τ 1: T (cid:2)(cid:80) ∞

Original: i =

Revised: i =

Original: D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)
Revised: D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)

Original: π T ( ·| s i )) (cid:3) = 

Revised: π T ( ·| s i )) (cid:3) =

Original: if and only if
Revised: if and only if

Original: Eq.
Revised: Eq.

Original: (9) is satisﬁed.
Revised: (9) is satisﬁed.

Original: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T (cid:2) f T ( π T , τ T
Revised: (a) If lim t →∞ η t = 0 , then T ∈ N , n<T , and c> 0 exist such that E τ 1: T (cid:2) f T ( π T , τ T

Original: [Equation]
Revised: [Equation]

Original: (b) If the step size is in the form of η t = 4 t +1 , then E τ 1: T (cid:2) (cid:80) ∞
Revised: (b) If the step size is in the form of η t = 4 t +1 , then E τ 1: T (cid:2) (cid:80) ∞

Original: i =

Revised: i =

Original: D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)
Revised: D Ω (cid:0) π ∗ ( ·| s i ) (cid:13)(cid:13)

Original: π T ( ·| s i )) (cid:3) =
Revised: π T ( ·| s i )) (cid:3) =

Original: O (1 /T ) .
Revised: O (1 /T ) .

Original: Next, we present Theorem 2, which addresses the convergence in a speciﬁc case when π E resides in Π .
Revised: Next, we present Theorem 2, which addresses the convergence in a speciﬁc case when π E resides in Π .

Original: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .
Revised: Additionally, the theorem addresses the bounds of the performance for ﬁxed size update η t ≡ η 1 .

Original: Theorem 2 (Optimal cases) .
Revised: Theorem 2 (Optimal cases) .

Original: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.
Revised: Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences be bounded.

Original: Assume π 1 (cid:54) =
Revised: Assume π 1 (cid:54) =

Original: π E and inf π ∈ Π E
Revised: π E and inf π ∈ Π E

Original: [ f ( π, τ t )]
Revised: [ f ( π, τ t )]

Original: = 0 .
Revised: = 0 .

Original: Then, E (cid:2) f ( π t , τ t ) (cid:3) = 

Revised: Then, E (cid:2) f ( π t , τ t ) (cid:3) =

Original: if and only if (cid:80) ∞ t =1 η t = ∞ .
Revised: if and only if (cid:80) ∞ t =1 η t = ∞ .

Original: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that
Revised: If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of the algorithm by imposing the stronger condition of step size in Eq.
Revised: Lastly, Proposition 1 provides the sufﬁcient condition for the almost certain convergence of the algorithm by imposing the stronger condition of step size in Eq.

Original: The proofs are in Appendix A.
Revised: The proofs are in Appendix A.

Original: Proposition 1 (General cases) .
Revised: Proposition 1 (General cases) .

Original: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π, τ t )] > 0 .
Revised: Assume that π E / ∈ Π , hence inf π ∈ Π E [ f ( π, τ t )] > 0 .

Original: If the step sizes satisﬁes Eq.
Revised: If the step sizes satisﬁes Eq.

Original: (10), then lim t →∞ (cid:80) ∞ i =

Revised: (10), then lim t →∞ (cid:80) ∞ i =

Original: γ i D Ω (cid:0) π ∗ ( · | s i ) (cid:13)(cid:13)
Revised: γ i D Ω (cid:0) π ∗ ( · | s i ) (cid:13)(cid:13)

Original: π t ( · | s i ) (cid:1) converges to 0 almost surely.
Revised: π t ( · | s i ) (cid:1) converges to 0 almost surely.

Original: Regrets.
Revised: Regrets.

Original: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as
Revised: For a sequence of state trajectories { τ t } t ∈ N , let us deﬁne a regret at the t -th iteration as

Original: [Equation]
Revised: [Equation]

Original: In the optimal case of inf π ∈ Π E [ f ( π, τ t )] = 0 , the cost f inherits the property of Bregman divergence so that the inﬁmum is achieved by 0 at π E .
Revised: In the optimal case of inf π ∈ Π E [ f ( π, τ t )] = 0 , the cost f inherits the property of Bregman divergence so that the inﬁmum is achieved by 0 at π E .

Original: In this case, the regret is bounded to O (1 /T ) by the theorems.
Revised: In this case, the regret is bounded to O (1 /T ) by the theorems.

Original: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π, τ t )] > 0 when the step sizes abide by Eq.
Revised: By Proposition 1, the MD updates converge for the case of inf π ∈ Π E [ f ( π, τ t )] > 0 when the step sizes abide by Eq.

Original: Thus, the regret is bounded to O (1 /T ) even for the general case.
Revised: Thus, the regret is bounded to O (1 /T ) even for the general case.

Original:  Algorithm: MD-IRL on an Adversarial Framework
Revised:  Algorithm: MD-IRL on an Adversarial Framework

Original: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized reward function with adversarial learning and the MD update rule.
Revised: In this section, we propose MD-AIRL, a novel AIL algorithm which trains a parameterized reward function with adversarial learning and the MD update rule.

Original: Neural network parameters θ , φ , and ν are newly presented representing agent policy, reward, and expert policy functions respectively.
Revised: Neural network parameters θ , φ , and ν are newly presented representing agent policy, reward, and expert policy functions respectively.

Original: Dual discriminators.
Revised: Dual discriminators.

Original: In order to bridge the gap between theory and practice, we propose a novel discriminative architecture, motivated by GAN studies regarding multiple discriminators [47, 48].
Revised: In order to bridge the gap between theory and practice, we propose a novel discriminative architecture, motivated by GAN studies regarding multiple discriminators [47, 48].

Original: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densities and imitating speciﬁc behavior.
Revised: Basically, the proposed discriminators separate two concepts in AIL: matching overall state densities and imitating speciﬁc behavior.

Original: Given a learning agent policy π θ , an estimation policy π ν , and a discriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as
Revised: Given a learning agent policy π θ , an estimation policy π ν , and a discriminative neural network for states d ξ : S → R , the two discriminators are deﬁned as

Original: D ν
Revised: D ν

Original: ( s, a ; θ , ξ ) =
Revised: ( s, a ; θ , ξ ) =

Original: σ (cid:0) log (cid:8) π ν ( a | s ) (cid:14) π θ ( a | s ) (cid:9) + d ξ ( s ) (cid:1) and D ξ ( s ) =
Revised: σ (cid:0) log (cid:8) π ν ( a | s ) (cid:14) π θ ( a | s ) (cid:9) + d ξ ( s ) (cid:1) and D ξ ( s ) =

Original: σ (cid:0) d ξ ( s ) (cid:1) , ∀ s ∈ S , a ∈ A , where σ ( · ) denotes the sigmoid function.
Revised: σ (cid:0) d ξ ( s ) (cid:1) , ∀ s ∈ S , a ∈ A , where σ ( · ) denotes the sigmoid function.

Original: The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:
Revised: The discriminators are trained using binary logistic regression losses with respect to mini-batch adversarial samples:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: α t η t 1 − η t t αη where d ξ and π θ are not trained for learning D ν .
Revised: α t η t 1 − η t t αη where d ξ and π θ are not trained for learning D ν .

Original: Let ρ π ∈ ∆ S denote the state visitation density of π , which is deﬁned as ρ π ( s ) :
Revised: Let ρ π ∈ ∆ S denote the state visitation density of π , which is deﬁned as ρ π ( s ) :

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E π (cid:2)(cid:80) ∞
Revised: E π (cid:2)(cid:80) ∞

Original: i =

Revised: i =

Original: γ i
Revised: γ i

Original: I { s i = s } (cid:3) , where I {·} is an indicator function.
Revised: I { s i = s } (cid:3) , where I {·} is an indicator function.

Original: The convergence of functions in an ideal case is found at π ν = π E and d ξ ( s )=log { ρ π E ( s ) /ρ
Revised: The convergence of functions in an ideal case is found at π ν = π E and d ξ ( s )=log { ρ π E ( s ) /ρ

Original: π ( s ) } .
Revised: π ( s ) } .

Original: Learning with MD-based rewards.
Revised: Learning with MD-based rewards.

Original: Based on the MD solution for a regularized reward function, we focus on developing an MD-based learning objective.
Revised: Based on the MD solution for a regularized reward function, we focus on developing an MD-based learning objective.

Original: Let ψ φ ∈ Ψ(Π) denote a parameterized regularized reward function, and π φ denotes a corresponding policy from φ .
Revised: Let ψ φ ∈ Ψ(Π) denote a parameterized regularized reward function, and π φ denotes a corresponding policy from φ .

Original: Note that the transformation between ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁc parameterizations [18].
Revised: Note that the transformation between ψ φ and π φ can be performed with shared φ without the additional computational costs under speciﬁc parameterizations [18].

Original: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν , we deﬁne the objective of φ as a direct interpretation of the update rule of Eq.
Revised: Using a step size η t , the RL agent π θ , and the estimated expert policy π ν , we deﬁne the objective of φ as a direct interpretation of the update rule of Eq.

Original: [Equation]
Revised: [Equation]

Original: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.
Revised: where the trajectory ¯ τ t denotes sample states using both agent and expert trajectories.

Original: As shown in
Revised: As shown in

Original: Fig. 4, η t is adjusted by linearly increasing α t , which originated from the analyses in Section 5.
Revised: Fig. 4, η t is adjusted by linearly increasing α t , which originated from the analyses in Section 5.

Original: Another important consideration is the way of handling covariate shifts [49] since it is likely that state densities between the expert and the agent are misaligned.
Revised: Another important consideration is the way of handling covariate shifts [49] since it is likely that state densities between the expert and the agent are misaligned.

Original: Thus, we deﬁne the IRL reward function as linear combinations of ψ φ and the state density discriminative signal:
Revised: Thus, we deﬁne the IRL reward function as linear combinations of ψ φ and the state density discriminative signal:

Original: [Equation]
Revised: [Equation]

Original: with a coefﬁcient λ ∈ R + .
Revised: with a coefﬁcient λ ∈ R + .

Original: Utilizing an arbitrary regularized RL algorithm with a regularizer λ Ω( · )
Revised: Utilizing an arbitrary regularized RL algorithm with a regularizer λ Ω( · )

Original: [Equation]
Revised: [Equation]

Original: the reward learning regarding agent policy π θ is decomposed into the following:
Revised: the reward learning regarding agent policy π θ is decomposed into the following:

Original: [Equation]
Revised: [Equation]

Original: Minimizing the ﬁrst term of E π θ (cid:2) D Ω ( π sθ (cid:107) π sφ ) (cid:3) represents learning with the MD formulation.
Revised: Minimizing the ﬁrst term of E π θ (cid:2) D Ω ( π sθ (cid:107) π sφ ) (cid:3) represents learning with the MD formulation.

Original: Meanimizing the second term D KL ( ρ π θ (cid:107) ρ π E ) plays an auxiliary role in facilitating the supports of state visitation densities to be correctly matched.
Revised: Meanimizing the second term D KL ( ρ π θ (cid:107) ρ π E ) plays an auxiliary role in facilitating the supports of state visitation densities to be correctly matched.

Original: With the hyperparameter λ , we report that learning the second term is helpful when the state densities are heavily misaligned in certain benchmarks.
Revised: With the hyperparameter λ , we report that learning the second term is helpful when the state densities are heavily misaligned in certain benchmarks.

Original: Algorithm 1 summarizes the entire procedure.
Revised: Algorithm 1 summarizes the entire procedure.

Original: We defer additional details to Appendices B and C.
Revised: We defer additional details to Appendices B and C.

Original:  Experimental Results
Revised:  Experimental Results

Original: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ω while retaining the scalability of AIL.
Revised: The aim of our experiments was to identify whether MD-AIRL facilitates robustness for various Ω while retaining the scalability of AIL.

Original: The comparative method was RAIRL with density-based models (RAIRL-DBM) which contained comparable expressiveness as MD-AIRL.
Revised: The comparative method was RAIRL with density-based models (RAIRL-DBM) which contained comparable expressiveness as MD-AIRL.

Original: For RL, we used
Revised: For RL, we used

Original: RAC [44], which is a generalization of the SAC algorithm
Revised: RAC [44], which is a generalization of the SAC algorithm

Original: We considered a class of regularizers Ω( p ) =
Revised: We considered a class of regularizers Ω( p ) =

Original: − E x ∼ p
Revised: − E x ∼ p

Original: [ ϕ ( p ( x ))] with 1 (cid:13) Shannon ( ϕ ( x )
Revised: [ ϕ ( p ( x ))] with 1 (cid:13) Shannon ( ϕ ( x )

Original: = log( x ) ), 2 (cid:13)
Revised: = log( x ) ), 2 (cid:13)

Original: Tsallis ( ϕ ( x ; q ) =
Revised: Tsallis ( ϕ ( x ; q ) =

Original: 1 q − 1 ( x q − 1 − 1) , q = 2 by default), 3 (cid:13) exp ( ϕ ( x )=
Revised: 1 q − 1 ( x q − 1 − 1) , q = 2 by default), 3 (cid:13) exp ( ϕ ( x )=

Original: e − e x ), 4 (cid:13) cos ( ϕ ( x )=
Revised: e − e x ), 4 (cid:13) cos ( ϕ ( x )=

Original: cos( π 2 x ) ), and 5 (cid:13) sin ( ϕ ( x )=1 − sin( π 2 x ) ).
Revised: cos( π 2 x ) ), and 5 (cid:13) sin ( ϕ ( x )=1 − sin( π 2 x ) ).

Original: 7.1 Large scale multiarmed bandits
Revised: 7.1 Large scale multiarmed bandits

Original: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the cardinality of action spaces varies largely.
Revised: To measure the performance of IRL, we ﬁrst considered multiarmed bandit problems, where the cardinality of action spaces varies largely.

Original: Learning the optimal distribution of π E becomes challenging as the cardinality of the space |A| increases, because the frequency of each sample becomes sparse due to the curse of dimensionality.
Revised: Learning the optimal distribution of π E becomes challenging as the cardinality of the space |A| increases, because the frequency of each sample becomes sparse due to the curse of dimensionality.

Original: The stateless expert distribution π E was generated by the parameters of softmax distribution π E ( a ) =
Revised: The stateless expert distribution π E was generated by the parameters of softmax distribution π E ( a ) =

Original: exp( z a ) / (cid:80) i exp( z i ) , where the logits z i were randomly initialized.
Revised: exp( z a ) / (cid:80) i exp( z i ) , where the logits z i were randomly initialized.

Original: We set the action size to |A| =
Revised: We set the action size to |A| =

Original: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.
Revised: 10 2 , 10 3 , 10 4 and restricted each sample size of 16.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when three different cardinalities and ﬁve regularizers were considered.
Revised: Tab. 2 shows that MD-AIRL achieved overall lower Bregman divergence on average when three different cardinalities and ﬁve regularizers were considered.

Original: Fig.
Revised: Fig.

Original: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1 to be greater than 1 ( α 1 = 0 . 5 ).
Revised: 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1 to be greater than 1 ( α 1 = 0 . 5 ).

Original: MD-AIRL exceeded the discriminative performance of RAIRL after certain steps, while the progression of RAIRL mostly stopped at local minima.
Revised: MD-AIRL exceeded the discriminative performance of RAIRL after certain steps, while the progression of RAIRL mostly stopped at local minima.

Original: MD-AIRL outperformed RAIRL in four cases by choosing an effectively low step size at η T to be less than 1 ( α T = 2 ).
Revised: MD-AIRL outperformed RAIRL in four cases by choosing an effectively low step size at η T to be less than 1 ( α T = 2 ).

Original: These results match the properties of MD algorithms and our convergence analyses.
Revised: These results match the properties of MD algorithms and our convergence analyses.

Original: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robust reward acquisition and imitation for situations when the total number of data samples is limited.
Revised: Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robust reward acquisition and imitation for situations when the total number of data samples is limited.

Original: [Figure]
Revised: [Figure]

Original: 7.2 A continuous multigoal environment
Revised: 7.2 A continuous multigoal environment

Original: We then considered a multigoal environment.
Revised: We then considered a multigoal environment.

Original: In this environment, an agent is a two-dimensional point mass initialized at the origin, and the four goals are located in the four cardinal directions.
Revised: In this environment, an agent is a two-dimensional point mass initialized at the origin, and the four goals are located in the four cardinal directions.

Original: The objective of imitation learning is to go toward each direction evenly as possible where the expert model was trained by the SAC algorithm.
Revised: The objective of imitation learning is to go toward each direction evenly as possible where the expert model was trained by the SAC algorithm.

Original: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with full covariance matrices instead of conventional diagonal Gaussian policies (see Appendices B and C).
Revised: To draw informative reward surfaces regarding stochastic actions, we considered the multivariate Gaussian distribution policies parameterized with full covariance matrices instead of conventional diagonal Gaussian policies (see Appendices B and C).

Original: Fig.
Revised: Fig.

Original: 6 (a) shows trajectories generated by the trained agent.
Revised: 6 (a) shows trajectories generated by the trained agent.

Original: Fig.
Revised: Fig.

Original: 6 (b) shows that MD-AIRL achieved higher entropy for reaching the multiple goals.
Revised: 6 (b) shows that MD-AIRL achieved higher entropy for reaching the multiple goals.

Original: Fig.
Revised: Fig.

Original: 6 (c) shows reward surfaces with regularizers, which were calculated by ψ φ ( s, a )
Revised: 6 (c) shows reward surfaces with regularizers, which were calculated by ψ φ ( s, a )

Original: + ϕ
Revised: + ϕ

Original: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .
Revised: ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .

Original: During the training, the MD reward was similar to the estimated ground truth using adversarial training.
Revised: During the training, the MD reward was similar to the estimated ground truth using adversarial training.

Original: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t was sufﬁciently close to the expert behavior.
Revised: However, the surface of MD-AIRL became ﬂatter than the ground-truth estimation when π t was sufﬁciently close to the expert behavior.

Original: As a result, we claim that a drastic change in the target distribution, which is one of the typical characteristics of adversarial frameworks, is prevented.
Revised: As a result, we claim that a drastic change in the target distribution, which is one of the typical characteristics of adversarial frameworks, is prevented.

Original: We argue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.
Revised: We argue that these characteristics mitigate overﬁtting caused by unreliable discriminative signals.

Original: 7.3 A continuous control benchmark: MuJoCo
Revised: 7.3 A continuous control benchmark: MuJoCo

Original: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.
Revised: Lastly, we validated MD-AIRL on the MuJoCo continuous control benchmark suite.

Original: We assumed full covariance Gaussian policies for both learner’s policy π and expert policy π E .
Revised: We assumed full covariance Gaussian policies for both learner’s policy π and expert policy π E .

Original: We used the hyperbolized environment assumption [18] where the action constraint is incorporated into the dynamics as a part of the environment using hyperbolic tangent activation.
Revised: We used the hyperbolized environment assumption [18] where the action constraint is incorporated into the dynamics as a part of the environment using hyperbolic tangent activation.

Original: Sample efﬁciency.
Revised: Sample efﬁciency.

Original: For each task, we considered two different numbers of episodes collected by an expert policy.
Revised: For each task, we considered two different numbers of episodes collected by an expert policy.

Original: In Fig.
Revised: In Fig.

Original: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [20] is shown with the expert and random agent performance.
Revised: 7, the performance of MD-AIRL, RAIRL, and behavior cloning (bc) algorithms [20] is shown with the expert and random agent performance.

Original: MD-AIRL was able
Revised: MD-AIRL was able

Original: [Figure]
Revised: [Figure]

Original: The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.
Revised: The agent and IRL reward functions were trained with 4 episodes of expert demonstrations.

Original: [Figure]
Revised: [Figure]

Original: to achieve consistent performance throughout the tasks and demonstration showed high tolerance to the scarcity of to RAIRL demonstrations.
Revised: to achieve consistent performance throughout the tasks and demonstration showed high tolerance to the scarcity of to RAIRL demonstrations.

Original: The plots in Fig.
Revised: The plots in Fig.

Original: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among we have tested.
Revised: 8 indicate that MD-AIRL showed higher average scores with lower variance compared to RAIRL, across 30 distinct cases among we have tested.

Original: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.
Revised: MD-AIRL inherits the scalability of AIL, and it is highly stable with respect to limited sample sizes.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Noisy demonstrations.
Revised: Noisy demonstrations.

Original: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with two levels of Gaussian additive noises, resulting in suboptimal demonstrations.
Revised: Tab. 3 shows the results of imitation learning experiments for 100 expert demonstrations with two levels of Gaussian additive noises, resulting in suboptimal demonstrations.

Original: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.
Revised: MD-AIRL is highly tolerant to noisy data, consistently achieving higher performance.

Original: The experiment is closely related to the general case in the theory; the results suggest that the characteristics of MDAIRL are in alignment with our analyses of the MD reward learning scheme
Revised: The experiment is closely related to the general case in the theory; the results suggest that the characteristics of MDAIRL are in alignment with our analyses of the MD reward learning scheme

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We present a detailed analysis of the noisy demonstration experiments (Fig. 9).
Revised: We present a detailed analysis of the noisy demonstration experiments (Fig. 9).

Original: Let the Bregman divergence between agent and ground-truth expert policies be the error, and we measured these errors by increasing the given noise level for the expert trajectories.
Revised: Let the Bregman divergence between agent and ground-truth expert policies be the error, and we measured these errors by increasing the given noise level for the expert trajectories.

Original: Fig.
Revised: Fig.

Original: 9 shows a general tendency that MD-AIRL has lower errors than
Revised: 9 shows a general tendency that MD-AIRL has lower errors than

Original: RAIRL.
Revised: RAIRL.

Original: With Tab. 3 and Fig.
Revised: With Tab. 3 and Fig.

Original: 9, we were able to ﬁnd the evident correlation between average Bregman divergence and performance since imitation learning convergence when the divergence is zero.
Revised: 9, we were able to ﬁnd the evident correlation between average Bregman divergence and performance since imitation learning convergence when the divergence is zero.

Original: Thus, this is another piece of empirical evidence that veriﬁes our theoretical claims.
Revised: Thus, this is another piece of empirical evidence that veriﬁes our theoretical claims.

Original:  Conclusions and Discussion
Revised:  Conclusions and Discussion

Original: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitation learning problem in the real world.
Revised: In this paper, we presented MD-AIRL, a practical AIL algorithm designed to solve the imitation learning problem in the real world.

Original: We proved that the proposed method has clear advantages over previous AIL methods in terms of robustness.
Revised: We proved that the proposed method has clear advantages over previous AIL methods in terms of robustness.

Original: We veriﬁed MD-AIRL in a variety of situations, including high-dimensional spaces, limited samples, and imperfect demonstrations.
Revised: We veriﬁed MD-AIRL in a variety of situations, including high-dimensional spaces, limited samples, and imperfect demonstrations.

Original: The empirical evidence showed that MD-AIRL outperforms previous methods on various benchmarks.
Revised: The empirical evidence showed that MD-AIRL outperforms previous methods on various benchmarks.

Original: We conclude that the rich foundation of optimization theories shows a promising direction for AIL studies.
Revised: We conclude that the rich foundation of optimization theories shows a promising direction for AIL studies.

Original: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.
Revised: Considering RL and IRL with geometric perspectives is vital for solving real-world problems.

Original: Although our work covers various imitation learning problems with the Bregman divergence, this does not include some other problems when the proximity term is of other statistical divergence families, such as the f-divergence [52].
Revised: Although our work covers various imitation learning problems with the Bregman divergence, this does not include some other problems when the proximity term is of other statistical divergence families, such as the f-divergence [52].

Original: If the relationship between these classes of divergences is studied in more detail, it is expected to proceed with applications to various subﬁelds of machine learning.
Revised: If the relationship between these classes of divergences is studied in more detail, it is expected to proceed with applications to various subﬁelds of machine learning.

Original: The assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but some outliers might have substantial meaning for certain tasks.
Revised: The assumptions on Ω in our analyses are usually justiﬁed by enforcing a speciﬁc policy space, but some outliers might have substantial meaning for certain tasks.

Original: Therefore, extensive analyses on these parameterizations remain as future works.
Revised: Therefore, extensive analyses on these parameterizations remain as future works.

Original: The “impurity” of the MD-AIRL reward function compared to Ψ Ω (Π) can be regarded as a limitation.
Revised: The “impurity” of the MD-AIRL reward function compared to Ψ Ω (Π) can be regarded as a limitation.

Original: To fully resolve this problem, all data must be treated as on-policy samples, which might require a sophisticated sampling mechanism.
Revised: To fully resolve this problem, all data must be treated as on-policy samples, which might require a sophisticated sampling mechanism.

Original: Societal impacts.
Revised: Societal impacts.

Original: The evolution of imitation learning algorithms is expected to bring a structural shift in the labor market.
Revised: The evolution of imitation learning algorithms is expected to bring a structural shift in the labor market.

Original: The negative impact could be mitigated by diversiﬁcation, uniﬁcation, and redeﬁnition of routine and manual jobs.
Revised: The negative impact could be mitigated by diversiﬁcation, uniﬁcation, and redeﬁnition of routine and manual jobs.

Original: The results of our work can be abused as a tool for analyzing individual data.
Revised: The results of our work can be abused as a tool for analyzing individual data.

Original: Therefore, we stress that certain acts should be carefully regulated, such as collecting a substantial amount of individuals’ data and aggressively tracking personal identity.
Revised: Therefore, we stress that certain acts should be carefully regulated, such as collecting a substantial amount of individuals’ data and aggressively tracking personal identity.

Original: Acknowledgments
Revised: Acknowledgments

Original: The authors would like to thank the anonymous reviewers, Woosuk Choi, Jaein Kim, and Min
Revised: The authors would like to thank the anonymous reviewers, Woosuk Choi, Jaein Kim, and Min

Original: Whoo Lee for their helpful discussion and comments.
Revised: Whoo Lee for their helpful discussion and comments.

Original: This work was partly supported by the
Revised: This work was partly supported by the

Original: IITP (2022-0-00951-LBA/25%, 2022-0-00953-PICA/25%, 2015-0-00310-SW.StarLab/10%, 2021-002068-AIHub/10%, 2021-0-01343-GSAI/10%, 2019-0-01371-BabyMind/10%) grant funded by the
Revised: IITP (2022-0-00951-LBA/25%, 2022-0-00953-PICA/25%, 2015-0-00310-SW.StarLab/10%, 2021-002068-AIHub/10%, 2021-0-01343-GSAI/10%, 2019-0-01371-BabyMind/10%) grant funded by the

Original: Korean government, and the CARAI (UD190031RD/10%) grant funded by the DAPA and ADD.
Revised: Korean government, and the CARAI (UD190031RD/10%) grant funded by the DAPA and ADD.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A. 3.
Revised: [Yes] See Appendix A. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Our empirical studies can be reproduced by from the detailed information in Appendices B and C. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Our empirical studies can be reproduced by from the detailed information in Appendices B and C. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 7.
Revised: [Yes] See Section 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: In experiments, each algorithm was executed in CPU (a single thread).
Revised: In experiments, each algorithm was executed in CPU (a single thread).

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: The MuJoCo simulator used in our experiments is freely available to everyone.
Revised: The MuJoCo simulator used in our experiments is freely available to everyone.

Original: See the site (https://mujoco.org).
Revised: See the site (https://mujoco.org).

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]