Original: R EGULARIZED I NVERSE R EINFORCEMENT L EARNING
Revised: R EGULARIZED I NVERSE R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions.
Revised: Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions.

Original: Regularized IRL applies  convex regularizers to the learner’s policy in order to avoiddegenerate solutions, i.e., whereby any constant reward rationalizes the expert behavior.
Revised: Regularized IRL applies strongly convex regularizers to the learner’s policy in order to avoid the expert’s behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions.

Original: We propose analytical solutions, and practical methods to obtain them, for regularized IRL.
Revised: We propose tractable solutions, and practical methods to obtain them, for regularized IRL.

Original: Current methods are restricted to the maximumentropy IRL framework, limiting them to Shannon-entropy regularizers, as well as proposing functional-form solutions that are generally intractable.
Revised: Current methods are restricted to the maximum-entropy IRL framework, limiting them to Shannon-entropy regularizers, as well as proposing the solutions that are intractable in practice.

Original: We present theoretical backing for our proposed IRL method’s applicability for both discrete and continuous controls, empirically validating our performance on a variety of tasks.
Revised: We present theoretical backing for our proposed IRL method’s applicability for both discrete and continuous controls, empirically validating our performance on a variety of tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has been successfully applied to many challenging domains including games (Mnih et al., 2015; 2016) and robot control (Schulman et al., 2015; Fujimoto et al., 2018;
Revised: Reinforcement learning (RL) has been successfully applied to many challenging domains including games (Mnih et al., 2015; 2016) and robot control (Schulman et al., 2015; Fujimoto et al., 2018;

Original: Haarnoja et al., 2018).
Revised: Haarnoja et al., 2018).

Original: Advanced RL methods often employ policy regularization motivated by, e.g., boosting exploration (Haarnoja et al., 2018) or safe policy improvement (Schulman et al., 2015).
Revised: Advanced RL methods often employ policy regularization motivated by, e.g., boosting exploration (Haarnoja et al., 2018) or safe policy improvement (Schulman et al., 2015).

Original: While Shannon entropy is often used as a policy regularizer (Ziebart et al., 2008), Geist et al.
Revised: While Shannon entropy is often used as a policy regularizer (Ziebart et al., 2008), Geist et al.

Original: (2019) recently proposed a theoretical foundation of regularized Markov decision processes (MDPs) – a framework that uses arbitrary convex functions as policy regularizers.
Revised: (2019) recently proposed a theoretical foundation of regularized Markov decision processes (MDPs)—a framework that uses strongly convex functions as policy regularizers.

Original: Here, one crucial advantage is that an optimal policy can be shown to uniquely exists, whereas multiple optimal policies may exist in the absence of policy regularization (and depending on the given reward structure).
Revised: Here, one crucial advantage is that an optimal policy is shown to uniquely exist, whereas multiple optimal policies may exist in the absence of policy regularization (and depending on the given reward structure).

Original: Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering), Inverse Reinforce Learning (IRL) (Russell, 1998; Ng et al., 2000) – the problem of acquiring a reward function that promotes expert-like behavior – is more generally adopted in practical scenarios like robotic manipulation (Finn et al., 2016b), autonomous driving (Sharifzadeh et al., 2016; Wu et al., 2020) and clinical motion analysis (Li et al., 2018).
Revised: Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering), Inverse Reinforce Learning (IRL) (Russell, 1998; Ng et al., 2000)—the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation (Finn et al., 2016b), autonomous driving (Sharifzadeh et al., 2016; Wu et al., 2020) and clinical motion analysis (Li et al., 2018).

Original: In these scenarios, deﬁning a reward function beforehand is particularly challenging and IRL is simply more pragmatic.
Revised: In these scenarios, deﬁning a reward function beforehand is particularly challenging and IRL is simply more pragmatic.

Original: However, complications with IRL in unregularized MDPs relate to the issue of degeneracy and early work has shown that degenerate solutions, where any constant function can be a solution of the expert’s behavior, can lead to suboptimal learned policies (Ng et al., 2000).
Revised: However, complications with IRL in unregularized MDPs relate to the issue of degeneracy, where any constant function can rationalize the expert’s behavior (Ng et al., 2000).

Original: Fortunately, Geist et al.
Revised: Fortunately, Geist et al.

Original: (2019) show that IRL in regularized MDPs – regularized IRL – do not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs.
Revised: (2019) show that IRL in regularized MDPs— regularized IRL —does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs.

Original: Despite this, no analytical solutions of regularized IRL – other than maximum-Shannon-entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010; Ho & Ermon, 2016; Finn et al., 2016a;Fu et al., 2018) – have been proposed.
Revised: Despite this, no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010; Ho & Ermon, 2016; Finn et al., 2016a;

Original: Solutions of a functional form were proposed in Geist et al.
Revised: Fu et al. , 2018)—have been proposed. In Geist et al.

Original: (2019), however they are generally intractable since  a closed-form relation between the policy and optimal value function is needed to derive their solutions. This is not applicable with arbitrary policy regularization.
Revised: (2019), solutions for regularized IRL were proposed. However, they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics.

Original: Furthermore, practical algorithms for solving regularized IRL problems have not yet been proposed.
Revised: Furthermore, practical algorithms for solving regularized IRL problems have not yet been proposed.

Original: We summarize our contribution as follows: unlike the solutions in Geist et al.
Revised: We summarize our contributions as follows: unlike the solutions in Geist et al.

Original: (2019), we propose tractable solutions for regularized IRL problems(Section 3) that require only knowledge of the analytical form of policy regularization.
Revised: (2019), we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and its gradient in discrete control problems (Section 3.1).

Original: We  show that our solutions are valid in continuous control problems including solutions for Tsallis entropy regularization with multi-variate Gaussian policies  (Section 3.2).
Revised: We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems (Section 3.2).

Original: We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized MDPs, which generalizes adversarial IRL (AIRL, Fu et al.
Revised: We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized MDPs, which generalizes adversarial IRL (AIRL, Fu et al.

Original: (Section 4).
Revised: (Section 4).

Original: Finally, we empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating both episodic scores and divergence minimization perspective  Ghasemipour et al. ; Dadashi et al.
Revised: Finally, we empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating both episodic scores and divergence minimization perspective (Ke et al., 2019; Ghasemipour et al., 2019; Dadashi et al., 2020)

Original: (Section 5).
Revised: (Section 5).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notation For ﬁnite sets X and Y , Y X is a set of functions from X to Y .
Revised: Notation For ﬁnite sets X and Y , Y X is a set of functions from X to Y .

Original: ∆ X ( ∆ XY ) is a set of (conditional) probabilities over X (conditioned on Y ).
Revised: ∆ X ( ∆ XY ) is a set of (conditional) probabilities over X (conditioned on Y ).

Original: Especially for the conditional probabilities p  ∈ ∆ XY , we say p  ( ·| y ) ∈
Revised: Especially for the conditional probabilities p X | Y ∈ ∆ XY , we say p X | Y ( ·| y ) ∈

Original: ∆ X for y ∈ Y .
Revised: ∆ X for y ∈ Y .

Original: R is the set of real numbers.
Revised: R is the set of real numbers.

Original: For functions f 1 , f 2 ∈ R X , the inner product between f 1 and f 2 on X is deﬁned as (cid:104) f 1 , f 2 (cid:105) X := (cid:80) x ∈X
Revised: For functions f 1 , f 2 ∈ R X , the inner product between f 1 and f 2 on X is deﬁned as (cid:104) f 1 , f 2 (cid:105) X := (cid:80) x ∈

Original:  f 1 ( x ) f 2 ( x ) .
Revised: X f 1 ( x ) f 2 ( x ) .

Original: Regularized Markov Decision Processes and Reinforcement Learning
Revised: Regularized Markov Decision Processes and Reinforcement Learning

Original: We consider sequential decision making problems where an agent sequentially chooses its actions after observing the state of the environment, and the environment in turn emits a reward and state transition.
Revised: We consider sequential decision making problems where an agent sequentially chooses its actions after observing the state of the environment, and the environment in turn emits a reward with state transition.

Original: Such an interaction between the agent and the environment is modeled as an inﬁnite-horizon MDP, M r :=
Revised: Such an interaction between the agent and the environment is modeled as an inﬁnite-horizon Markov Decision Process (MDP), M r :=

Original: (cid:104)S , A , P 0 , P, r, γ (cid:105) and the agent’s policy π ∈ ∆ AS .
Revised: (cid:104)S , A , P 0 , P, r, γ (cid:105) and the agent’s policy π ∈ ∆ AS .

Original: The terms within the MDP are deﬁned as follows: 
Revised: The terms within the MDP are deﬁned as follows: S is a ﬁnite state space,

Original: S is a ﬁnite state space, A is a ﬁnite action space,
Revised: A is a ﬁnite action space,

Original: P 0 ∈
Revised: P 0 ∈

Original: ∆ S is an initial state distribution, P ∈ ∆ S S×A is a state transition probability, r ∈ R S×A is a reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: ∆ S is an initial state distribution, P ∈ ∆ SS×A is a state transition probability, r ∈ R S×A is a reward function, and γ ∈ [0 , 1) is the discount factor.

Original: We also deﬁne an MDP without reward as M − := (cid:104)S , A , P 0 , P, γ (cid:105) .
Revised: We also deﬁne an MDP without reward as M − := (cid:104)S , A , P 0 , P, γ (cid:105) .

Original: The normalized state-action visitation distribution, d π ∈ ∆ S×A , associated with π is deﬁned as the expected discounted stateaction visitation of π , i.e., d π ( s, a ) :
Revised: The normalized state-action visitation distribution, d π ∈ ∆ S×A , associated with π is deﬁned as the expected discounted state-action visitation of π , i.e., d π ( s, a ) :

Original: = (1 − γ ) · E π [ (cid:80) ∞ i =

Revised: = ( [Equation] γ ) · E π [ (cid:80) ∞ i =

Original: γ i
Revised: γ i

Original: I { s i = s, a i = a } ] , where the subscript π on E means that a trajectory ( s 0 , a 0 , s 1 , a 1 , ... ) is randomly generated from M − and π , and I {·} is an indicator function.
Revised: I { s i = s, a i = a } ] , where the subscript π on E means that a trajectory ( s 0 , a 0 , s 1 , a 1 , ... ) is randomly generated from M − and π , and I {·} is an indicator function.

Original: Note that d π satisﬁes the transposed Bellman recurrence (Boularias & Chaib-Draa, 2010; Zhang et al., 2019):
Revised: Note that d π satisﬁes the transposed Bellman recurrence (Boularias & Chaib-Draa, 2010; Zhang et al., 2019):

Original: [Equation]
Revised: [Equation]

Original: We consider RL in regularized MDPs (Geist et al., 2019), where the policy is optimized with a causal convex regularizer.
Revised: We consider RL in regularized MDPs (Geist et al., 2019), where the policy is optimized with a causal policy regularizer.

Original: Mathematically for an MDP M r and a strongly convex function Ω : ∆ A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω :
Revised: Mathematically for an MDP M r and a strongly convex function Ω : ∆ A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω :

Original: [Equation]
Revised: [Equation]

Original: [Equation] It turns out that the optimal solution of Eq.(1) is unique (Geist et al., 2019), whereas multiple optimal policies may exist in unregularized MDPs.
Revised: It turns out that the optimal solution of Eq.(1) is unique (Geist et al., 2019), whereas multiple optimal policies may exist in unregularized MDPs (See Appendix A for detailed explanation).

Original: In later work (Yang et al., 2019), Ω( π ) = 
Revised: In later work (Yang et al., 2019), Ω( p ) = − λ [Equation] φ [Equation]

Original: − λ E a ∼ π φ ( π ( a )) , π ∈ D A was considered for λ > 0 and φ : (0 , 1] → R satisfying some mild conditions.
Revised: p [Equation] a )) , p ∈ ∆ A was considered for λ >and φ : (0 , 1] → R satisfying some mild conditions.

Original: For example, RL with Shannon entropy regularization (Haarnoja et al., 2018) can be recovered by φ ( π ( a | s ))
Revised: For example, RL with Shannon entropy regularization (Haarnoja et al., 2018) can be recovered by φ ( x )

Original: = − log π ( a | s ) , while RL with Tsallis entropy regularization (Lee et al., 2020) can be recovered from φ ( π ( a | s )) =
Revised: = − log x , while RL with Tsallis entropy regularization (Lee et al., 2020) can be recovered from φ ( x ) =

Original: kq − 1 (1 − π ( a | s ) q − 1 ) for k > 0 , q > 1 .
Revised: kq − 1 (1 − x q − 1 ) for k > 0 , q > 1 .

Original: The optimal policy π ∗ for Eq.(1) with Ω from Yang et al.
Revised: The optimal policy π ∗ for Eq.(1) with Ω from Yang et al.

Original: (2019) is shown to be
Revised: (2019) is shown to be

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where φ (cid:48)
Revised: where φ (cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: ∂∂x φ ( x ) , g φ is an inverse function of f (cid:48) φ for f φ ( x ) :
Revised: ∂∂x φ ( x ) , g φ is an inverse function of f (cid:48) φ for f φ ( x ) :

Original: =
Revised: =

Original: xφ ( x ) , x ∈ (0 , 1] , and µ ∗ is a normalization term such that (cid:80) a ∈A π ∗ ( a | s ) = 1 .
Revised: xφ ( x ) , x ∈ (0 , 1] , and µ ∗ is a normalization term such that (cid:80) a ∈A π ∗ ( a | s ) = 1 .

Revised: Note that we need to solve constraint optimization problem w.r.t.

Original: Depending on λ and φ , the sparse optimal policy can be derived in Eq.(3) where sparsity is deﬁned as having zero probability for some actions. In contrast, dense optimal policies can only be obtained with Shannon entropy regularization  and bounded rewards.
Revised: µ ∗ to derive a closed-form relation between optimal policy π ∗ and value function Q ∗ . However, such closed-form relations have not been discovered except for Shannonentropy regularization (Haarnoja et al., 2018) and speciﬁc instances ( q = 1 , 2 , ∞ ) of Tsallis-entropy regularization (Lee et al., 2019) to the best of our knowledge.

Original: Inverse Reinforcement Learning Given a set of demonstrations from an expert’s policy π E ,
Revised: Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E ,

Original: IRL (Russell, 1998; Ng et al., 2000) is the problem of seeking a reward function from which we can recover π E through RL.
Revised: IRL (Russell, 1998; Ng et al., 2000) is the problem of seeking a reward function from which we can recover π E through RL.

Original:  IRL in unregularized MDPs hashowever been shown to be an ill-deﬁned problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng et al., 2000).
Revised: However, IRL in unregularized MDPs has been shown to be an ill-deﬁned problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng et al., 2000).

Original: Maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010) is capable of solving the ﬁrst issue by seeking a reward function that maximizes expert’s return along with Shannon entropy of expert’s policy.
Revised: Maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010) is capable of solving the ﬁrst issue by seeking a reward function that maximizes expert’s return along with Shannon entropy of expert policy.

Original: Mathematically for the RL objective J Ω in Eq.(1) and Ω = −H for negative Shannon entropy H ( π ) =
Revised: Mathematically for the RL objective J Ω in Eq.(1) and Ω = −H for negative Shannon entropy H ( p ) =

Original: E a ∼ π [ − log π ( a )] 
Revised: E a ∼ p [ − log p ( a )] (Ho &

Original: (Ho & Ermon, 2016), the objective of MaxEntIRL is
Revised: Ermon, 2016), the objective of MaxEntIRL is

Original: [Equation]
Revised: [Equation]

Original: Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) which involves generative adversarial training (Goodfellow et al., 2014; Ho & Ermon, 2016) to acquire a solution of MaxEntIRL.
Revised: Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) which involves generative adversarial training (Goodfellow et al., 2014; Ho & Ermon, 2016) to acquire a solution of MaxEntIRL.

Original: AIRL considers the structured discriminator (Finn et al., 2016a)
Revised: AIRL considers the structured discriminator (Finn et al., 2016a)

Original: D ( s, a ) =
Revised: D ( s, a ) =

Original: σ ( r ( s, a ) − log π ( a | s ))
Revised: σ ( r ( s, a ) − log π ( a | s ))

Original: =
Revised: =

Original: [Equation]

Original: s

Original: [Equation]

Original: a

Original: [Equation]

Original: e

Original: [Equation] ( s [Equation] a ) + π ( a | s ) for σ ( x ) :
Revised: e r ( s,a ) e r ( s,a ) + π ( a | s ) for σ ( x ) :

Original: = 1 / (1 + e − x ) and iteratively optimizes the following objective:
Revised: = 1 / (1 + e − x ) and iteratively optimizes the following objective:

Original: [Equation]
Revised: [Equation]

Original: It turns out that AIRL minimizes the divergence between visitation distributions d π and d π E by solving min π ∈ ∆ AS KL( d π || d π E ) for Kullback-Leibler (KL) divergence KL (Ghasemipour et al., 2019) .
Revised: It turns out that AIRL minimizes the divergence between visitation distributions d π and d π E by solving min π ∈ ∆ AS KL( d π || d π E ) for Kullback-Leibler (KL) divergence KL (Ghasemipour et al., 2019) .

Original:  I NVERSE R EINFORCEMENT L EARNING IN R EGULARIZED MDP S
Revised:  I NVERSE R EINFORCEMENT L EARNING IN R EGULARIZED MDP S

Original: In this section, we propose the solution of IRL in regularized MDPs and relevant properties in Section 3.1.
Revised: In this section, we propose the solution of IRL in regularized MDPs— regularized IRL —and relevant properties in Section 3.1.

Original: We then discuss a speciﬁc instance of our proposed solution where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.
Revised: We then discuss a speciﬁc instance of our proposed solution where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.

Original: 3. S OLUTIONS OF REGULARIZED IRL
Revised: 3. S OLUTIONS OF REGULARIZED IRL

Original: We consider regularized IRL that generalizes MaxEntIRL in Eq.(4) to IRL with ageneral class of  convex policy regularizers:
Revised: We consider regularized IRL that generalizes MaxEntIRL in Eq.(4) to IRL with a class of strongly convex policy regularizers:

Original: [Equation]
Revised: [Equation]

Original: For any convex policy regularizer Ω , regularized IRL does not suffer from degenerate solutions since there is a unique optimal policy in any regularized MDP (Geist et al., 2019).
Revised: For any convex policy regularizer Ω , regularized IRL does not suffer from degenerate solutions since there is a unique optimal policy in any regularized MDP (Geist et al., 2019).

Original: While Geist et al.
Revised: While Geist et al.

Original: (2019) proposedfunctional-form solutions of regularized IRL – which is intractable in general
Revised: (2019) proposed solutions of regularized IRL, those are intractable solutions (See Appendix F.1 for detailed explanation).

Original: – we propose tractable solutions in the following lemma.
Revised: In the following lemma, we propose tractable that only requires the evaluation of policy regularizer ( Ω ) and its gradient ( ∇ Ω ) which is more tractable in practice.

Original: Our solution is motivated from ﬁguring out a reward function that is capable of converting regularized RL into equivalent divergence minimization problem between π and π E :
Revised: Our solution is motivated from ﬁguring out a reward function that is capable of converting regularized RL into equivalent divergence minimization problem associated with π and π E :

Original: Lemma 1.
Revised: Lemma 1.

Original: For a policy regularizer Ω : ∆ A → R , let us deﬁne
Revised: For a policy regularizer Ω : ∆ A → R , let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) for Ω (cid:48) ( s, · ; π ) := ∇ Ω( π ( ·| s )) :=
Revised: for Ω (cid:48) ( s, · ; π ) := ∇ Ω( π ( ·| s )) :

Original:  [ ∇ x Ω( x )] x = π ( ·| s ) ∈ R A , s ∈ S .
Revised: = [ ∇ p Ω( p )] p = π ( ·| s ) ∈ R A , s ∈ S .

Original: Then, t ( s, a ; π E ) for expert’s policy π E is a solution of regularized IRL with Ω .
Revised: Then, t ( s, a ; π E ) for expert policy π E is a solution of regularized IRL with Ω .

Original: Proof.
Revised: Proof.

Original: (Abbreviated.
Revised: (Abbreviated.

Original: See Appendix A for full version.)
Revised: See Appendix B for full version.)

Original: With r ( s, a ) =
Revised: With r ( s, a ) =

Original: t ( s, a ; π E ) , the RL objective in Eq.(1) becomes equivalent to a problem of minimizing the discounted sum of Bregman divergences between π and π Earg min
Revised: t ( s, a ; π E ) , the RL objective in Eq.(1) becomes equivalent to a problem of minimizing the discounted sum of Bregman divergences between π and π E

Original: [Equation]
Revised: [Equation]

Original: where D A Ω is the Bregman divergence (Bregman, 1967) deﬁned by D A Ω ( π 1 || π 2 )
Revised: where D A Ω is the Bregman divergence (Bregman, 1967) deﬁned by D A Ω ( p 1 || p 2 )

Original: =
Revised: =

Original: Ω( π 1 ) − Ω( π 2 ) − (cid:104)∇ Ω( π 2 ) , π 1 − π 2 (cid:105)
Revised: Ω( p 1 ) − Ω ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105)

Original: A for π 1 , π 2 ∈ ∆ A .
Revised: A for p 1 , p 2 ∈ ∆ A .

Original: Due to the non-negativity of the Bregman divergence, π = π E is a solution of Eq.(8) and is unique since Eq.(1) has the unique solution for arbitrary reward functions (Geist et al., 2019).
Revised: Due to the non-negativity of the Bregman divergence, π = π E is a solution of Eq.(8) and is unique since Eq.(1) has the unique solution for arbitrary reward functions (Geist et al., 2019).

Original: Especially for any policy regularizer Ω represented with an expectation over the policy (Yang et al., 2019), Lemma 1 can be reduced to the following solution in Corollary 1 :
Revised: Especially for any policy regularizer Ω represented by an expectation over the policy (Yang et al., 2019), Lemma 1 can be reduced to the following solution in Corollary 1 :

Original: Corollary 1.
Revised: Corollary 1.

Original: (See Appendix A for proof).

Original: For Ω( π ) = − λ E a ∼ π φ ( π ( a )) with π ∈ ∆
Revised: For Ω( p ) = − λ E a ∼ p φ ( p ( a )) with p ∈ ∆

Original: A (Yang et al., 2019) , Eq.(7) becomes
Revised: A (Yang et al., 2019) , Eq.(7) becomes

Original: [Equation]
Revised: [Equation]

Original: for f (cid:48) φ ( x ) =
Revised: for f (cid:48) φ ( x ) =

Original: ∂∂x ( xφ ( x )) .
Revised: ∂∂x ( xφ ( x )) .

Original:  Throughout the paper, we denote reward baseline by the expectation E a ∼ π ( ·| s )
Revised: The proof is in Appendix B. Throughout the paper, we denote reward baseline by the expectation E a ∼ π ( ·| s )

Original: [ f (cid:48) φ ( π ( a | s )) − φ
Revised: [ f (cid:48) φ ( π ( a | s )) − φ

Original: π ( a | s ))] .
Revised: π ( a | s ))] .

Original: For continuous control tasks with Ω( π ) =
Revised: Note that for continuous control tasks with Ω( p ) =

Original: − λ E a ∼ π φ ( π ( a )) , we can obtain the same form of the reward in Eq.(9)
Revised: − λ E a ∼ p φ ( p ( a )) , we can obtain the same form of the reward in Eq.(9)

Original: (The proof is in Appendix C).
Revised: (The proof is in Appendix D).

Original: When λ
Revised: Although the reward baseline is generally not a intractable in continuous control tasks, we derive a tractable reward baseline for a special case (See Section 3.2).

Original:  = 1 and φ ( x ) = − log x , it can be shown that t ( s, a ; π ) =
Revised: Additionally, when λ = 1 and φ ( x ) = − log x , it can be shown that t ( s, a ; π ) =

Original: log π ( a | s ) , which was used as a reward objective in the previous work (Fu et al., 2018), and that the Bregman divergence in Eq.(8) becomes the KL divergence KL( π ( ·| s ) ||
Revised: log π ( a | s ) —for both discrete and continuous control problems—which was used as a reward objective in the previous work (Fu et al., 2018), and that the Bregman divergence in Eq.(8) becomes the KL divergence KL( π ( ·| s ) ||

Original: π E ( ·| s )) .
Revised: π E ( ·| s )) .

Original: Additionally, other solutions of  IRL can be found by shaping t ( s, a ; π E ) as stated in the following lemma:
Revised: Additional solutions of regularized IRL can be found by shaping t ( s, a ; π E ) as stated in the following lemma:

Original: Lemma 2.
Revised: Lemma 2 (Potential-based reward shaping) .

Original: Let π ∗ be the solution of Eq.(1) in a regularized MDP M r with a regularizer Ω : ∆ A → R and an arbitrary reward function r ∈ R S×A .
Revised: Let π ∗ be the solution of Eq.(1) in a regularized MDP M r with a regularizer Ω : ∆ A → R and a reward function r ∈ R S×A .

Original: Then, rewards r ( s, a )  − Φ( s ) with Φ : S → R does not change the solution of Eq.(1).
Revised: Then for Φ ∈ R S , using either r ( s, a ) + γ Φ( s (cid:48) ) − Φ( s ) or r ( s, a ) + E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) ) − Φ( s ) as a reward does not change the solution of Eq.(1).

Original: The proof is in Section D.4.
Revised: The proof is in Appendix E.

Original: of Geist et al.

Original: From Lemma 1 and Lemma 2 , we prove the sufﬁcient condition of rewards being solutions of the
Revised: From Lemma 1 and Lemma 2 , we prove the sufﬁcient condition of rewards being solutions of the

Original: IRL problem.
Revised: IRL problem.

Original: However, the necessary condition – a set of those solutions are the only possible solutions – is not proved.
Revised: However, the necessary condition—a set of those solutions are the only possible solutions for an arbitrary MDP—is not proved (Ng et al., 1999).

Original: We also check how the  solutionof the IRL problem is related to the normalized state-visitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho & Ermon, 2016;Fu et al., 2018;
Revised: In the following lemma, we check how the proposed solution is related to the normalized statevisitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho & Ermon, 2016;

Original:  Ghasemipour et al., 2019):
Revised: Fu et al., 2018; Ke et al., 2019; Ghasemipour et al., 2019):

Original: Lemma 3.
Revised: Lemma 3.

Original: Given the policy regularizer Ω , let us deﬁne ¯Ω( d ) :
Revised: Given the policy regularizer Ω , let us deﬁne ¯Ω( d ) :

Original: = E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] for an arbitrary normalized state-action visitation distribution d ∈ ∆ S×A and the policy ¯ π d ( a | s ) := d ( s,a ) [Equation] d
Revised: = E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] for an arbitrary normalized state-action visitation distribution d ∈ ∆ S×A and the policy ¯ π d ( a | s ) :

Original: [Equation] s,a [Equation] ) induced by d .
Revised: = d ( s,a ) (cid:80) a (cid:48)∈A d ( s,a (cid:48) ) induced by d .

Original: Then, Eq.(7) is equal to
Revised: Then, Eq.(7) is equal to

Original: [Equation]
Revised: [Equation]

Original: When ¯Ω( d ) is strictly convex and a solution t ( s, a ; π E ) = ∇ ¯Ω( d π E ) of IRL in Eq.(10) is used, the RL objective in Eq.
Revised: When ¯Ω( d ) is strictly convex and a solution t ( s, a ; π E ) = ∇ ¯Ω( d π E ) of IRL in Eq.(10) is used, the RL objective in Eq.

Original: (1) is equal to arg min π ∈ ∆ AS
Revised: (1) is equal to [Equation]

Original: D S×A ¯ Ω ( d π || d π E ) , where D S×A ¯Ω is a Bregman divergence among visitation distributions deﬁned by D S×A ¯Ω ( d 1 || d 2 ) =
Revised: where D S×A ¯Ω is the Bregman divergence among visitation distributions deﬁned by D S×A ¯Ω ( d 1 || d [Equation]

Original:  ¯Ω( d 1 ) − ¯Ω( d 2 ) −(cid:104)∇ ¯Ω( d 2 ) , d 1 − d 2 (cid:105) for visitation distributions d 1 and d 2 .
Revised: = ¯Ω( d 1 ) − ¯Ω( d 2 ) −(cid:104)∇ ¯Ω( d 2 ) , d 1 − d 2 (cid:105) for visitation distributions d 1 and d 2 .

Original: The proof is in Appendix D.
Revised: The proof is in Appendix G.

Revised: Note that the strict convexity of ¯Ω is required for D S×A ¯Ω to become a valid Bregman divergence.

Original: Here, although the strict convexity of a policy regularizer Ω does not guarantee our assumption on the strict convexity of ¯Ω , it has been shown to be true for Shannon (Ho & Ermon, 2016) and Tsallis entropy regularizations (Lee et al., 2018).
Revised: Although the strict convexity of a policy regularizer Ω does not guarantee the assumption on the strict convexity of ¯Ω , it has been shown to be true for Shannon entropy regularizer ( ¯Ω = − ¯ H of Lemma 3.1 in Ho & Ermon (2016)) and Tsallis entropy regularizer with its constants k = 12 , q =( ¯ Ω = − ¯ W of Theorem 3 in Lee et al.

Original: 3. IRL WITH T SALLIS ENTROPY REGULARIZATION AND G AUSSIAN POLICIES
Revised: 3. IRL WITH T SALLIS ENTROPY REGULARIZATION AND G AUSSIAN POLICIES

Original: For continuous controls, multi-variate Gaussian policies are widely used in practice (Schulman et al., 2015; 2017) and we consider IRL problems with those policies in this subsection.
Revised: For continuous controls, multi-variate Gaussian policies are often used in practice (Schulman et al., 2015; 2017) and we consider IRL problems with those policies in this subsection.

Original: In particular, we consider IRL with Tsallis entropy regularizer Ω
Revised: In particular, we consider IRL with Tsallis entropy regularizer Ω

Original: ( π ( ·| s ))
Revised: ( p ) =

Original: = −T kq ( π ( ·| s )) =
Revised: −T kq ( p ) =

Original: − E a ∼ π ( ·| s )
Revised: − E a ∼ p

Original: [ kq − 1 (1 − π ( a| s ) q − 1 )] (Lee et al., 2018;
Revised: [ kq − 1 (1 − p ( a ) q − 1 )] (Lee et al., 2018;

Original: Yang et al., 2019; Lee et al., 2020) for a multi-variate Gaussian policy π ( ·| s ) =
Revised: Yang et al., 2019; Lee et al., 2020) for a multi-variate Gaussian policy π ( ·| s ) =

Original: N ( µµµ ( s ) , ΣΣΣ( s )) with µµµ ( s ) =
Revised: N ( µµµ ( s ) , ΣΣΣ( s )) with µµµ ( s ) =

Original: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } .
Revised: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } .

Original: In such a case, we can obtain analytical forms of the following quantities:
Revised: In such a case, we can obtain tractable forms of the following quantities:

Original: Tsallis entropy.
Revised: Tsallis entropy.

Original: The analytical form of Tsallis entropy for a multi-variate Gaussian policy is
Revised: The tractable form of Tsallis entropy for a multi-variate Gaussian policy is

Original: [Equation]
Revised: [Equation]

Original: for Renyi entropy R q .
Revised: for Renyi entropy R q .

Original: Its derivation is given in Appendix F.
Revised: Its derivation is given in Appendix I.

Original: Reward baseline.
Revised: Reward baseline.

Original: In general, the reward baseline term E a ∼ π ( ·| s )
Revised: The reward baseline term E a ∼ π ( ·| s )

Original: [ f (cid:48) φ ( π ( a | s )) − φ
Revised: [ f (cid:48) φ ( π ( a | s )) − φ

Original: π ( a | s ))] in Corollary 1 is generally intractable except for either discrete control problems orwhen Shannon entropy regularizationis considered – where the reward baseline is equal to − 1 .
Revised: π ( a | s ))] in Corollary 1 is generally intractable except for either discrete control problems or Shannon entropy regularization where the reward baseline is equal to − 1 .

Original: Interestingly, as long as the analytical form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisﬁes
Revised: Interestingly, as long as the tractable form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Here, the ﬁrst equality holds with f 
Revised: Here, the ﬁrst equality holds with f (cid:48) φ ( x ) = kq −

Original: [Equation]
Revised: [Equation]

Original:  for Tsallis entropy regularization.
Revised: 1 − qx q [Equation] = qφ ( x ) − k for Tsallis entropy regularization.

Original: Bregman divergence associated with Tsallis entropy.
Revised: Bregman divergence associated with Tsallis entropy.

Original: For two different multivariate Gaussian policies, we derive the analytical form of the Bregman divergence (associated with Tsallis entropy) between two policies.
Revised: For two different multivariate Gaussian policies, we derive the tractable form of the Bregman divergence (associated with Tsallis entropy) between two policies.

Original: The resultant divergence has a complicated form, so we leave it in Appendix F. with its derivation.
Revised: The resultant divergence has a complicated form, so we leave it in Appendix I. with its derivation.

Original: For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized
Revised: For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized

Original: MDPs, we consider an example in Figure 1.
Revised: MDPs, we consider an example in Figure 1.

Original: We ﬁrst assume that both learning agents’ and experts’ policies follow uni-variate Gaussian distributions π = N ( µ, σ 2 ) and π E = N (0 , ( e − 3 ) 2 ) , respectively.
Revised: We ﬁrst assume that both learning agents’ and experts’ policies follow uni-variate Gaussian distributions π = N ( µ, σ 2 ) and π E = N (0 , ( e − 3 ) 2 ) , respectively.

Original: We then evaluate the Bregman divergenceby using the analytical form of the above quantities in Figure 1 by  varying q from 1.0 – which corresponds to the KL divergence – to 2.0.
Revised: We then evaluate the Bregman divergence in Figure 1 by using its tractable form and varying q from 1.0—which corresponds to the KL divergence—to 2.0.

Original: We observe that the constant q from the Tsallis entropy affects the sensitivity of the associated Bregman divergence w.r t. the mean and standard deviation of the learning agent’s policy π .
Revised: We observe that the constant q from the Tsallis entropy affects the sensitivity of the associated Bregman divergence w.r.t. the mean and standard deviation of the learning agent’s policy π .

Original: Speciﬁcally, as q increases, the size of the valley – relatively red regime in Figure 1 – across the µ -axis and log σ -axis decreases.
Revised: Speciﬁcally, as q increases, the size of the valley— relatively red regime in Figure 1—across the µ -axis and log σ -axis decreases.

Original: This suggests that for larger q , minimizing the Bregman divergence requires more tightly matching means and variances of π and π E .
Revised: This suggests that for larger q , minimizing the Bregman divergence requires more tightly matching means and variances of π and π E .

Original:  A LGORITHMIC C ONSIDERATION
Revised:  A LGORITHMIC C ONSIDERATION

Original: Although we derive a solution for  IRLproblems with a general policy regularizer Ω in Lemma 1 , we focus on developing an IRL algorithm for Ω( π ) = − λ E a ∼ π [ φ ( π ( a ))] (Yang et al., 2019).
Revised: Based on a solution for regularized IRL in the previous section, we focus on developing an IRL algorithm in this section.

Original: To recover the reward function t ( s, a ; π E ) in Corollary 1 , we design an adversarial training objective as follows.
Revised: Particularly to recover the reward function t ( s, a ; π E ) in Lemma 1 , we design an adversarial training objective as follows.

Original: Similar to AIRL (Fu et al., 2018), we consider the following structured discriminator associated with π, r and t in Lemma 1 :
Revised: Motivated by AIRL (Fu et al., 2018), we consider the following structured discriminator associated with π, r and t in Lemma 1 :

Original: [Equation]
Revised: [Equation]

Original: Note that we can recover the discriminator of AIRL in Eq.(5) when t ( s, a ) = log π ( a | s ) ( φ ( x ) = log x and λ = 1 ).
Revised: Note that we can recover the discriminator of AIRL in Eq.(5) when t ( s, a ) = log π ( a | s ) ( φ ( x ) = log x and λ = 1 ).

Original: Then, we consider the following optimization objective of the discriminator which is the same as that of AIRL:
Revised: Then, we consider the following optimization objective of the discriminator which is the same as that of AIRL:

Original: [Equation]
Revised: [Equation]

Original: Since the function x (cid:55)→ a log σ ( x ) + b log(1 − σ ( x )) attains its maximum at σ ( x ) = aa + b , or equivalently at x = log ab (Goodfellow et al., 2014; Mescheder et al., 2017), it can be shown that
Revised: Since the function x (cid:55)→ a log σ ( x ) + b log(1 − σ ( x )) attains its maximum at σ ( x ) = aa + b , or equivalently at x = log ab (Goodfellow et al., 2014; Mescheder et al., 2017), it can be shown that

Original: [Equation]
Revised: [Equation]

Original: When π = π E in Eq.(12), we have ˆ t ( s, a ; π E ) =
Revised: When π = π E in Eq.(12), we have ˆ t ( s, a ; π E ) =

Original: t ( s, a ; π E ) since d π = d π E , which means the maximizer ˆ t becomes the solution of IRL after the agent successfully imitates the expert’s policy π E .
Revised: t ( s, a ; π E ) since d π = d π E , which means the maximizer ˆ t becomes the solution of IRL after the agent successfully imitates the expert policy π E .

Original: To do so, we consider the following iterative algorithm.
Revised: To do so, we consider the following iterative algorithm.

Original: Assuming that we ﬁnd out the optimal  ˆ t ( s, a ; π ( i ) ) in Eq.(12) for the policy π ( i ) of the i -th iteration, we get the policy π
Revised: Assuming that we ﬁnd out the optimal reward approximator ˆ t ( s, a ; π ( i ) ) in Eq.(12) for the policy π ( i ) of the i -th iteration, we get the policy π

Original: ( i +1) by optimizing the following objective:
Revised: ( i +1) by optimizing the following objective with gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: Note that the expectation in Eq.(13) can be decomposed into the following two terms:
Revised: The above expectation in Eq.(13) can be decomposed into the following two terms

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows since Lemma 1 shows us that t ( s, a ; π ( i ) ) is a reward function that makes π ( i ) an optimal policy in the Ω -regularized MDP.
Revised: where the second equality follows since Lemma 1 tells us that t ( s, a ; π ( i ) ) is a reward function that makes π ( i ) an optimal policy in the Ω -regularized MDP.

Original: Minimizing term (II) in Eq.(14) makes π ( i +1) close to π E while minimizing term (I) can be regarded as a conservative policy optimization around the policy π
Revised: Minimizing term (II) in Eq.(14) makes π ( i +1) close to π E while minimizing term (I) can be regarded as a conservative policy optimization around the policy π

Original: ( i ) (Schulman et al., 2015).
Revised: ( i ) (Schulman et al., 2015).

Original: In practice, we parameterize our reward and policy approximations with neural networks and train them using an off-policy Regularized Actor-Critic (RAC)
Revised: In practice, we parameterize our reward and policy approximations with neural networks and train them using an off-policy Regularized Actor-Critic (RAC)

Original: (Yang et al., 2019) as described in Algorithm 1 in Appendix G. We evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios, below.
Revised: (Yang et al., 2019) as described in Algorithm 1 .We evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios, below.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We summarize the experimental setup as follows.
Revised: We summarize the experimental setup as follows.

Original: In our experiments, we consider Ω( π ) =
Revised: In our experiments, we consider Ω( p ) =

Original: − λ E a ∼ π
Revised: − λ E a ∼ p

Original: [ φ ( π ( a )] with the following regularizers from Yang et al.
Revised: [ φ ( p ( a )] with the following regularizers from Yang et al.

Original: (2019): (1) Shannon entropy ( φ ( x ) = − log x ), (2) Tsallis entropy regularizer ( φ ( x ) =
Revised: (2019): (1) Shannon entropy ( φ ( x ) = − log x ), (2) Tsallis entropy regularizer ( φ ( x ) =

Original: kq − 1 (1 − x q − 1 ) ), (3) exp regularizer ( φ ( x ) =
Revised: kq − 1 (1 − x q − 1 ) ), (3) exp regularizer ( φ ( x ) =

Original: e − e x ), (4) cos regularizer ( φ ( x ) = cos( π 2 x ) ), (5) sin regularizer ( φ ( x ) = 1 − sin π 2 x ).
Revised: e − e x ), (4) cos regularizer ( φ ( x ) = cos( π 2 x ) ), (5) sin regularizer ( φ ( x ) = 1 − sin π 2 x ).

Revised: The above regularizers were chosen since other regularizers have not been empirically validated to the best of our knowledge.

Original:  In addition, we model the reward approximator in Algorithm 1 as a neural network with either one of the following models: (1) Non-structured model (NSM) – a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018) – and (2) Density-based model (DBM) (Figure H. in Appendix) – a model using a neural network for π (softmax for discrete controls and multi-variate 
Revised: We chose those regularizers to make our empirical analysis more tractable. In addition, we model the reward approximator of RAIRL as a neural network with either one of the following models: (1) Non-structured model (NSM) —a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018)—and (2) Density-based model (DBM) —a model using a neural network for π (softmax for discrete controls and multi-variate Gaussian model for continuous controls) of the solution in Eq.(1) (See Appendix J.2 for detailed explanation).

Original: Gaussian model for continuous controls) of the solution in Eq.(1).
Revised: For RL algorithm of RAIRL, we implement Regularized Actor Critic (RAC) (Yang et al., 2019) on top of SAC implementation of Rlpyt (Stooke & Abbeel, 2019).

Original: Other settings are summarized in
Revised: Other settings are summarized in

Original: Appendix H. For all experiments, we use 5 runs and report 95% conﬁdence interval.
Revised: Appendix J. For all experiments, we use 5 runs and report 95% conﬁdence interval.

Original: 5. E XPERIMENT 1: M ULTI - ARMED B ANDIT (D ISCRETE A CTION )
Revised: 5. E XPERIMENT 1: M ULTI - ARMED B ANDIT (D ISCRETE A CTION )

Original: We consider a 4-armed bandit environment as shown in Figure 2 (left).
Revised: We consider a 4-armed bandit environment as shown in Figure 2 (left).

Original: An expert’s policy π E is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for a = 0 , 1 , 2 , 3 ) or sparse (with probability 0, 0, 1/3, 2/3 for a = 0 , 1 , 2 , 3 ).
Revised: An expert policy π E is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for a = 0 , 1 , 2 , 3 ) or sparse (with probability 0, 0, 1/3, 2/3 for a = 0 , 1 , 2 , 3 ).

Original: For those experts, we use RAIRL with actions sampled from π E and compare learned rewards with the ground truth reward t ( s, a ; π E ) in Lemma 1 .
Revised: For those experts, we use RAIRL with actions sampled from π E and compare learned rewards with the ground truth reward t ( s, a ; π E ) in Lemma 1 .

Original: When π E is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices.
Revised: When π E is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices.

Original: When sparse π E is used, however, RAIRL with a non-structured model (RAIRL-NSM) failed to recover the rewards for a = 0 , 1 [-– where π E ( a ) = 
-]
Revised: When sparse π E is used, however, RAIRL with a non-structured model (RAIRL-NSM) failed to recover the rewards for a = 0 , 1 [-– where π E ( a ) = 
-]—

Original: – due to the lack of samples at the end of the imitation.
Revised: where π E ( a ) = 0 —due to the lack of samples at the end of the imitation.

Original: On the other hand, RAIRL with a density-based model (RAIRL-DBM) can recover the correct rewards due to the softmax layer which maintains the sum over the outputs equal to 1.
Revised: On the other hand, RAIRL with a density-based model (RAIRL-DBM) can recover the correct rewards due to the softmax layer which maintains the sum over the outputs equal to 1.

Original: Therefore, we argue that using DBM is necessary for correct reward acquisition since a set of demonstration is generally sparse.
Revised: Therefore, we argue that using DBM is necessary for correct reward acquisition since a set of demonstration is generally sparse.

Original: In the following experiment, we show the choice of reward models indeed affects the performance of rewards.
Revised: In the following experiment, we show the choice of reward models indeed affects the performance of rewards.

Original: 5. E XPERIMENT 2: B ERMUDA W ORLD (C ONTINUOUS O BSERVATION , D ISCRETE A CTION )
Revised: 5. E XPERIMENT 2: B ERMUDA W ORLD (C ONTINUOUS O BSERVATION , D ISCRETE A CTION )

Original: We consider an environment with a 2-dimensional continuous state space as described in Figure 3.
Revised: We consider an environment with a 2-dimensional continuous state space as described in Figure 3.

Original: At each episode, the learning agent is initialized uniformly on the x -axis between − 5 and 5 , and there are 8 possible actions – an angle in {− π, − 34 π, ..., 12 π, 34 π } that determines the direction of movement.
Revised: At each episode, the learning agent is initialized uniformly on the x -axis between − 5 and 5 , and there are 8 possible actions—an angle in {− π, − 34 π, ..., 12 π, 34 π } that determines the direction of movement.

Original: An expert in Bermuda World considers 3 target positions ( − 5 , 10) , (0 , 10) , (5 , 10) and behaves stochastically as shown in Figure 3 ( Left ).
Revised: An expert in Bermuda World considers 3 target positions ( − 5 , 10) , (0 , 10) , (5 , 10) and behaves stochastically. We state how we mathematically deﬁne the expert policy π E in Appendix J.3.

Original: With 1000 demonstrations sampled from the expert, the agent is evaluated during training with the Bregman divergences that correspond to each policy regularizer.
Revised: During RAIRL’s training (Figure 3, Top row ), we use 1000 demonstrations sampled from the expert and periodically measure mean Bregman divergence, i.e., for D A Ω ( p 1 || p 2 ) = E a ∼ p 1 [ f (cid:48) φ ( p 2 ( a )) − φ ( p 1 ( a ))] − E a ∼ p 2 [ f (cid:48) φ ( p 2 ( a )) − φ ( p 2 ( a ))] , N

Revised: N (cid:88) i =

Revised: D A Ω ( π ( ·| s i ) ||

Original: RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM although both achieve comparable performances during training.
Revised: π E ( ·| s i )) . Here, the states s 1 , ..., s N comes from 30 evaluation trajectories that are stochastically sampled from the agent’s policy π —which is ﬁxed during evaluation—in a separate evaluation environment. During the evaluation of learned reward (Figure 3, Bottom row ), we train randomly initialized agents with RAC and rewards acquired from RAIRL’s training and check if mean Bregman divergence is properly minimized.

Revised: We measure mean Bregman divergence as was done in RAIRL’s training.

Revised: RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM during reward evaluation although both achieve comparable performances during RAIRL’s training.

Original: Moreover, we substitute λ with 1 , 5 , 10 and observe that learning with λ larger than 1 returns better rewards – only λ = 1 was considered in AIRL (Fu et al., 2018).
Revised: Moreover, we substitute λ with 1 , 5 , 10 and observe that learning with λ larger than 1 returns better rewards—only λ = 1 was considered in AIRL (Fu et al., 2018).

Original: Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC).
Revised: Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC).

Original: This is because BC performs sufﬁciently well whensufﬁciently many demonstration is given.
Revised: This is because BC performs sufﬁciently well when many demonstration is given.

Original: We think the divergence of BC may be the near-optimal divergence that can be achieved with our policy neural network model.
Revised: We think the divergence of BC may be the near-optimal divergence that can be achieved with our policy neural network model.

Original: 5. E XPERIMENT 3: M U J O C O (C ONTINUOUS O BSERVATION AND A CTION )
Revised: 5. E XPERIMENT 3: M U J O C O (C ONTINUOUS O BSERVATION AND A CTION )

Original: We evaluate RAIRL on MuJoCo continuous control tasks ( Hopper-v2 , Walker-v2 , HalfCheetah-v2 ,
Revised: We validate RAIRL on MuJoCo continuous control tasks ( Hopper-v2 , Walker-v2 , HalfCheetah-v2 ,

Original: Ant-v2 ) as follows.
Revised: Ant-v2 ) as follows.

Original: We assume multivariate-Gaussian policies (with diagonal covairance matrices) for both learner’s policy π and expert’s policy π E .
Revised: We assume multivariate-Gaussian policies (with diagonal covairance matrices) for both learner’s policy π and expert policy π E .

Original: Instead of tanh -squashed policy in Soft-Actor
Revised: Instead of tanh -squashed policy in Soft-Actor

Original: Critic (Haarnoja et al., 2018), we use hyperbolized environments – where tanh is regarded as a part of the environment – with additional engineering on the policy networks (See Appendix H.4 for details).
Revised: Critic (Haarnoja et al., 2018), we use hyperbolized environments—where tanh is regarded as a part of the environment—with additional engineering on the policy networks (See Appendix J.4 for details).

Original: We use 100 demonstrations  from π E to evaluate RAIRL.
Revised: We use 100 demonstrations stochastically sampled from π E to validate RAIRL.

Original: In MuJoCo experiments, we focus on a set of Tsallis entropy regularizer ( Ω( π ) = −T 1 q( π ) ) with q = 1 , 1 . 5 , 2 – where Tsallis entropy becomes Shannon entropy for q = 1 .
Revised: In MuJoCo experiments, we focus on a set of Tsallis entropy regularizer ( Ω = −T 1 q ) with q = 1 , 1 . 5 , 2 —where Tsallis entropy becomes Shannon entropy for q = 1 .

Original: We then exploit the analytical quantities for multi-variate Gaussian distributions in Section 3.2 to stabilize RAIRL and evaluate its performance in terms of various Bregman divergences.
Revised: We then exploit the tractable quantities for multi-variate Gaussian distributions in Section 3.2 to stabilize RAIRL and check its performance in terms of mean Bregman divergence similar to the previous experiment.

Revised: Note that since both π and π E are multi-variate Gaussian and can be evaluated, we can evaluate the individual Bregman divergence

Original:  Theevaluation performances for RAIRL are depicted in Figure 4.
Revised: D A Ω ( π ( ·| s ) || π E ( ·| s )) on s by using the derivation in Appendix I.3. The performances during RAIRL’s training are described as follows.

Original: We evaluate π with  an episodic score (Figure 4, Left ) and three Bregman divergences  (Figure 4, Right ) Ω( π ) =
Revised: We report π with both an episodic score (Figure 4) and mean Bregman divergences with respect to three types of Tsallis entropies (Figure 5)

Original:  −T 1 q (cid:48)( π ) with q (cid:48)
Revised: Ω = −T 1 q (cid:48) with q (cid:48)

Original: = 1 , 1 . 5 , 2 , respectively.
Revised: = 1 , 1 . 5 , 2 .

Original: Note that the objective of RAIRL with Ω = −T  q is to minimize the corresponding  Bregman divergence with q (cid:48)
Revised: Note that the objective of RAIRL with Ω = −T 1 q is to minimize the corresponding mean Bregman divergence with q (cid:48)

Original: =
Revised: =

Original: q .
Revised: q .

Original: In Figure 4 ( Left ), both RAIRL-DBM and RAIRLNSM are shown to achieve the expert performance, irrespective of q , in Hopper-v2 , Walker-v2 , and HalfCheetah-v2 .
Revised: In Figure 4, both RAIRLDBM and RAIRL-NSM are shown to achieve the expert performance, irrespective of q , in Hopperv2 , Walker-v2 , and HalfCheetah-v2 .

Original: In contrast, RAIRL in Ant-v2 fails to achieve the expert’s performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in our setting.
Revised: In contrast, RAIRL in Ant-v2 fails to achieve the expert’s performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in our setting.

Original: Although the evaluation scores are comparable for all methods in Hopper-v2 , Walker-v2 , and HalfCheetah-v2 , respective divergences are shown to be highly different from one another as shown in Figure( Right ).
Revised: Although the episodic scores are comparable for all methods in Hopper-v2 , Walker-v2 , and HalfCheetah-v2 , respective divergences are shown to be highly different from one another as shown in Figure 5.

Original: RAIRL with q = 2 in most cases achieves the minimum  Bregman divergence (for all three divergences with q (cid:48)
Revised: RAIRL with q = 2 in most cases achieves the minimum mean Bregman divergence (for all three divergences with q (cid:48)

Original: = 1 , 1 . 5 , 2 ), whereas RAIRL with q = 1 – which corresponds to AIRL (Fu et al., 2018) – achieves the maximum divergence in most cases.
Revised: = 1 , 1 . 5 , 2 ), whereas RAIRL with q = 1 —which corresponds to AIRL (Fu et al., 2018)—achieves the maximum divergence in most cases.

Original: This result is in alignment with our intuition from Section 3.2; as q increases, minimizing the Bregman divergence requires much tighter matching between π and π E .
Revised: This result is in alignment with our intuition from Section 3.2; as q increases, minimizing the Bregman divergence requires much tighter matching between π and π E .

Original: Therefore, we believe RAIRL is suitable for making imitation learning safer in practice.

Original: Unfortunately, RAIRL fails to acquire a reward function that effectively minimizes the  target divergence  in continuous controls.
Revised: Unfortunately, while evaluating the acquired reward—RAC with randomly initialized agent and acquired reward—the target divergence is not properly decreased in continuous controls.

Original: We believe this is because π is a probability density function in continuous controls and causes large variance during training, while π is a mass function and is well-bounded in discrete control problems.
Revised: We believe this is because π is a probability density function in continuous controls and causes large variance during training, while π is a mass function and is well-bounded in discrete control problems.

Original:  D ISCUSSION AND F UTURE W ORKS
Revised:  D ISCUSSION AND F UTURE W ORKS

Original: We consider the problem of IRL in regularized MDPs (Geist et al., 2019), assuming ageneral class of  convex policy regularizers.
Revised: We consider the problem of IRL in regularized MDPs (Geist et al., 2019), assuming a class of strongly convex policy regularizers.

Original: We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a speciﬁc instance of imitation learning – i.e., one that minimizes the Bregman divergence associated with policy regularizers.
Revised: We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a speciﬁc instance of imitation learning—i.e., one that minimizes the Bregman divergence associated with policy regularizers.

Original: We propose RAIRL – a practical sampled-based IRL algorithm in regularized MDPs – and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).
Revised: We propose RAIRL—a practical sampled-based IRL algorithm in regularized MDPs—and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).

Original: Finally, recent advances in imitation learning and IRL are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ghasemipour et al., 2019).
Revised: Finally, recent advances in imitation learning and IRL are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ke et al., 2019; Ghasemipour et al., 2019).

Original: Although Bregman divergence is known to cover various divergences, it does not include some divergence families such as f -divergence (Csiszár, 1963; Amari, 2009).
Revised: Although Bregman divergence is known to cover various divergences, it does not include some divergence families such as f -divergence (Csiszár, 1963; Amari, 2009).

Original: Therefore, we believe that considering RL with policy regularization different from Geist et al.
Revised: Therefore, we believe that considering RL with policy regularization different from Geist et al.

Original: and its inverse problem is a possible way of ﬁnding the links between imitation learning and various statistical distances.
Revised: and its inverse problem is a possible way of ﬁnding the links between imitation learning and various statistical distances.

Original: A P ROOF OF L EMMA
Revised: A B ELLMAN O PERATORS , V ALUE F UNCTIONS IN REGULARIZED MDP S

Revised: Let a policy regularizer Ω : ∆ A → R be strongly convex, and deﬁne the convex conjugate of Ω is Ω ∗ : R A → R as [Equation]

Revised: Then, Bellman operators, equations and value functions in regularied MDPs are deﬁned as follows.

Revised: Deﬁnition 1 (Regularized Bellman Operators) .

Revised: For V ∈ R S , let us deﬁne Q ( s, a ) =

Revised: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s

Revised: [Equation]

Revised: a ) [ V ( s (cid:48) )] .

Revised: The regularized Bellman evaluation operator is deﬁned as

Revised: [Equation]

Revised: and T π V =

Revised: V is called the regularized Bellman equation.

Revised: Also, the regularized Bellman optimality operator is deﬁned as

Revised: [Equation]

Revised: and T ∗ V =

Revised: V is called the regularized Bellman optimality equation.

Revised: Deﬁnition 2 (Regularized value functions) .

Revised: The unique ﬁxed point V π of the operator T π is called the state value function, and Q π ( s, a ) =

Revised: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a )

Revised: [ V π ( s (cid:48) )] is called the state-action value function.

Revised: Also, the unique ﬁxed point V ∗ of the operator T ∗ is called the optimal state value function, and Q ∗ ( s, a ) =

Original: Let us deﬁne π s = π ( ·| s ) .
Revised: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ( s (cid:48) )] is called the optimal state-action value function. It should be noted that Proposition 1 in Geist et al. (2019) tells us ∇ Ω ∗ ( Q ( s, · )) is a policy that uniquely maximizes Eq.(15). For example, when Ω( π ( ·| s ))

Original: For r (s, a  )  =
Revised: = (cid:80) a ∼ π ( ·| s ) log π ( a | s ) (negative Shannon entropy), ∇ Ω ∗ ( Q ( s, · )) is a softmax policy, i.e., ∇ Ω ∗ ( Q ( s, · )) =

Original: t ( s,  a ; π E ) , the RL objective Eq.(1) satisﬁes E π
Revised: exp( Q ( s, · )) (cid:80) a (cid:48)∈A exp( Q ( s,a (cid:48) )) .

Original: [Equation] [Equation] i [Equation]
Revised: Due to this property, the optimal policy π ∗ of a regularized MDP is uniquely found for the optimal state-action value function Q ∗ which is also uniquely deﬁned as the ﬁxed point of T ∗ .

Original: (cid:111)(cid:35)
Revised: B P ROOF OF L EMMA

Original: [Equation] (cid:111)(cid:35) [Equation]
Revised: Let us deﬁne π s = π ( ·| s ) .

Original: i [Equation] γ
Revised: For r ( s, a ) =

Original: [Equation] Ω( π [Equation] iE ) 
Revised: t ( s, a ; π E ) , the RL objective Eq.(1) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: π 
Revised: π (cid:34) ∞ (cid:88) i = γ i (cid:110) E a ∼ π si Ω (cid:48) ( s i , a ; π E ) − E a ∼ π siE Ω (cid:48)

Original: [Equation] −[Equation] Ω( 
Revised: ( s i , a ; π E ) + Ω( π s iE ) − Ω( π s i ) (cid:111)(cid:35) ( iv ) =

Original: [Equation] iE ) [Equation] π 
Revised: E π (cid:34) ∞ (cid:88) i

Original: [Equation] iE (cid:105) [Equation] + Ω( π 
Revised: = γ i (cid:8) (cid:104)∇ Ω( π s iE ) , π s i (cid:105)

Original: [Equation]
Revised: A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)

Original:  iE ) − Ω( π [Equation] ) (cid:9)(cid:35) =
Revised: A + Ω( π s iE ) − Ω( π s i ) (cid:9)(cid:35) =

Original: [Equation] (cid:34) [Equation] (cid:88) i 
Revised: E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s iE ) − Ω( π s i )

Original:  = γ i (cid:8) Ω( π s  iE ) −  Ω( π s i ) 
Revised: + (cid:104)∇ Ω( π s iE ) , π s i (cid:105) A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105) A (cid:9)(cid:35) = − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) , π s i (cid:105) A

Original: + (cid:104)∇ Ω( π s iE ) , 
Revised: + (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)

Original:  π s i 
Revised: A (cid:9) (cid:35) = − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) , π s i − π s iE (cid:105) A (cid:9)(cid:35) ( v ) =

Original: [Equation]
Revised: − E π (cid:34) ∞ (cid:88) i = γ i D A Ω ( π s i || π s iE ))

Original:  where (i) follows from the assumption r ( s, a )
Revised: (cid:35) , (16) where (i) follows from the assumption r ( s, a )

Original: =
Revised: =

Original: t ( s, a ; π E ) in Lemma 1 , (ii) follows from the deﬁnition of t ( s, a ; π ) in Eq.(7), (iii) follows since taking the inner expectation ﬁrst does not change the overall expectation, (iv) follows from the deﬁnition of Ω (cid:48) in Lemma 1 and (cid:80) a ∈A π ( a )[ ∇ Ω( π )] ( a ) =
Revised: t ( s, a ; π E ) in Lemma 1 , (ii) follows from the deﬁnition of t ( s, a ; π ) in Eq.(7), (iii) follows since taking the inner expectation ﬁrst does not change the overall expectation, (iv) follows from the deﬁnition of Ω (cid:48) in Lemma 1 and (cid:80) a ∈A p ( a )[ ∇ Ω( p )]( a ) =

Original: (cid:104)∇ Ω( π ) , π (cid:105) A , and (v) follows from the deﬁnition of Bregman divergence, i.e., D A Ω ( π 1 || π 2 )
Revised: (cid:104)∇ Ω( p ) , p (cid:105) A , and (v) follows from the deﬁnition of Bregman divergence, i.e., D A Ω ( p 1 || p 2 )

Original: =
Revised: =

Original: Ω( π 1 ) − Ω( π 2 ) − (cid:104)∇ Ω( π 2 ) , π 1 − π 2 (cid:105) A .
Revised: Ω( p 1 ) − Ω ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105) A .

Original: Due to the non-negativity of D A Ω ,
Revised: Due to the non-negativity of D A Ω ,

Original: Eq.(15) is less than or equal to zero which can be achieved when π = π E .
Revised: Eq.(16) is less than or equal to zero which can be achieved when π = π E .

Original: B P ROOF OF C OROLLARY
Revised: C P ROOF OF C OROLLARY

Original: Let a ∈ { 1 , ..., |A|} and π a = π ( a ) for simplicity.
Revised: Let a ∈ { 1 , ..., |A|} and π a = π ( a ) for simplicity.

Original: For
Revised: For

Original: [Equation]
Revised: [Equation]

Original: with f φ ( x ) =
Revised: with f φ ( x ) =

Original: xφ ( x ) , we have∇ [Equation] π ) [Equation]
Revised: xφ ( x ) , we have

Original: [Equation]

Original: |A| (cid:88) a ∈A f

Original: [Equation]
Revised: [Equation]

Original: π a ) =− λ [Equation] f
Revised: for f (cid:48) φ ( x ) =

Original: [Equation]

Original: φ

Original: [Equation]

Original: π 1 ) , ..., f (cid:48) φ ( π |A| )] T for f (cid:48) φ ( x ) =

Original: ∂∂x ( xφ ( x )) .
Revised: ∂∂x ( xφ ( x )) .

Original: Therefore, for π s = π ( ·| s ) we havet ( s, a ; π ) = [ ∇ Ω( π s )]( a ) − (cid:104)∇ Ω( π s ) , π s (cid:105) A + Ω( π s ) = − λf (cid:48) φ ( π sa ) − (cid:32) (cid:88) a (cid:48) ∈A π sa (cid:48) ( − λf (cid:48) φ ( π sa (cid:48) ))
Revised: Therefore, for π s = π ( ·| s ) we have

Original: (cid:33) + (cid:32) − λ (cid:88) a (cid:48)

Original: ∈A π sa (cid:48) φ ( π sa (cid:48) ) (cid:33) =

Original: − λ (cid:40)

Original: f (cid:48) φ ( π sa ) − (cid:32) (cid:88) a (cid:48) ∈A π sa (cid:48)

Original: f (cid:48) φ ( π sa (cid:48) )

Original: (cid:33) + (cid:32) (cid:88) a (cid:48)

Original: ∈A π sa (cid:48) φ ( π sa (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF O PTIMAL R EWARDS ON C ONTINUOUS C ONTROLS
Revised: D P ROOF OF O PTIMAL R EWARDS ON C ONTINUOUS C ONTROLS

Original: Note that for two continuous distributions P 1 and P 2 having probability density functions p 1 ( x ) and p 2 ( x ) , respectively, the Bregman divergence can be deﬁned as (Guo et al., 2017; Jones & Byrne, 1990) 
Revised: Note that for two continuous distributions P 1 and P 2 having probability density functions p 1 ( x ) and p 2 ( x ) , respectively, the Bregman divergence can be deﬁned as (Guo et al., 2017; Jones & Byrne, 1990) [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ω (cid:48) ( x ) :
Revised: where ω (cid:48) ( x ) :

Original: = ∂∂x ω ( x ) and the divergence is measure point-wisely on x ∈ X .
Revised: = ∂∂x ω ( x ) and the divergence is measure point-wisely on x ∈ X .

Original: Let us assume
Revised: Let us assume

Original: [Equation]
Revised: [Equation]

Original: 17) for a probability density function π on A and deﬁne
Revised: for a probability density function π on A and deﬁne

Original: [Equation]
Revised: [Equation]

Original: 18) for π s = π ( ·| s ) . For r ( s, a ) = t ( s, a ; π E ) , the RL objective in Eq.(1) satisﬁes E π (cid:34) ∞ (cid:88)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: where (i) follows from r ( s, a )
Revised: where (i) follows from r ( s, a )

Original: =
Revised: =

Original: t ( s, a ; π E ) , (ii) follows from Eq.(17) and Eq.(18), and (iii) follows from the deﬁnition of Bregman divergence in Eq.(16).
Revised: t ( s, a ; π E ) , (ii) follows from Eq.(18) and Eq.(19), and (iii) follows from the deﬁnition of Bregman divergence in Eq.(17).

Original: Due to the non-negativity of D ω , Eq.(21) is less than or equal to zero which can be achieved when π = π E .
Revised: Due to the non-negativity of D ω , Eq.(24) is less than or equal to zero which can be achieved when π = π E .

Original: Also, π = π E is a unique solution since Eq.(1) has a unique solution for arbitrary reward functions.
Revised: Also, π = π E is a unique solution since Eq.(1) has a unique solution for arbitrary reward functions.

Original: D P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Revised: Since Lemma 2 was mentioned but not proved in Geist et al.

Revised: (2019), we remain the rigorous proof for Lemma 2 in this subsection.

Revised: Note that we follow the proof idea in Ng et al. (1999).

Original: RL objective in Regularized MDPs w.r.t.
Revised: First, let us assume an MDP M r with a reward r and corresponding optimal policy π ∗ . From Deﬁnition 1 , the optimal state-value function V ∗ ,r and its corresponding state-action value function Q ∗ ,r ( s, a ) :

Revised: = r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ,r ( s (cid:48) )] should satisﬁes the regularized Bellman optimality equation

Revised: [Equation]

Revised: (21) where we explicitize the dependencies on r .

Revised: Also, the optimal policy π ∗ is a unique maximizer for the above maximization.

Revised: Now, let us consider the shaped rewards r ( s, a )+Φ( s (cid:48) ) − Φ( s ) and r ( s, a )+ E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) )

Revised: − Φ( s ) .

Revised: Please note that for both rewards, the expectation over s (cid:48) for given s, a is equal to

Revised: [Equation]

Revised: Φ( s ) , and thus, it is sufﬁcient to only consider the optimality for ˜ r .

Revised: By subtracting Φ( s ) from the regularized optimality Bellman equation for r in Eq.(21), we have

Revised: [Equation]

Revised: That is, V ∗ ,r − Φ is the ﬁxed point of the regularized Bellman optimality operator T ∗ , ˜ r associated with the shaped reward ˜ r .

Revised: Also, a maximizer for the above maximization is π ∗ since subtracting Φ( s ) from Eq.(21) does not change the unique maximizer π ∗ .

Revised: F C OMPARISON BETWEEN OUR SOLUTION AND EXISTING SOLUTION IN G EIST ET AL . (2019)

Revised: F.

Revised: P ROBLEMS OF E XISTING S OLUTIONS IN G EIST ET

Revised: In Proposition 5 of Geist et al.

Revised: (2019), a solution of regularized IRL is given, and we rewrite the relevant theorem in this subsection.

Revised: Let us consider a regularized IRL problem, where π E ∈ ∆ AS is an expert policy.

Revised: Assuming that both the model (dynamics, discount factor and regularizer) and the expert policy are known, Geist et al.

Revised: (2019) proposed a solution of regularized IRL as follows:

Revised: Lemma 4 (A solution of regularized IRL from Geist et al.

Revised: Let Q E ∈ R

Revised: S×A be a function such that π E ( ·| s ) =

Revised: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S .

Revised: Also, deﬁne

Revised: [Equation]

Revised: Then in the Ω -regularized MDP with the reward r E , π E is an optimal policy.

Revised: Proof.

Revised: Although the brief idea of the proof is given in Geist et al.

Revised: (2019), we rewrite the proof in a more rigorous way as follows.

Revised: Let us deﬁne V E ( s )

Revised: = (cid:104) π E ( ·| s ) , Q E ( s, · ) (cid:105)

Revised: A − Ω( π E ( ·| s )) .

Revised: Then, r E ( s, a ) =

Revised: [Equation]

Revised: , a

Revised: [Equation]

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: )] , and thus, Q E ( s, a ) =

Revised: r E

Revised: [Equation]

Revised: s, a

Revised: [Equation]

Revised: + γ

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: ∼ P ( ·| s

Revised: [Equation]

Revised: a ) [ V E ( s (cid:48) )] .

Revised: By using this and regularized Bellman optimality operator, we have

Revised: [Equation]

Revised: where (i) and (ii) follow from Deﬁnition 1 , and (iii) follows since π E is a unique maximizer.

Revised: Thus,

Revised: V E is the ﬁxed point of T ∗ , and π E ( ·| s ) =

Revised: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S , becomes an optimal policy.

Revised: For example, when negative Shannon entropy is used as a regularizer, we can get r E ( s, a )

Revised: = log π E ( a | s ) by setting Q E ( s, a )

Revised: = log π E ( a | s ) .

Revised: However, a solution proposed in Lemma 4 has two crucial issues:

Revised: Issue 1.

Revised: It requires the knowledge on the model dynamics, which is generally intractable.

Revised: Issue 2.

Revised: We need to ﬁgure out Q E that satisﬁes π E ( ·| s ) =

Revised: ∇ Ω ∗ ( Q E ( s, · )) , which comes from the relationship between optimal value function and optimal policy (Geist et al., 2019).

Revised: In the following subsection, we show how our solution in Lemma 1 is related to the solution from

Revised: Geist et al.

Revised: (2019) in Lemma 4 .

Revised: F. F ROM THE SOLUTION OF G EIST ET AL . (2019) TO THE TRACTABLE SOLUTION

Revised: Let us consider the expert policy π E and quantities Q E and r E satisfying the conditions in Lemma 4 .

Revised: From Lemma 2 , a regularized MDP with the shaped reward

Revised: [Equation]

Revised: for Φ ∈ R S has its optimal policy as π E .

Revised: Since Φ can be arbitrarily chosen, let us assume Φ( s )

Revised: =

Revised: Ω ∗ ( Q E ( s, · )) .

Revised: Then, we have

Revised: [Equation]

Revised: Note that the reward in Eq.

Revised: (22) does not require the knowledge on the model dynamics, which addresses Issue 1 in Appendix F.1.

Revised: Also, by using V E ( s ) =

Revised: Ω ∗ ( Q E ( s, · )) in the proof of Lemma 4 , the reward in Eq.

Revised: (22) can be written as

Revised: [Equation]

Revised: which means ˜ r E ( s, a ) is an advantage function for the optimal policy π E in the Ω -regularized MDP.

Revised: However, we still have Issue 2 in Appendix F.1 since Ω ∗ in Eq.

Revised: (22) is generally intractable (Geist et al., 2019), which prevents us to ﬁnd out appropriate Q E ( s, a ) .

Revised: Interestingly, we show that for all s ∈ S and Q E ( s, · ) =

Revised: ∇ Ω( π E ( ·| s )) ,

Revised: [Equation]

Revised: where the last equality holds since the Bregman divergence

Revised: D A Ω ( π ( ·| s )

Revised: ||

Revised: π E ( ·| s )) is greater than or equal to zero and its lowest value is achieved when π ( ·| s )

Revised: =

Revised: π E ( ·| s ) .

Revised: This means that when

Revised: Q E ( s, · ) = ∇ Ω( π E ( ·| s )) is used, the condition π E ( ·| s )

Revised: = ∇ Ω ∗ ( Q E ( s, · )) in Lemma 4 is satisﬁed without knowing the tractable form of Ω ∗ or ∇ Ω ∗ , and thus, Issue 2 in Appendix F.1 is addressed— while we require the knowledge on the gradient ∇ Ω of the policy regularizer Ω , which is practically more tractable.

Revised: Finally, by substituting Q E ( s, · ) =

Revised: Ω (cid:48)

Revised: ( s, · ; π E ) for Ω (cid:48) ( s, · ; π ) :=

Revised: ∇ Ω( π ( ·| s )) , s ∈ S , to Eq.(22), we have

Revised: [Equation]

Revised: where t ( s, a ; π E ) is our proposed solution in Lemma

Revised: [Equation]

Revised: G P ROOF OF L EMMA

Revised: RL objective in Regularized MDPs w.r

Revised: [Equation]

Original:  normalized visitation distributions.
Revised: t. normalized visitation distributions.

Original: For a reward function r ∈ R S×A and a strongly convex function Ω : ∆ AS → R , the RL objetive J Ω ( r, π ) in Eq.(1) is equivalent to
Revised: For a reward function r ∈ R S×A and a strongly convex function Ω : ∆ A → R , the RL objetive J Ω ( r, π ) in Eq.(1) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: 20) where for a set D of normalized visitation distributions (Syed et al., 2008) 
Revised: where for a set D of normalized visitation distributions (Syed et al., 2008) D

Original: [Equation]
Revised: [Equation]

Original: we deﬁne ¯Ω( d ) :
Revised: we deﬁne ¯Ω( d ) :

Original: =
Revised: =

Original: E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] and ¯ π d ( ·| s ) :
Revised: E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] and ¯ π d ( ·| s ) :

Original: = d ( s, · ) (cid:80) a (cid:48)
Revised: = d ( s, · ) (cid:80) a (cid:48)

Original: d ( s,a (cid:48) ) ∈
Revised: d ( s,a (cid:48) ) ∈

Original: ∆ AS for d ∈ D and use ¯ π d π ( ·| s )
Revised: ∆ AS for d ∈ D and use ¯ π d π ( ·| s )

Original: =
Revised: =

Original: π ( ·| s ) for all s ∈ S .
Revised: π ( ·| s ) for all s ∈ S .

Original: For ¯Ω : D → R , its convex conjugate ¯Ω ∗ is
Revised: For ¯Ω : D → R , its convex conjugate ¯Ω ∗ is

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from using the one-to-one correspondence between policies and visitation distributions (Syed et al., 2008; Ho & Ermon, 2016).
Revised: where (i) follows from using the one-to-one correspondence between policies and visitation distributions (Syed et al., 2008; Ho & Ermon, 2016).

Original: Note that Eq.(21) is equal to the optimal discounted average return in regularized MDPs.
Revised: Note that Eq.(24) is equal to the optimal discounted average return in regularized MDPs.

Original: IRL objective in Regularized MDPs w.r.t.
Revised: IRL objective in Regularized MDPs w.r.t.

Original: normalized visitation distributions.
Revised: normalized visitation distributions.

Original: By using the
Revised: By using the

Original: RL objective in Eq.(20), we can rewrite the IRL objective in Eq.(6) w.r.t.
Revised: RL objective in Eq.(23), we can rewrite the IRL objective in Eq.(6) w.r.t.

Original: the normalized visitation distributions as the maximization of the following objective over r ∈ R S×A :
Revised: the normalized visitation distributions as the maximization of the following objective over r ∈ R S×A :

Original: [Equation]
Revised: [Equation]

Original: Note that if ∇ ¯Ω( d ) is well-deﬁned and r = ∇ ¯Ω( d π E ) for any strictly convex ¯Ω , Eq.(22) is equal to
Revised: Note that if ∇ ¯Ω( d ) is well-deﬁned and r = ∇ ¯Ω( d π E ) for any strictly convex ¯Ω , Eq.(25) is equal to

Original: [Equation]
Revised: [Equation]

Original: where the equality comes from the deﬁnition of Bregman divergence.
Revised: where the equality comes from the deﬁnition of Bregman divergence.

Original: Proof of t ( s, a ; π d ) = ∇ [¯Ω( d )]( s, a ) .
Revised: Proof of t ( s, a ; π d ) = ∇ [¯Ω( d )]( s, a ) .

Original: For simpler notation, we use matrix-vector notation for the proof when discrete state and action spaces S
Revised: For simpler notation, we use matrix-vector notation for the proof when discrete state and action spaces S

Original: = { 1 , ..., |S|} and A = { 1 , ..., |A|} are considered.
Revised: = { 1 , ..., |S|} and A = { 1 , ..., |A|} are considered.

Original: For a normalized visitation distribution d ∈ D , let us deﬁne
Revised: For a normalized visitation distribution d ∈ D , let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: where 111 A = [1 , ..., 1] T ∈
Revised: where 111 A = [1 , ..., 1] T ∈

Original: R
Revised: R

Original: A is an |A| -dimensional all-one vector.
Revised: A is an |A| -dimensional all-one vector.

Original: By using these notations
Revised: By using these notations

Original: [Equation]
Revised: [Equation]

Original: the original ¯Ω can be rewritten as
Revised: the original ¯Ω can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: The gradient of ¯Ω w.r.t.
Revised: The gradient of ¯Ω w.r.t.

Original: DDD (using denominator-layout notation) is
Revised: DDD (using denominator-layout notation) is

Original: [Equation]
Revised: [Equation]

Original: where each element of ∇ DDD ¯Ω( DDD ) satisﬁes
Revised: where each element of ∇ DDD ¯Ω( DDD ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: 23) for
Revised: 26) for

Original: [Equation]
Revised: [Equation]

Original: Note that each element of ∂ ¯ π a ( ddd s ) ∂ddd s satisﬁes
Revised: Note that each element of ∂ ¯ π a ( ddd s ) ∂ddd s satisﬁes

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: By substituting Eq.(24) into Eq
Revised: By substituting Eq.(27) into Eq

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: If we use the function notation, Eq.(25) can be written as
Revised: If we use the function notation, Eq.(28) can be written as

Original: [Equation]
Revised: [Equation]

Original: for t of Eq.(7) in Lemma 1 .
Revised: for t of Eq.(7) in Lemma 1 .

Original: E D ERIVATION OF B REGMAN -D IVERGENCE -B ASED M EASURE IN C ONTINUOUS C ONTROLS
Revised: H D ERIVATION OF B REGMAN -D IVERGENCE -B ASED M EASURE IN C ONTINUOUS C ONTROLS

Original: In Eq.(16), the Bregman divergence in the control task is deﬁned as
Revised: In Eq.(17), the Bregman divergence in the control task is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that we consider Ω( p ) =
Revised: Note that we consider Ω( p ) =

Original: (cid:82) X ω ( p ( x )) dx =
Revised: (cid:82) X ω ( p ( x )) dx =

Original: (cid:82) X [ − f φ ( p ( x ))] dx for f φ ( x ) =
Revised: (cid:82) X [ − f φ ( p ( x ))] dx for f φ ( x ) =

Original: xφ ( x )
Revised: xφ ( x )

Original: [Equation]
Revised: [Equation]

Original: which makes Eq.(26) equal to
Revised: which makes Eq.(29) equal to

Original: [Equation]
Revised: [Equation]

Original: Thus, by considering a learning agent’s policy π s = π ( ·| s ) , expert’s policy π s E = π E ( ·| s )
Revised: Thus, by considering a learning agent’s policy π s = π ( ·| s ) , expert policy π s E = π E ( ·| s )

Original: [Equation]
Revised: [Equation]

Original: and the objective in Eq.(8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:[Equation]
Revised: and the objective in Eq.(8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:

Original: [Equation]
Revised: [Equation]

Original: F T SALLIS ENTROPY AND ASSOCIATED B REGMAN DIVERGENCE AMONG MULTI - VARIATE G AUSSIAN DISTRIBUTIONS
Revised: I T SALLIS ENTROPY AND ASSOCIATED B REGMAN DIVERGENCE AMONG MULTI - VARIATE G AUSSIAN DISTRIBUTIONS

Original: Based on the derivation in Nielsen & Nock (2011)
Revised: Based on the derivation in Nielsen & Nock (2011)

Original: [Equation]
Revised: [Equation]

Original: we derive the Tsallis entropy and associated
Revised: we derive the Tsallis entropy and associated

Original: Bremgan divergence as follows.
Revised: Bremgan divergence as follows.

Original: We ﬁrst consider the distributions in the exponential family
Revised: We ﬁrst consider the distributions in the exponential family

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that for
Revised: Note that for

Original: [Equation]
Revised: [Equation]

Original: we can recover the multi-variate Gaussian distribution (Nielsen & Nock, 2011):
Revised: we can recover the multi-variate Gaussian distribution (Nielsen & Nock, 2011):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For two distributions with k ( x ) = 0 ,
Revised: For two distributions with k ( x ) = 0 ,

Original: [Equation]
Revised: [Equation]

Original: that share t, F , and k , it can be shown that
Revised: that share t, F , and k , it can be shown that

Original: [Equation]
Revised: [Equation]

Original: since (cid:90)
Revised: since (cid:90)

Original: [Equation]
Revised: [Equation]

Original: = (cid:90)
Revised: = (cid:90)

Original: [Equation]
Revised: [Equation]

Original: θ, t ( x ) (cid:105) − αF ( θ ) + β (cid:104)
Revised: θ, t ( x ) (cid:105) − αF ( θ ) + β (cid:104)

Original: ˆ θ, t ( x ) (cid:105) − βF (ˆ θ ) (cid:17) dx = (cid:90) exp (cid:16) (cid:104) αθ +
Revised: ˆ θ, t ( x ) (cid:105) − βF (ˆ θ ) (cid:17) dx = (cid:90) exp (cid:16) (cid:104) αθ +

Original: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) exp (cid:16) F ( αθ + β ˆ θ )
Revised: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) exp (cid:16) F ( αθ + β ˆ θ )

Original: − αF ( θ ) − βF
Revised: − αF ( θ ) − βF

Original: (ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) (cid:90) exp (cid:16) (cid:104)
Revised: (ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) (cid:90) exp (cid:16) (cid:104)

Original: αθ +
Revised: αθ +

Original: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) .
Revised: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) .

Original: F. T SALLIS E NTROPY
Revised: I. T SALLIS E NTROPY

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) and k = 1 , the Tsallis entropy of π can be written as 
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) and k = 1 , the Tsallis entropy of π can be written as T [Equation] [Equation]

Revised: [Equation]

Revised: q

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: If π is a multivariate Gaussian distribution, we have
Revised: If π is a multivariate Gaussian distribution, we have

Original: [Equation]
Revised: [Equation]

Original: For Σ = diag { σ 21 , ..., σ 2 d } , we have
Revised: For Σ = diag { σ 21 , ..., σ 2 d } , we have

Original: [Equation]
Revised: [Equation]

Original: F. A NALYTICAL FORM OF BASELINE
Revised: I. T RACTABLE FORM OF BASELINE

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) , we have
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the baseline can be rewritten as
Revised: Therefore, the baseline can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: For a multivariate Gaussian distribution π , the analytical form of E x ∼ π [ − f (cid:48) φ ( x ) + φ ( x )] can be derived by using that of Tsallis entropy T q ( π ) of π .
Revised: For a multivariate Gaussian distribution π , the tractable form of E x ∼ π [ − f (cid:48) φ ( x )+ φ ( x )] can be derived by using that of Tsallis entropy T q ( π ) of π .

Original: F. B REGMAN DIVERGENCE WITH T SALLIS ENTROPY REGULARIZATION
Revised: I. B REGMAN DIVERGENCE WITH T SALLIS ENTROPY REGULARIZATION

Original: In Eq.(27), we consider the following form of the Bregman divergence:
Revised: In Eq.(30) [Equation] we consider the following form of the Bregman divergence:

Original: [Equation]
Revised: [Equation]

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) , f (cid:48) φ ( x ) = kq −
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) , f (cid:48) φ ( x ) = kq −

Original: [Equation]
Revised: [Equation]

Original: 1 − qx q − 1 ) =
Revised: 1 − qx q − 1 ) =

Original: qφ ( x ) − k , and k = 1 , the above form is equal to
Revised: qφ ( x ) − k [Equation] and k = 1 , the above form is equal to

Original: [Equation]
Revised: [Equation]

Original: For multivariate Gaussians
Revised: For multivariate Gaussians

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: G A LGORITHM
Revised: J E XPERIMENT S ETTING J. P OLICY R EGULARIZERS IN E XPERIMENTS

Original: 1: Input: A set D E of expert demonstration generated by expert’s policy π E , a reward approximator r θ , a policy π ψ for neural network parameters θ and ψ 2: for each iteration do 3:

Original: Sample rollout trajectories by using the learners policy π ψ . Optimize θ with the discriminator D r θ ,π ψ and the learning objective in Eq.(11).
Revised: [Table]

Original: Optimize ψ with Regularized Actor Critic by using r θ as a reward function.
Revised: By exploiting the knowledge on the reward in Corollary

Original: 6: end for 7: Output: π ψ ≈ π E , r θ ( s, a ) ≈
Revised: [Equation]

Original: t ( s, a ; π E ) (a solution of the IRL problem in Lemma 1 ). H E XPERIMENT S ETTING H. P OLICY R EGULARIZERS IN E XPERIMENTS
Revised: we consider the density-based model (DBM) which is deﬁned by

Original: [Table]
Revised: [Equation]

Original: H. D ENSITY - BASED MODEL H. E XPERT IN B ERMUDA W ORLD ENVIRONMENT
Revised: for θ = ( θ 1 , θ 2 ) . Here, f (cid:48) φ is a function that can be known priorly, and π θ 1 is a neural network which is deﬁned separately from the policy neural network π ψ . The DBM for discrete control problems are depicted in Figure 6 ( Left ). The model outputs rewards over all actions in parallel, where softmax is used for π θ 1 ( ·| s ) and − f (cid:48) φ is elementwisely applied to those softmax outputs followed by elementwisely adding B θ 2 ( s ) .

Revised: For continuous control (Figure 6, Right ), we use the network architecture similar to that in discrete control, where multivariate Gaussian distribution is used instead of softmax layer.

Revised: J. E XPERT IN B ERMUDA W ORLD ENVIRONMENT

Original: We assume a stochastic expert deﬁned by
Revised: We assume a stochastic expert deﬁned by

Original: [Equation]
Revised: [Equation]

Original: ( d ( t ) ) − 1 I { a = Proj( θ ( t ) ) } (cid:80) 3 t =1 ( d ( t ) ) −, θ ( t ) = arctan2(¯ y ( t ) − y, ¯ x ( t ) − x ) , d ( t ) = (cid:107) ¯ s ( t ) − s [-(cid:107) 
-]
Revised: for s [-(cid:107) 
-]=

Original: + (cid:15), t = 1 , 2 , 3 , for s = ( x, y ) , ¯ s (1) =
Revised: ( x, y ) , ¯ s (1) =

Original: (¯ x (1) , ¯ y (1) ) =
Revised: (¯ x (1) , ¯ y (1) ) =

Original: ( − 5 , 10) , ¯ s (2) =
Revised: ( − 5 , 10) , ¯ s (2) =

Original: (¯ x (2) , ¯ y (2) ) =
Revised: (¯ x (2) , ¯ y (2) ) =

Original: (0 , 10) , ¯ s (3) =
Revised: (0 , 10) , ¯ s (3) =

Original: (¯ x (3) , ¯ y (3) ) =
Revised: (¯ x (3) , ¯ y (3) ) =

Original: (5 , 10) , (cid:15) =
Revised: (5 , 10) , (cid:15) =

Original: 10 − 4 and an operator Proj( θ ) : R →
Revised: 10 − 4 and an operator Proj( θ ) : R →

Original: A that maps θ to the closest angle in A .
Revised: A that maps θ to the closest angle in A .

Original: H. M U J O C O EXPERIMENT SETTING
Revised: In Figure 7, we depicted the expert policy.

Original:  Instead of directly using MuJoCo environments with tanh -squashed policies proposed in SoftActor Critic (Haarnoja et al., 2018), we move tanh to a part of environment – named hyperbolized environments in short – and assume Gaussian policies.
Revised: [Figure] Instead of directly using MuJoCo environments with tanh -squashed policies proposed in Soft ActorCritic (SAC) (Haarnoja et al., 2018), we move tanh to a part of environment—named hyperbolized environments in short—and assume Gaussian policies.

Original: Speciﬁcally, after an action a is sampled from the policies, we pass tanh( a ) to the environment.
Revised: Speciﬁcally, after an action a is sampled from the policies, we pass tanh( a ) to the environment.

Original: We then consider multi-variate Gaussian policy
Revised: We then consider multi-variate Gaussian policy

Original: [Equation]
Revised: [Equation]

Original: with µµµ ( s ) =
Revised: with µµµ ( s ) =

Original: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } , where − arctanh(0 . 99) ≤ µ i ( s ) ≤ arctanh(0 . 99) , log(0 . 01)
Revised: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } , where − arctanh(0 . 99) ≤ µ i ( s ) ≤ arctanh(0 . 99) , log(0 . 01)

Original: ≤ log σ i ( s ) ≤ log(2) for all i
Revised: ≤ log σ i ( s ) ≤ log(2) for all i

Original: = 1 , ..., d .
Revised: = 1 , ..., d .

Original: Instead of using clipping, we use tanh -activated outputs and scale them to be ﬁt in the above ranges, which empirically improves the performance.
Revised: Instead of using clipping, we use tanh -activated outputs and scale them to be ﬁt in the above ranges, which empirically improves the performance.

Original: Also, instead of using potential-based reward shaping used in AIRL (Fu et al., 2018), we update moving mean of intermediate reward values and update  value networks with mean-subtracted rewards to stabilize RL algorithm.
Revised: Also, instead of using potential-based reward shaping used in AIRL (Fu et al., 2018), we update moving mean of intermediate reward values and update the value network with mean-subtracted rewards—so that the value network gets approximately mean-zero reward—to stabilize RL part of RAIRL.

Original: Note that this is motivated by Lemma 2 from which we can guarantee that any constant shift of reward functions does not change optimality.
Revised: Note that this is motivated by Lemma 2 from which we can guarantee that any constant shift of reward functions does not change optimality.

Original: H. H YPERPARAMETERS
Revised: J. H YPERPARAMETERS

Original: Table 2, Table 3 and Table 4 list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.[Table] [Table] [Equation]
Revised: Table 2, Table 3 and Table 4 list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 0 .q

Original: [Table]
Revised: [Table]

Original: T kq ) 2 .k

Original: [Table]

Original: T kq ) 1 .

Original: [Table]

Original: [Table]

Original: 0 .(F

Original: [Table]

Original: ) λ

Original: [Table]

Original: ) L

Original: [Table]

Original: [Table]

Original: [Table]

Original: ) T

Original: [Table]

Original: ( τ )

Original: [Table]

Original: [Table]

Original: W

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: τ )

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: T kq ) 1 .1 .1 .1 .

Original: [Table][Table] [Table] [Table]
Revised: [Table]