Original: R EGULARIZED I NVERSE R EINFORCEMENT L EARNING
Revised: R EGULARIZED I NVERSE R EINFORCEMENT L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions.
Revised: Inverse Reinforcement Learning (IRL) aims to facilitate a learner’s ability to imitate expert behavior by acquiring reward functions that explain the expert’s decisions.

Original: Regularized IRL applies strongly convex regularizers to the learner’s policy in order to avoid the expert’s behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions.
Revised: Regularized IRL applies strongly convex regularizers to the learner’s policy in order to avoid the expert’s behavior being rationalized by arbitrary constant rewards, also known as degenerate solutions.

Original: We propose tractable solutions, and practical methods to obtain them, for regularized IRL.
Revised: We propose tractable solutions, and practical methods to obtain them, for regularized IRL.

Original: Current methods are restricted to the maximum-entropy IRL framework, limiting them to Shannon-entropy regularizers as well as proposing solutions that are intractable in practice.
Revised: Current methods are restricted to the maximum-entropy IRL framework, limiting them to Shannon-entropy regularizers as well as proposing solutions that are intractable in practice.

Original: We present theoretical backing for our proposed IRL method’s applicability to both discrete and continuous controls empirically validating our performance on a variety of tasks.
Revised: We present theoretical backing for our proposed IRL method’s applicability to both discrete and continuous controls empirically validating our performance on a variety of tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has been successfully applied to many challenging domains including games (Mnih et al., 2015; 2016) and robot control (Schulman et al., 2015; Fujimoto et al., 2018;
Revised: Reinforcement learning (RL) has been successfully applied to many challenging domains including games (Mnih et al., 2015; 2016) and robot control (Schulman et al., 2015; Fujimoto et al., 2018;

Original: Haarnoja et al., 2018).
Revised: Haarnoja et al., 2018).

Original: Advanced RL methods often employ policy regularization motivated by, e.g., boosting exploration (Haarnoja et al., 2018) or safe policy improvement (Schulman et al., 2015).
Revised: Advanced RL methods often employ policy regularization motivated by, e.g., boosting exploration (Haarnoja et al., 2018) or safe policy improvement (Schulman et al., 2015).

Original: While Shannon entropy is often used as a policy regularizer (Ziebart et al., 2008), Geist et al.
Revised: While Shannon entropy is often used as a policy regularizer (Ziebart et al., 2008), Geist et al.

Original: (2019) recently proposed a theoretical foundation of regularized Markov decision processes (MDPs)—a framework that uses strongly convex functions as policy regularizers.
Revised: (2019) recently proposed a theoretical foundation of regularized Markov decision processes (MDPs)—a framework that uses strongly convex functions as policy regularizers.

Original: Here, one crucial advantage is that an optimal policy is shown to uniquely exist, whereas multiple optimal policies may exist in the absence of policy regularization.
Revised: Here, one crucial advantage is that an optimal policy is shown to uniquely exist, whereas multiple optimal policies may exist in the absence of policy regularization.

Original: Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering), Inverse Reinforcement Learning (IRL) (Russell, 1998; Ng et al., 2000)—the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation (Finn et al., 2016b), autonomous driving (Sharifzadeh et al., 2016; Wu et al., 2020) and clinical motion analysis (Li et al., 2018).
Revised: Meanwhile, since RL requires a given or known reward function (which can often involve non-trivial reward engineering), Inverse Reinforcement Learning (IRL) (Russell, 1998; Ng et al., 2000)—the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation (Finn et al., 2016b), autonomous driving (Sharifzadeh et al., 2016; Wu et al., 2020) and clinical motion analysis (Li et al., 2018).

Original: In these scenarios, deﬁning a reward function beforehand is particularly challenging and IRL is simply more pragmatic.
Revised: In these scenarios, deﬁning a reward function beforehand is particularly challenging and IRL is simply more pragmatic.

Original: However, complications with IRL in unregularized MDPs relate to the issue of degeneracy, where any constant function can rationalize the expert’s behavior (Ng et al., 2000).
Revised: However, complications with IRL in unregularized MDPs relate to the issue of degeneracy, where any constant function can rationalize the expert’s behavior (Ng et al., 2000).

Original: Fortunately, Geist et al.
Revised: Fortunately, Geist et al.

Original: (2019) show that IRL in regularized MDPs— regularized IRL —does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs.
Revised: (2019) show that IRL in regularized MDPs— regularized IRL —does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs.

Original: Despite this, no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010; Ho & Ermon, 2016; Finn et al., 2016a;
Revised: Despite this, no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010; Ho & Ermon, 2016; Finn et al., 2016a;

Original: Fu et al.
Revised: Fu et al.

Original: , 2018)—have been proposed.
Revised: , 2018)—have been proposed.

Original: In Geist et al.
Revised: In Geist et al.

Original: (2019), solutions for regularized IRL were introduced.
Revised: (2019), solutions for regularized IRL were introduced.

Original: However, they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics.
Revised: However, they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics.

Original: Furthermore, practical algorithms for solving regularized
Revised: Furthermore, practical algorithms for solving regularized

Original: IRL problems have not yet been proposed.
Revised: IRL problems have not yet been proposed.

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original: Unlike the solutions in Geist et al.
Revised: Unlike the solutions in Geist et al.

Original: (2019), we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and its gradient in discrete control problems (Section 3.1).
Revised: (2019), we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and its gradient in discrete control problems (Section 3.1).

Original: We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems (Section 3.2).
Revised: We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems (Section 3.2).

Original: We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized
Revised: We devise Regularized Adversarial Inverse Reinforcement Learning (RAIRL), a practical sample-based method for policy imitation and reward learning in regularized

Original: MDPs, which generalizes adversarial IRL (AIRL, Fu et al.
Revised: MDPs, which generalizes adversarial IRL (AIRL, Fu et al.

Original: (Section 4).
Revised: (Section 4).

Original: Finally, we empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating RAIRL via episodic scores and from divergence minimization perspective (Ke et al., 2019; Ghasemipour et al., 2019; Dadashi et al., 2020)
Revised: Finally, we empirically validate our RAIRL method on both discrete and continuous control tasks, evaluating RAIRL via episodic scores and from divergence minimization perspective (Ke et al., 2019; Ghasemipour et al., 2019; Dadashi et al., 2020)

Original: (Section 5).
Revised: (Section 5).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notation For ﬁnite sets X and Y , Y X is a set of functions from X to Y .
Revised: Notation For ﬁnite sets X and Y , Y X is a set of functions from X to Y .

Original: ∆ X ( ∆ XY ) is a set of (conditional) probabilities over X (conditioned on Y ).
Revised: ∆ X ( ∆ XY ) is a set of (conditional) probabilities over X (conditioned on Y ).

Original: Especially for the conditional probabilities p X | Y ∈ ∆ XY , we say p X | Y ( ·| y ) ∈
Revised: Especially for the conditional probabilities p X | Y ∈ ∆ XY , we say p X | Y ( ·| y ) ∈

Original: ∆ X for y ∈ Y .
Revised: ∆ X for y ∈ Y .

Original: R is the set of real numbers.
Revised: R is the set of real numbers.

Original: For functions f 1 , f 2 ∈ R X , we deﬁne (cid:104) f 1 , f 2 (cid:105) X :=
Revised: For functions f 1 , f 2 ∈ R X , we deﬁne (cid:104) f 1 , f 2 (cid:105) X :=

Original: (cid:80) x ∈ X
Revised: (cid:80) x ∈ X

Original: f 1 ( x ) f 2 ( x ) .
Revised: f 1 ( x ) f 2 ( x ) .

Original: Regularized Markov Decision Processes and Reinforcement Learning
Revised: Regularized Markov Decision Processes and Reinforcement Learning

Original: We consider sequential decision making problems where an agent sequentially chooses its action after observing the state of the environment, and the environment in turn emits a reward with state transition.
Revised: We consider sequential decision making problems where an agent sequentially chooses its action after observing the state of the environment, and the environment in turn emits a reward with state transition.

Original: Such an interaction between the agent and the environment is modeled as an inﬁnite-horizon Markov Decision Process (MDP), M r :=
Revised: Such an interaction between the agent and the environment is modeled as an inﬁnite-horizon Markov Decision Process (MDP), M r :=

Original: (cid:104)S , A , P 0 , P, r, γ (cid:105) and the agent’s policy π ∈ ∆ AS .
Revised: (cid:104)S , A , P 0 , P, r, γ (cid:105) and the agent’s policy π ∈ ∆ AS .

Original: The terms within the MDP are deﬁned as follows: S is a ﬁnite state space,
Revised: The terms within the MDP are deﬁned as follows: S is a ﬁnite state space,

Original: A is a ﬁnite action space,
Revised: A is a ﬁnite action space,

Original: P 0 ∈
Revised: P 0 ∈

Original: ∆ S is an initial state distribution, P ∈ ∆ SS×A is a state transition probability, r ∈ R S×A is a reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: ∆ S is an initial state distribution, P ∈ ∆ SS×A is a state transition probability, r ∈ R S×A is a reward function, and γ ∈ [0 , 1) is the discount factor.

Original: We also deﬁne an MDP without reward as M − :=
Revised: We also deﬁne an MDP without reward as M − :=

Original: (cid:104)S , A , P 0 , P, γ (cid:105) .
Revised: (cid:104)S , A , P 0 , P, γ (cid:105) .

Original: The normalized state-action visitation distribution, d π ∈ ∆ S×A , associated with π is deﬁned as the expected discounted state-action visitation of π , i.e., d π ( s, a ) :
Revised: The normalized state-action visitation distribution, d π ∈ ∆ S×A , associated with π is deﬁned as the expected discounted state-action visitation of π , i.e., d π ( s, a ) :

Original: = (1 − γ ) · E π [ (cid:80) ∞ i =

Revised: = (1 − γ ) · E π [ (cid:80) ∞ i =

Original: γ i
Revised: γ i

Original: I { s i = s, a i = a } ] , where the subscript π on E means that a trajectory ( s 0 , a 0 , s 1 , a 1 , ... ) is randomly generated from M − and π , and I {·} is an indicator function.
Revised: I { s i = s, a i = a } ] , where the subscript π on E means that a trajectory ( s 0 , a 0 , s 1 , a 1 , ... ) is randomly generated from M − and π , and I {·} is an indicator function.

Original: Note that d π satisﬁes the transposed Bellman recurrence (Boularias & Chaib-Draa, 2010; Zhang et al., 2019):
Revised: Note that d π satisﬁes the transposed Bellman recurrence (Boularias & Chaib-Draa, 2010; Zhang et al., 2019):

Original: [Equation]
Revised: [Equation]

Original: We consider RL in regularized MDPs (Geist et al., 2019), where the policy is optimized with a causal policy regularizer.
Revised: We consider RL in regularized MDPs (Geist et al., 2019), where the policy is optimized with a causal policy regularizer.

Original: Mathematically for an MDP M r and a strongly convex function Ω : ∆ A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω :
Revised: Mathematically for an MDP M r and a strongly convex function Ω : ∆ A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards, or return in short, with policy regularizer Ω :

Original: [Equation]
Revised: [Equation]

Original: It turns out that the optimal solution of Eq.(1) is unique (Geist et al., 2019), whereas multiple optimal policies may exist in unregularized MDPs (See Appendix A for a detailed explanation).
Revised: It turns out that the optimal solution of Eq.(1) is unique (Geist et al., 2019), whereas multiple optimal policies may exist in unregularized MDPs (See Appendix A for a detailed explanation).

Original: In later work (Yang et al., 2019), Ω( p ) =
Revised: In later work (Yang et al., 2019), Ω( p ) =

Original: − λ E a ∼ p φ ( p ( a )) ,
Revised: − λ E a ∼ p φ ( p ( a )) ,

Original: p ∈ ∆ A was considered for λ >and φ : (0 , 1] → R satisfying some mild conditions.
Revised: p ∈ ∆ A was considered for λ >and φ : (0 , 1] → R satisfying some mild conditions.

Original: For example, RL with Shannon entropy regularization (Haarnoja et al., 2018) can be recovered by φ ( x )
Revised: For example, RL with Shannon entropy regularization (Haarnoja et al., 2018) can be recovered by φ ( x )

Original: = − log x , while RL with Tsallis entropy regularization (Lee et al., 2020) can be recovered from φ ( x ) =
Revised: = − log x , while RL with Tsallis entropy regularization (Lee et al., 2020) can be recovered from φ ( x ) =

Original: kq − 1 (1 − x q − 1 ) for k > 0 , q > 1 .
Revised: kq − 1 (1 − x q − 1 ) for k > 0 , q > 1 .

Original: The optimal policy π ∗ for Eq.(1) with Ω from Yang et al.
Revised: The optimal policy π ∗ for Eq.(1) with Ω from Yang et al.

Original: (2019) is shown to be
Revised: (2019) is shown to be

Original: [Equation]
Revised: [Equation]

Original: where φ (cid:48)
Revised: where φ (cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: ∂∂x φ ( x ) , g φ is an inverse function of f (cid:48) φ for f φ ( x ) :
Revised: ∂∂x φ ( x ) , g φ is an inverse function of f (cid:48) φ for f φ ( x ) :

Original: =
Revised: =

Original: xφ ( x ) , x ∈ (0 , 1] , and µ ∗ is a normalization term such that (cid:80) a ∈A π ∗ ( a | s ) = 1 .
Revised: xφ ( x ) , x ∈ (0 , 1] , and µ ∗ is a normalization term such that (cid:80) a ∈A π ∗ ( a | s ) = 1 .

Original: Note that we still need to ﬁnd out µ ∗ to acquire a closed-form relation between optimal policy π ∗ and value function Q ∗ .
Revised: Note that we still need to ﬁnd out µ ∗ to acquire a closed-form relation between optimal policy π ∗ and value function Q ∗ .

Original: However, such relations have not been discovered except for Shannon-entropy regularization (Haarnoja et al., 2018) and speciﬁc instances ( q = 1 , 2 , ∞ ) of Tsallis-entropy regularization (Lee et al., 2019) to the best of our knowledge.
Revised: However, such relations have not been discovered except for Shannon-entropy regularization (Haarnoja et al., 2018) and speciﬁc instances ( q = 1 , 2 , ∞ ) of Tsallis-entropy regularization (Lee et al., 2019) to the best of our knowledge.

Original: Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E ,
Revised: Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E ,

Original: IRL (Russell, 1998; Ng et al., 2000) is the problem of seeking a reward function from which we can recover π E through RL.
Revised: IRL (Russell, 1998; Ng et al., 2000) is the problem of seeking a reward function from which we can recover π E through RL.

Original: However, IRL in unregularized MDPs has been shown to be an ill-deﬁned problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng et al., 2000).
Revised: However, IRL in unregularized MDPs has been shown to be an ill-deﬁned problem since (1) any constant reward function can rationalize every expert and (2) multiple rewards meet the criteria of being a solution (Ng et al., 2000).

Original: Maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010) is capable of solving the ﬁrst issue by seeking a reward function that maximizes the expert’s return along with Shannon entropy of expert policy.
Revised: Maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2008; Ziebart, 2010) is capable of solving the ﬁrst issue by seeking a reward function that maximizes the expert’s return along with Shannon entropy of expert policy.

Original: Mathematically, for the
Revised: Mathematically, for the

Original: RL objective J Ω in Eq.(1) and Ω = −H for negative Shannon entropy H ( p ) =
Revised: RL objective J Ω in Eq.(1) and Ω = −H for negative Shannon entropy H ( p ) =

Original: E a ∼ p [ − log p ( a )] (Ho & Ermon, 2016), the objective of MaxEntIRL is
Revised: E a ∼ p [ − log p ( a )] (Ho & Ermon, 2016), the objective of MaxEntIRL is

Original: [Equation]
Revised: [Equation]

Original: Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) which involves generative adversarial training (Goodfellow et al., 2014; Ho & Ermon, 2016) to acquire a solution of MaxEntIRL.
Revised: Another commonly used IRL method is Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) which involves generative adversarial training (Goodfellow et al., 2014; Ho & Ermon, 2016) to acquire a solution of MaxEntIRL.

Original: AIRL considers the structured discriminator (Finn et al., 2016a)
Revised: AIRL considers the structured discriminator (Finn et al., 2016a)

Original: D ( s, a ) =
Revised: D ( s, a ) =

Original: σ ( r ( s, a ) − log π ( a | s ))
Revised: σ ( r ( s, a ) − log π ( a | s ))

Original: =
Revised: =

Original: e r ( s,a ) e r ( s,a ) + π ( a | s ) for σ ( x ) :
Revised: e r ( s,a ) e r ( s,a ) + π ( a | s ) for σ ( x ) :

Original: = 1 / (1 + e − x ) and iteratively optimizes the following objective:
Revised: = 1 / (1 + e − x ) and iteratively optimizes the following objective:

Original: [Equation]
Revised: [Equation]

Original: It turns out that AIRL minimizes the divergence between visitation distributions d π and d π E by solving min π ∈ ∆ AS KL( d π || d π E ) for Kullback-Leibler (KL) divergence KL( ·||· ) (Ghasemipour et al., 2019) .
Revised: It turns out that AIRL minimizes the divergence between visitation distributions d π and d π E by solving min π ∈ ∆ AS KL( d π || d π E ) for Kullback-Leibler (KL) divergence KL( ·||· ) (Ghasemipour et al., 2019) .

Original:  I NVERSE R EINFORCEMENT L EARNING IN R EGULARIZED MDP S
Revised:  I NVERSE R EINFORCEMENT L EARNING IN R EGULARIZED MDP S

Original: In this section, we propose the solution of IRL in regularized MDPs— regularized IRL —and relevant properties in Section 3.1.
Revised: In this section, we propose the solution of IRL in regularized MDPs— regularized IRL —and relevant properties in Section 3.1.

Original: We then discuss a speciﬁc instance of our proposed solution in Section 3. where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.
Revised: We then discuss a speciﬁc instance of our proposed solution in Section 3. where Tsallis entropy regularizers and multi-variate Gaussian policies are used in continuous action spaces.

Original: 3. S OLUTIONS OF REGULARIZED IRL
Revised: 3. S OLUTIONS OF REGULARIZED IRL

Original: We consider regularized IRL that generalizes MaxEntIRL in Eq.(4) to IRL with a class of strongly convex policy regularizers:
Revised: We consider regularized IRL that generalizes MaxEntIRL in Eq.(4) to IRL with a class of strongly convex policy regularizers:

Original: [Equation]
Revised: [Equation]

Original: For any strongly convex policy regularizer Ω , regularized IRL does not suffer from degenerate solutions since there is a unique optimal policy in any regularized MDP (Geist et al., 2019).
Revised: For any strongly convex policy regularizer Ω , regularized IRL does not suffer from degenerate solutions since there is a unique optimal policy in any regularized MDP (Geist et al., 2019).

Original: While
Revised: While

Original: Geist et al.
Revised: Geist et al.

Original: (2019) proposed solutions of regularized IRL, those are intractable solutions (See
Revised: (2019) proposed solutions of regularized IRL, those are intractable solutions (See

Original: Appendix F.1 for a detailed explanation).
Revised: Appendix F.1 for a detailed explanation).

Original: In the following lemma, we propose tractable solutions that only requires the evaluation of the policy regularizer ( Ω ) and its gradient ( ∇ Ω ) which are more manageable in practice.
Revised: In the following lemma, we propose tractable solutions that only requires the evaluation of the policy regularizer ( Ω ) and its gradient ( ∇ Ω ) which are more manageable in practice.

Original: Our solution is motivated from ﬁguring out a reward function that is capable of converting regularized RL into an equivalent divergence minimization problem associated with π and π E :
Revised: Our solution is motivated from ﬁguring out a reward function that is capable of converting regularized RL into an equivalent divergence minimization problem associated with π and π E :

Original: Lemma 1.
Revised: Lemma 1.

Original: For a policy regularizer Ω : ∆ A → R , let us deﬁne
Revised: For a policy regularizer Ω : ∆ A → R , let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: for Ω (cid:48) ( s, · ; π ) := ∇ Ω( π ( ·| s )) :
Revised: for Ω (cid:48) ( s, · ; π ) := ∇ Ω( π ( ·| s )) :

Original: = [ ∇ p Ω( p )] p = π ( ·| s ) ∈ R A , s ∈ S .
Revised: = [ ∇ p Ω( p )] p = π ( ·| s ) ∈ R A , s ∈ S .

Original: Then, t ( s, a ; π E ) for expert policy π E is a solution of regularized IRL with Ω .
Revised: Then, t ( s, a ; π E ) for expert policy π E is a solution of regularized IRL with Ω .

Original: Proof.
Revised: Proof.

Original: (Abbreviated.
Revised: (Abbreviated.

Original: See Appendix B for the full version.)
Revised: See Appendix B for the full version.)

Original: With r ( s, a ) =
Revised: With r ( s, a ) =

Original: t ( s, a ; π E ) , the RL objective in Eq.(1) becomes equivalent to the problem of minimizing the discounted sum of Bregman divergences between π and π E
Revised: t ( s, a ; π E ) , the RL objective in Eq.(1) becomes equivalent to the problem of minimizing the discounted sum of Bregman divergences between π and π E

Original: [Equation]
Revised: [Equation]

Original: where D A Ω is the Bregman divergence (Bregman, 1967) deﬁned by D A Ω ( p 1 || p 2 )
Revised: where D A Ω is the Bregman divergence (Bregman, 1967) deﬁned by D A Ω ( p 1 || p 2 )

Original: =
Revised: =

Original: Ω( p 1 ) − Ω
Revised: Ω( p 1 ) − Ω

Original: ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105)
Revised: ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105)

Original: A for p 1 , p 2 ∈ ∆ A .
Revised: A for p 1 , p 2 ∈ ∆ A .

Original: Due to the non-negativity of the Bregman divergence, π = π E is a solution of Eq.(8) and is unique since Eq.(1) has the unique solution for arbitrary reward functions (Geist et al., 2019).
Revised: Due to the non-negativity of the Bregman divergence, π = π E is a solution of Eq.(8) and is unique since Eq.(1) has the unique solution for arbitrary reward functions (Geist et al., 2019).

Original: In particular, for any policy regularizer Ω represented by an expectation over the policy (Yang et al., 2019), Lemma 1 can be reduced to the following solution in Corollary 1 :
Revised: In particular, for any policy regularizer Ω represented by an expectation over the policy (Yang et al., 2019), Lemma 1 can be reduced to the following solution in Corollary 1 :

Original: Corollary 1.
Revised: Corollary 1.

Original: For Ω( p ) =
Revised: For Ω( p ) =

Original: − λ E a ∼ p φ ( p ( a )) with p ∈ ∆
Revised: − λ E a ∼ p φ ( p ( a )) with p ∈ ∆

Original: A (Yang et al., 2019) , Eq.(7) becomes
Revised: A (Yang et al., 2019) , Eq.(7) becomes

Original: [Equation]
Revised: [Equation]

Original: for f (cid:48) φ ( x ) =
Revised: for f (cid:48) φ ( x ) =

Original: ∂∂x ( xφ ( x )) .
Revised: ∂∂x ( xφ ( x )) .

Original: The proof is in Appendix B.
Revised: The proof is in Appendix B.

Original: Throughout the paper, we denote reward baseline by the expectation E a ∼ π ( ·| s )
Revised: Throughout the paper, we denote reward baseline by the expectation E a ∼ π ( ·| s )

Original: [ f (cid:48) φ ( π ( a | s )) − φ
Revised: [ f (cid:48) φ ( π ( a | s )) − φ

Original: π ( a | s ))] .
Revised: π ( a | s ))] .

Original: Note that for continuous control tasks with Ω( p ) =
Revised: Note that for continuous control tasks with Ω( p ) =

Original: − λ E a ∼ p φ ( p ( a )) , we can obtain the same form of the reward in Eq.(9)
Revised: − λ E a ∼ p φ ( p ( a )) , we can obtain the same form of the reward in Eq.(9)

Original: (The proof is in Appendix D).
Revised: (The proof is in Appendix D).

Original: Although the reward baseline is generally not intractable in continuous control tasks, we derive a tractable reward baseline for a special case (See Section 3.2).
Revised: Although the reward baseline is generally not intractable in continuous control tasks, we derive a tractable reward baseline for a special case (See Section 3.2).

Original: Additionally, when λ = 1 and φ ( x ) = − log x , it can be shown that t ( s, a ; π ) =
Revised: Additionally, when λ = 1 and φ ( x ) = − log x , it can be shown that t ( s, a ; π ) =

Original: log π ( a | s ) —for both discrete and continuous control problems—which was used as a reward objective in previous work (Fu et al., 2018), and that the Bregman divergence in Eq.(8) becomes the KL divergence KL( π ( ·| s ) ||
Revised: log π ( a | s ) —for both discrete and continuous control problems—which was used as a reward objective in previous work (Fu et al., 2018), and that the Bregman divergence in Eq.(8) becomes the KL divergence KL( π ( ·| s ) ||

Original: π E ( ·| s )) .
Revised: π E ( ·| s )) .

Original: Additional solutions of regularized IRL can be found by shaping t ( s, a ; π E ) as stated in the following lemma:
Revised: Additional solutions of regularized IRL can be found by shaping t ( s, a ; π E ) as stated in the following lemma:

Original: Lemma 2 (Potential-based reward shaping) .
Revised: Lemma 2 (Potential-based reward shaping) .

Original: Let π ∗ be the solution of Eq.(1) in a regularized MDP
Revised: Let π ∗ be the solution of Eq.(1) in a regularized MDP

Original: M r with a regularizer Ω : ∆ A → R and a reward function r ∈ R S×A .
Revised: M r with a regularizer Ω : ∆ A → R and a reward function r ∈ R S×A .

Original: Then for Φ ∈ R S , using either r ( s, a ) + γ Φ( s (cid:48) ) − Φ( s ) or r ( s, a ) + E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) )
Revised: Then for Φ ∈ R S , using either r ( s, a ) + γ Φ( s (cid:48) ) − Φ( s ) or r ( s, a ) + E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) )

Original: − Φ( s ) as a reward does not change the solution of Eq.(1).
Revised: − Φ( s ) as a reward does not change the solution of Eq.(1).

Original: The proof is in Appendix E.
Revised: The proof is in Appendix E.

Original: From Lemma 1 and Lemma 2 , we prove the sufﬁcient condition of rewards being solutions of the
Revised: From Lemma 1 and Lemma 2 , we prove the sufﬁcient condition of rewards being solutions of the

Original: IRL problem.
Revised: IRL problem.

Original: However, the necessary condition—a set of those solutions are the only possible solutions for an arbitrary MDP—is not proved (Ng et al., 1999).
Revised: However, the necessary condition—a set of those solutions are the only possible solutions for an arbitrary MDP—is not proved (Ng et al., 1999).

Original: In the following lemma, we check how the proposed solution is related to the normalized statevisitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho & Ermon, 2016;
Revised: In the following lemma, we check how the proposed solution is related to the normalized statevisitation distribution which can be discussed in the line of distribution matching perspective on imitation learning problems (Ho & Ermon, 2016;

Original: Fu et al., 2018; Ke et al., 2019; Ghasemipour et al., 2019):
Revised: Fu et al., 2018; Ke et al., 2019; Ghasemipour et al., 2019):

Original: Lemma 3.
Revised: Lemma 3.

Original: Given the policy regularizer Ω , let us deﬁne ¯Ω( d ) :
Revised: Given the policy regularizer Ω , let us deﬁne ¯Ω( d ) :

Original: = E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] for an arbitrary normalized state-action visitation distribution d ∈ ∆ S×A and the policy ¯ π d ( a | s ) :
Revised: = E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] for an arbitrary normalized state-action visitation distribution d ∈ ∆ S×A and the policy ¯ π d ( a | s ) :

Original: = d ( s,a ) (cid:80) a
Revised: = d ( s,a ) (cid:80) a

Original: [Equation]
Revised: [Equation]

Original: d ( s,a (cid:48) ) induced by d .
Revised: d ( s,a (cid:48) ) induced by d .

Original: Then, Eq.(7) is equal to
Revised: Then, Eq.(7) is equal to

Original: [Equation]
Revised: [Equation]

Original: When ¯Ω( d ) is strictly convex and a solution t ( s, a ; π E ) = ∇ ¯Ω( d π E ) of IRL in Eq.(10) is used, the RL objective in Eq.
Revised: When ¯Ω( d ) is strictly convex and a solution t ( s, a ; π E ) = ∇ ¯Ω( d π E ) of IRL in Eq.(10) is used, the RL objective in Eq.

Original: (1) is equal to
Revised: (1) is equal to

Original: [Equation]
Revised: [Equation]

Original: where D S×A ¯Ω is the Bregman divergence among visitation distributions deﬁned by D S×A ¯Ω ( d 1 || d 2 )
Revised: where D S×A ¯Ω is the Bregman divergence among visitation distributions deﬁned by D S×A ¯Ω ( d 1 || d 2 )

Original: =
Revised: =

Original: ¯ Ω( d 1 ) − ¯ Ω( d 2 ) −(cid:104)∇ ¯ Ω( d 2 ) , d 1 − d 2 (cid:105) for visitation distributions d 1 and d 2 .
Revised: ¯ Ω( d 1 ) − ¯ Ω( d 2 ) −(cid:104)∇ ¯ Ω( d 2 ) , d 1 − d 2 (cid:105) for visitation distributions d 1 and d 2 .

Original: The proof is in Appendix G.
Revised: The proof is in Appendix G.

Original: Note for to become a valid Bregman divergence.
Revised: Note for to become a valid Bregman divergence.

Original: Although the strict convexity of a policy regularizer Ω does not guarantee the assumption on the strict convexity of ¯Ω , it has been shown to be true for Shannon entropy regularizer ( ¯Ω = − ¯ H of
Revised: Although the strict convexity of a policy regularizer Ω does not guarantee the assumption on the strict convexity of ¯Ω , it has been shown to be true for Shannon entropy regularizer ( ¯Ω = − ¯ H of

Original: Lemma 3.1 in Ho & Ermon (2016)) and Tsallis entropy regularizer with its constants k = 12 , q =( ¯Ω = − ¯ W of Theorem 3 in Lee et al.
Revised: Lemma 3.1 in Ho & Ermon (2016)) and Tsallis entropy regularizer with its constants k = 12 , q =( ¯Ω = − ¯ W of Theorem 3 in Lee et al.

Original: 3. IRL WITH T SALLIS ENTROPY REGULARIZATION AND G AUSSIAN POLICIES
Revised: 3. IRL WITH T SALLIS ENTROPY REGULARIZATION AND G AUSSIAN POLICIES

Original: For continuous controls, multi-variate Gaussian policies are often used in practice (Schulman et al., 2015; 2017) and we consider IRL problems with those policies in this subsection.
Revised: For continuous controls, multi-variate Gaussian policies are often used in practice (Schulman et al., 2015; 2017) and we consider IRL problems with those policies in this subsection.

Original: In particular, we consider IRL with Tsallis entropy regularizer Ω
Revised: In particular, we consider IRL with Tsallis entropy regularizer Ω

Original: ( p ) =
Revised: ( p ) =

Original: −T kq ( p ) =
Revised: −T kq ( p ) =

Original: − E a ∼ p
Revised: − E a ∼ p

Original: [ kq − 1 (1 − p ( a ) q − 1 )] (Lee et al., 2018;
Revised: [ kq − 1 (1 − p ( a ) q − 1 )] (Lee et al., 2018;

Original: Yang et al., 2019; Lee et al., 2020) for a multi-variate Gaussian policy π ( ·| s ) =
Revised: Yang et al., 2019; Lee et al., 2020) for a multi-variate Gaussian policy π ( ·| s ) =

Original: N ( µµµ ( s ) , ΣΣΣ( s )) with µµµ ( s ) =
Revised: N ( µµµ ( s ) , ΣΣΣ( s )) with µµµ ( s ) =

Original: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } .
Revised: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } .

Original: In such a case, we can obtain tractable forms of the following quantities:
Revised: In such a case, we can obtain tractable forms of the following quantities:

Original: Tsallis entropy.
Revised: Tsallis entropy.

Original: The tractable form of Tsallis entropy for a multi-variate Gaussian policy is
Revised: The tractable form of Tsallis entropy for a multi-variate Gaussian policy is

Original: [Equation]
Revised: [Equation]

Original: for Renyi entropy R q .
Revised: for Renyi entropy R q .

Original: Its derivation is given in Appendix I.
Revised: Its derivation is given in Appendix I.

Original: Reward baseline.
Revised: Reward baseline.

Original: The reward baseline term E a ∼ π ( ·| s )
Revised: The reward baseline term E a ∼ π ( ·| s )

Original: [ f (cid:48) φ ( π ( a | s )) − φ
Revised: [ f (cid:48) φ ( π ( a | s )) − φ

Original: π ( a | s ))] in Corollary 1 is generally intractable except for either discrete control problems or Shannon entropy regularization where the reward baseline is equal to − 1 .
Revised: π ( a | s ))] in Corollary 1 is generally intractable except for either discrete control problems or Shannon entropy regularization where the reward baseline is equal to − 1 .

Original: Interestingly, as long as the tractable form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisﬁes
Revised: Interestingly, as long as the tractable form of Tsallis entropy can be derived, that of the corresponding reward baseline can also be derived since the reward baseline satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Here, the ﬁrst equality holds with f (cid:48) φ ( x ) =
Revised: Here, the ﬁrst equality holds with f (cid:48) φ ( x ) =

Original: kq − 1 (1 − qx q − 1 )
Revised: kq − 1 (1 − qx q − 1 )

Original: =
Revised: =

Original: qφ ( x ) − k for Tsallis entropy regularization.
Revised: qφ ( x ) − k for Tsallis entropy regularization.

Original: Bregman divergence associated with Tsallis entropy.
Revised: Bregman divergence associated with Tsallis entropy.

Original: For two different multivariate Gaussian policies, we derive the tractable form of the Bregman divergence (associated with Tsallis entropy) between two policies.
Revised: For two different multivariate Gaussian policies, we derive the tractable form of the Bregman divergence (associated with Tsallis entropy) between two policies.

Original: The resultant divergence has a complicated form, so we leave its derivation in
Revised: The resultant divergence has a complicated form, so we leave its derivation in

Original: Appendix I.3.
Revised: Appendix I.3.

Original: For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized
Revised: For deeper understanding of Tsallis entropy and its associated Bregman divergence in regularized

Original: MDPs, we consider an example in Figure 1.
Revised: MDPs, we consider an example in Figure 1.

Original: We ﬁrst assume that both learning agents’ and experts’ policies follow uni-variate Gaussian distributions π = N ( µ, σ 2 ) and π E = N (0 , ( e − 3 ) 2 ) , respectively.
Revised: We ﬁrst assume that both learning agents’ and experts’ policies follow uni-variate Gaussian distributions π = N ( µ, σ 2 ) and π E = N (0 , ( e − 3 ) 2 ) , respectively.

Original: We then evaluate the Bregman divergence in Figure 1 by using its tractable form and varying q from 1.0—which corresponds to the KL divergence—to 2.0.
Revised: We then evaluate the Bregman divergence in Figure 1 by using its tractable form and varying q from 1.0—which corresponds to the KL divergence—to 2.0.

Original: We observe that the constant q from the Tsallis entropy affects the sensitivity of the associated Bregman divergence w.r.t.
Revised: We observe that the constant q from the Tsallis entropy affects the sensitivity of the associated Bregman divergence w.r.t.

Original: the mean and standard deviation of the learning agent’s policy π .
Revised: the mean and standard deviation of the learning agent’s policy π .

Original: Speciﬁcally, as q increases, the size of the valley—relatively red region in Figure 1—across the µ -axis and log σ -axis decreases.
Revised: Speciﬁcally, as q increases, the size of the valley—relatively red region in Figure 1—across the µ -axis and log σ -axis decreases.

Original: This suggests that for larger q , minimizing the Bregman divergence requires more tightly matching means and variances of π and π E .
Revised: This suggests that for larger q , minimizing the Bregman divergence requires more tightly matching means and variances of π and π E .

Original:  A LGORITHMIC C ONSIDERATION
Revised:  A LGORITHMIC C ONSIDERATION

Original: Based on a solution for regularized IRL in the previous section, we focus on developing an IRL algorithm in this section.
Revised: Based on a solution for regularized IRL in the previous section, we focus on developing an IRL algorithm in this section.

Original: Particularly to recover the reward function t ( s, a ; π E ) in Lemma 1 , we design an adversarial training objective as follows.
Revised: Particularly to recover the reward function t ( s, a ; π E ) in Lemma 1 , we design an adversarial training objective as follows.

Original: Motivated by AIRL (Fu et al., 2018), we consider the following structured discriminator associated with π, r and t in Lemma 1 :
Revised: Motivated by AIRL (Fu et al., 2018), we consider the following structured discriminator associated with π, r and t in Lemma 1 :

Original: [Equation]
Revised: [Equation]

Original: Note that we can recover the discriminator of AIRL in Eq.(5) when t ( s, a ) = log π ( a | s ) ( φ ( x ) = log x and λ = 1 ).
Revised: Note that we can recover the discriminator of AIRL in Eq.(5) when t ( s, a ) = log π ( a | s ) ( φ ( x ) = log x and λ = 1 ).

Original: Then, we consider the following optimization objective of the discriminator which is the same as that of AIRL:
Revised: Then, we consider the following optimization objective of the discriminator which is the same as that of AIRL:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since the function x (cid:55)→ a log σ ( x ) + b log(1 − σ ( x )) attains its maximum at σ ( x ) = aa + b , or equivalently at x = log ab (Goodfellow et al., 2014; Mescheder et al., 2017), it can be shown that
Revised: Since the function x (cid:55)→ a log σ ( x ) + b log(1 − σ ( x )) attains its maximum at σ ( x ) = aa + b , or equivalently at x = log ab (Goodfellow et al., 2014; Mescheder et al., 2017), it can be shown that

Original: [Equation]
Revised: [Equation]

Original: When π = π E in Eq.(12), we have ˆ t ( s, a ; π E ) =
Revised: When π = π E in Eq.(12), we have ˆ t ( s, a ; π E ) =

Original: t ( s, a ; π E ) since d π = d π E , which means the maximizer ˆ t becomes the solution of IRL after the agent successfully imitates the expert policy π E .
Revised: t ( s, a ; π E ) since d π = d π E , which means the maximizer ˆ t becomes the solution of IRL after the agent successfully imitates the expert policy π E .

Original: To do so, we consider the following iterative algorithm.
Revised: To do so, we consider the following iterative algorithm.

Original: Assuming that we ﬁnd out the optimal reward approximator ˆ t ( s, a ; π ( i ) ) in Eq.(12) for the policy π ( i ) of the i -th iteration, we get the policy π
Revised: Assuming that we ﬁnd out the optimal reward approximator ˆ t ( s, a ; π ( i ) ) in Eq.(12) for the policy π ( i ) of the i -th iteration, we get the policy π

Original: ( i +1) by optimizing the following objective with gradient ascent:
Revised: ( i +1) by optimizing the following objective with gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: The above expectation in Eq.(13) can be decomposed into the following two terms
Revised: The above expectation in Eq.(13) can be decomposed into the following two terms

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) (II) , (14) where the second equality follows since Lemma 1 tells us that t ( s, a ; π ( i ) ) is a reward function that makes π ( i ) an optimal policy in the Ω -regularized MDP.
Revised: (cid:124) (cid:123)(cid:122) (cid:125) (II) , (14) where the second equality follows since Lemma 1 tells us that t ( s, a ; π ( i ) ) is a reward function that makes π ( i ) an optimal policy in the Ω -regularized MDP.

Original: Minimizing term (II) in Eq.(14) makes π ( i +1) close to π E while minimizing term (I) can be regarded as a conservative policy optimization around the policy π
Revised: Minimizing term (II) in Eq.(14) makes π ( i +1) close to π E while minimizing term (I) can be regarded as a conservative policy optimization around the policy π

Original: ( i ) (Schulman et al., 2015).
Revised: ( i ) (Schulman et al., 2015).

Original: In practice, we parameterize our reward and policy approximations with neural networks and train them using an off-policy Regularized Actor-Critic (RAC)
Revised: In practice, we parameterize our reward and policy approximations with neural networks and train them using an off-policy Regularized Actor-Critic (RAC)

Original: (Yang et al., 2019) as described in Algorithm 1 .Below, we evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios.
Revised: (Yang et al., 2019) as described in Algorithm 1 .Below, we evaluate our Regularized Adversarial Inverse Reinforcement Learning (RAIRL) approach across various scenarios.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We summarize the experimental setup as follows.
Revised: We summarize the experimental setup as follows.

Original: In our experiments, we consider Ω( p ) =
Revised: In our experiments, we consider Ω( p ) =

Original: − λ E a ∼ p
Revised: − λ E a ∼ p

Original: [ φ ( p ( a )] with the following regularizers from Yang et al.
Revised: [ φ ( p ( a )] with the following regularizers from Yang et al.

Original: (2019): (1) Shannon entropy ( φ ( x ) = − log x ), (2)
Revised: (2019): (1) Shannon entropy ( φ ( x ) = − log x ), (2)

Original: Tsallis entropy regularizer ( φ ( x ) =
Revised: Tsallis entropy regularizer ( φ ( x ) =

Original: kq − 1 (1 − x q − 1 ) ), (3) exp regularizer ( φ ( x ) =
Revised: kq − 1 (1 − x q − 1 ) ), (3) exp regularizer ( φ ( x ) =

Original: e − e x ), (4) cos regularizer ( φ ( x ) = cos( π 2 x ) ), (5) sin regularizer ( φ ( x ) = 1 − sin π 2 x ).
Revised: e − e x ), (4) cos regularizer ( φ ( x ) = cos( π 2 x ) ), (5) sin regularizer ( φ ( x ) = 1 − sin π 2 x ).

Original: The above regularizers were chosen since other regularizers have not been empirically validated to the best of our knowledge.
Revised: The above regularizers were chosen since other regularizers have not been empirically validated to the best of our knowledge.

Original: We chose these regularizers to make our empirical analysis more tractable.
Revised: We chose these regularizers to make our empirical analysis more tractable.

Original: In addition, we model the reward approximator of RAIRL as a neural network with either one of the following models: (1) Non-structured model (NSM) —a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018)—and (2) Density-based model (DBM) —a model using a neural network for π (softmax for discrete controls and multi-variate Gaussian model for continuous controls) of the solution in Eq.(1) (See Appendix J.2 for a detailed explanation).
Revised: In addition, we model the reward approximator of RAIRL as a neural network with either one of the following models: (1) Non-structured model (NSM) —a simple feed-forward neural network that outputs real values used in AIRL (Fu et al., 2018)—and (2) Density-based model (DBM) —a model using a neural network for π (softmax for discrete controls and multi-variate Gaussian model for continuous controls) of the solution in Eq.(1) (See Appendix J.2 for a detailed explanation).

Original: For the
Revised: For the

Original: RL algorithm of RAIRL, we implement Regularized Actor Critic (RAC)
Revised: RL algorithm of RAIRL, we implement Regularized Actor Critic (RAC)

Original: (Yang et al., 2019) on top of the SAC implementation from Rlpyt (Stooke & Abbeel, 2019).
Revised: (Yang et al., 2019) on top of the SAC implementation from Rlpyt (Stooke & Abbeel, 2019).

Original: Other settings are summarized in
Revised: Other settings are summarized in

Original: Appendix J. For all experiments, we use 5 runs and report 95% conﬁdence intervals.
Revised: Appendix J. For all experiments, we use 5 runs and report 95% conﬁdence intervals.

Original: 5. E XPERIMENT 1: M ULTI - ARMED B ANDIT (D ISCRETE A CTION )
Revised: 5. E XPERIMENT 1: M ULTI - ARMED B ANDIT (D ISCRETE A CTION )

Original: We consider a 4-armed bandit environment as shown in Figure 2 (left).
Revised: We consider a 4-armed bandit environment as shown in Figure 2 (left).

Original: An expert policy π E is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for a = 0 , 1 , 2 , 3 ) or sparse (with probability 0, 0, 1/3, 2/3 for a = 0 , 1 , 2 , 3 ).
Revised: An expert policy π E is assumed to be either dense (with probability 0.1, 0.2, 0.3, 0.4 for a = 0 , 1 , 2 , 3 ) or sparse (with probability 0, 0, 1/3, 2/3 for a = 0 , 1 , 2 , 3 ).

Original: For those experts, we use RAIRL with actions sampled from π E and compare learned rewards with the ground truth reward t ( s, a ; π E ) in Lemma 1 .
Revised: For those experts, we use RAIRL with actions sampled from π E and compare learned rewards with the ground truth reward t ( s, a ; π E ) in Lemma 1 .

Original: When π E is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices.
Revised: When π E is dense, RAIRL successfully acquires the ground truth rewards irrespective of the reward model choices.

Original: When sparse π E is used, however, RAIRL with a non-structured model (RAIRL-NSM) fails to recover the rewards for a = 0 , 1 —where π E ( a ) = 0 —due to the lack of samples at the end of the imitation.
Revised: When sparse π E is used, however, RAIRL with a non-structured model (RAIRL-NSM) fails to recover the rewards for a = 0 , 1 —where π E ( a ) = 0 —due to the lack of samples at the end of the imitation.

Original: On the other hand, RAIRL with a density-based model (RAIRL-DBM) can recover the correct rewards due to the softmax layer which maintains the sum over the outputs equal to 1.
Revised: On the other hand, RAIRL with a density-based model (RAIRL-DBM) can recover the correct rewards due to the softmax layer which maintains the sum over the outputs equal to 1.

Original: Therefore, we argue that using DBM is necessary for correct reward acquisition since a set of demonstrations is generally sparse.
Revised: Therefore, we argue that using DBM is necessary for correct reward acquisition since a set of demonstrations is generally sparse.

Original: In the following experiment, we show that the choice of reward models indeed affects the performance of rewards.
Revised: In the following experiment, we show that the choice of reward models indeed affects the performance of rewards.

Original: 5. E XPERIMENT 2: B ERMUDA W ORLD (C ONTINUOUS O BSERVATION , D ISCRETE A CTION )
Revised: 5. E XPERIMENT 2: B ERMUDA W ORLD (C ONTINUOUS O BSERVATION , D ISCRETE A CTION )

Original: We consider an environment with a 2-dimensional continuous state space as described in Figure 3.
Revised: We consider an environment with a 2-dimensional continuous state space as described in Figure 3.

Original: At each episode, the learning agent is initialized uniformly on the x -axis between − 5 and 5 , and there are 8 possible actions—an angle in {− π, − 34 π, ..., 12 π, 34 π } that determines the direction of movement.
Revised: At each episode, the learning agent is initialized uniformly on the x -axis between − 5 and 5 , and there are 8 possible actions—an angle in {− π, − 34 π, ..., 12 π, 34 π } that determines the direction of movement.

Original: An expert in Bermuda World considers 3 target positions ( − 5 , 10) , (0 , 10) , (5 , 10) and behaves stochastically.
Revised: An expert in Bermuda World considers 3 target positions ( − 5 , 10) , (0 , 10) , (5 , 10) and behaves stochastically.

Original: We state how we mathematically deﬁne the expert policy π E in Appendix J.3.
Revised: We state how we mathematically deﬁne the expert policy π E in Appendix J.3.

Original: During RAIRL’s training (Figure 3, Top row ), we use 1000 demonstrations sampled from the expert and periodically measure mean Bregman divergence, i.e.,
Revised: During RAIRL’s training (Figure 3, Top row ), we use 1000 demonstrations sampled from the expert and periodically measure mean Bregman divergence, i.e.,

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a )
Revised: a )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, the states s 1 , ..., s N come from 30 evaluation trajectories that are stochastically sampled from the agent’s policy π —which is ﬁxed during evaluation—in a separate evaluation environment.
Revised: Here, the states s 1 , ..., s N come from 30 evaluation trajectories that are stochastically sampled from the agent’s policy π —which is ﬁxed during evaluation—in a separate evaluation environment.

Original: During the evaluation of learned reward (Figure 3, Bottom row ), we train randomly initialized agents with
Revised: During the evaluation of learned reward (Figure 3, Bottom row ), we train randomly initialized agents with

Original: RAC and rewards acquired from RAIRL’s training and check if the mean Bregman divergence is properly minimized.
Revised: RAC and rewards acquired from RAIRL’s training and check if the mean Bregman divergence is properly minimized.

Original: We measure the mean Bregman divergence as was done in RAIRL’s training.
Revised: We measure the mean Bregman divergence as was done in RAIRL’s training.

Original: RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM during reward evaluation, although both achieve comparable performances during RAIRL’s training.
Revised: RAIRL-DBM is shown to minimize the target divergence more effectively compared to RAIRL-NSM during reward evaluation, although both achieve comparable performances during RAIRL’s training.

Original: Moreover, we substitute λ with 1 , 5 , 10 and observe that learning with λ larger than 1 returns better rewards—only λ = 1 was considered in AIRL (Fu et al., 2018).
Revised: Moreover, we substitute λ with 1 , 5 , 10 and observe that learning with λ larger than 1 returns better rewards—only λ = 1 was considered in AIRL (Fu et al., 2018).

Original: Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC).
Revised: Note that in all cases, the minimum divergence achieved by RAIRL is comparable with that of behavioral cloning (BC).

Original: This is because
Revised: This is because

Original: BC performs sufﬁciently well when many demonstrations are given.
Revised: BC performs sufﬁciently well when many demonstrations are given.

Original: We think the divergence of BC may be the near-optimal divergence that can be achieved with our policy neural network model.
Revised: We think the divergence of BC may be the near-optimal divergence that can be achieved with our policy neural network model.

Original: 5. E XPERIMENT 3: M U J O C O (C ONTINUOUS O BSERVATION AND A CTION )
Revised: 5. E XPERIMENT 3: M U J O C O (C ONTINUOUS O BSERVATION AND A CTION )

Original: We validate RAIRL on MuJoCo continuous control tasks ( Hopper-v2 , Walker-v2 , HalfCheetah-v2 ,
Revised: We validate RAIRL on MuJoCo continuous control tasks ( Hopper-v2 , Walker-v2 , HalfCheetah-v2 ,

Original: Ant-v2 ) as follows.
Revised: Ant-v2 ) as follows.

Original: We assume multivariate-Gaussian policies (with diagonal covariance matrices) for both learner’s policy π and expert policy π E .
Revised: We assume multivariate-Gaussian policies (with diagonal covariance matrices) for both learner’s policy π and expert policy π E .

Original: Instead of tanh -squashed policy in Soft-Actor
Revised: Instead of tanh -squashed policy in Soft-Actor

Original: Critic (Haarnoja et al., 2018), we use hyperbolized environments—where tanh is regarded as a part of the environment—with additional engineering on the policy networks (See Appendix J.4 for details).
Revised: Critic (Haarnoja et al., 2018), we use hyperbolized environments—where tanh is regarded as a part of the environment—with additional engineering on the policy networks (See Appendix J.4 for details).

Original: We use 100 demonstrations stochastically sampled from π E to validate RAIRL.
Revised: We use 100 demonstrations stochastically sampled from π E to validate RAIRL.

Original: In MuJoCo experiments, we focus on a set of Tsallis entropy regularizer ( Ω = −T 1 q ) with q = 1 , 1 . 5 , 2 —where
Revised: In MuJoCo experiments, we focus on a set of Tsallis entropy regularizer ( Ω = −T 1 q ) with q = 1 , 1 . 5 , 2 —where

Original: Tsallis entropy becomes Shannon entropy for q = 1 .
Revised: Tsallis entropy becomes Shannon entropy for q = 1 .

Original: We then exploit the tractable quantities for multi-variate Gaussian distributions in Section 3.2 to stabilize RAIRL and check its performance in terms of its mean Bregman divergence similar to the previous experiment.
Revised: We then exploit the tractable quantities for multi-variate Gaussian distributions in Section 3.2 to stabilize RAIRL and check its performance in terms of its mean Bregman divergence similar to the previous experiment.

Original: Note that since both π and π E are multi-variate Gaussian and can be evaluated, we can evaluate the individual Bregman divergence
Revised: Note that since both π and π E are multi-variate Gaussian and can be evaluated, we can evaluate the individual Bregman divergence

Original: D A Ω ( π ( ·| s )
Revised: D A Ω ( π ( ·| s )

Original: ||
Revised: ||

Original: π E ( ·| s )) on s by using the derivation in Appendix I.3.
Revised: π E ( ·| s )) on s by using the derivation in Appendix I.3.

Original: The performances during RAIRL’s training are described as follows.
Revised: The performances during RAIRL’s training are described as follows.

Original: We report π with both an episodic score (Figure 4) and mean Bregman divergences with respect to three types of Tsallis entropies (Figure 5)
Revised: We report π with both an episodic score (Figure 4) and mean Bregman divergences with respect to three types of Tsallis entropies (Figure 5)

Original: Ω = −T 1 q (cid:48) with q (cid:48)
Revised: Ω = −T 1 q (cid:48) with q (cid:48)

Original: = 1 , 1 . 5 , 2 .
Revised: = 1 , 1 . 5 , 2 .

Original: Note that the objective of RAIRL with Ω = −T 1 q is to minimize the corresponding mean Bregman divergence with q (cid:48)
Revised: Note that the objective of RAIRL with Ω = −T 1 q is to minimize the corresponding mean Bregman divergence with q (cid:48)

Original: =
Revised: =

Original: q .
Revised: q .

Original: In Figure 4, both RAIRLDBM and RAIRL-NSM are shown to achieve the expert performance, irrespective of q , in Hopperv2 , Walker-v2 , and HalfCheetah-v2 .
Revised: In Figure 4, both RAIRLDBM and RAIRL-NSM are shown to achieve the expert performance, irrespective of q , in Hopperv2 , Walker-v2 , and HalfCheetah-v2 .

Original: In contrast, RAIRL in Ant-v2 fails to achieve the expert’s performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in our setting.
Revised: In contrast, RAIRL in Ant-v2 fails to achieve the expert’s performance within 2,000,000 steps and RAIRL-NSM highly outperforms RAIRL-DBM in our setting.

Original: Although the episodic scores are comparable for all methods in Hopper-v2 , Walker-v2 , and
Revised: Although the episodic scores are comparable for all methods in Hopper-v2 , Walker-v2 , and

Original: HalfCheetah-v2 , respective divergences are shown to be highly different from one another as shown in
Revised: HalfCheetah-v2 , respective divergences are shown to be highly different from one another as shown in

Original: Figure 5.
Revised: Figure 5.

Original: RAIRL with q = 2 in most cases achieves the minimum mean Bregman divergence (for all three divergences with q (cid:48)
Revised: RAIRL with q = 2 in most cases achieves the minimum mean Bregman divergence (for all three divergences with q (cid:48)

Original: = 1 , 1 . 5 , 2 ), whereas RAIRL with q = 1 —which corresponds to AIRL (Fu et al., 2018)—achieves the maximum divergence in most cases.
Revised: = 1 , 1 . 5 , 2 ), whereas RAIRL with q = 1 —which corresponds to AIRL (Fu et al., 2018)—achieves the maximum divergence in most cases.

Original: This result is in alignment with our intuition from Section 3.2; as q increases, minimizing the Bregman divergence requires much tighter matching between π and π E .
Revised: This result is in alignment with our intuition from Section 3.2; as q increases, minimizing the Bregman divergence requires much tighter matching between π and π E .

Original: Unfortunately, while evaluating the acquired reward—RAC with a randomly initialized agent and acquired reward—the target divergence is not properly decreased in continuous controls.
Revised: Unfortunately, while evaluating the acquired reward—RAC with a randomly initialized agent and acquired reward—the target divergence is not properly decreased in continuous controls.

Original: We believe this is because π is a probability density function in continuous controls and causes large variance during training, while π is a mass function and is well-bounded in discrete control problems.
Revised: We believe this is because π is a probability density function in continuous controls and causes large variance during training, while π is a mass function and is well-bounded in discrete control problems.

Original:  D ISCUSSION AND F UTURE W ORKS
Revised:  D ISCUSSION AND F UTURE W ORKS

Original: We consider the problem of IRL in regularized MDPs (Geist et al., 2019), assuming a class of strongly convex policy regularizers.
Revised: We consider the problem of IRL in regularized MDPs (Geist et al., 2019), assuming a class of strongly convex policy regularizers.

Original: We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a speciﬁc instance of imitation learning—i.e., one that minimizes the Bregman divergence associated with policy regularizers.
Revised: We theoretically derive its solution (a set of reward functions) and show that learning with these rewards is equivalent to a speciﬁc instance of imitation learning—i.e., one that minimizes the Bregman divergence associated with policy regularizers.

Original: We propose RAIRL—a practical sampled-based IRL algorithm in regularized MDPs—and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).
Revised: We propose RAIRL—a practical sampled-based IRL algorithm in regularized MDPs—and evaluate its applicability on policy imitation (for discrete and continuous controls) and reward acquisition (for discrete control).

Original: Finally, recent advances in imitation learning and IRL are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ke et al., 2019; Ghasemipour et al., 2019).
Revised: Finally, recent advances in imitation learning and IRL are built from the perspective of regarding imitation learning as statistical divergence minimization problems (Ke et al., 2019; Ghasemipour et al., 2019).

Original: Although Bregman divergence is known to cover various divergences, it does not include some divergence families such as f -divergence (Csiszár, 1963; Amari, 2009).
Revised: Although Bregman divergence is known to cover various divergences, it does not include some divergence families such as f -divergence (Csiszár, 1963; Amari, 2009).

Original: Therefore, we believe that considering RL with policy regularization different from Geist et al.
Revised: Therefore, we believe that considering RL with policy regularization different from Geist et al.

Original: and its inverse problem is a possible way of ﬁnding the links between imitation learning and various statistical distances.
Revised: and its inverse problem is a possible way of ﬁnding the links between imitation learning and various statistical distances.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank Daewoo Kim at Waymo for his valuable comments on this work.

Original: A B ELLMAN O PERATORS , V ALUE F UNCTIONS IN REGULARIZED MDP S
Revised: A B ELLMAN O PERATORS , V ALUE F UNCTIONS IN REGULARIZED MDP S

Original: Let a policy regularizer Ω : ∆ A → R be strongly convex, and deﬁne the convex conjugate of Ω is Ω ∗ :
Revised: Let a policy regularizer Ω : ∆ A → R be strongly convex, and deﬁne the convex conjugate of Ω is Ω ∗ :

Original: R A → R as
Revised: R A → R as

Original: [Equation]
Revised: [Equation]

Original: Then, Bellman operators, equations and value functions in regularied MDPs are deﬁned as follows.
Revised: Then, Bellman operators, equations and value functions in regularied MDPs are deﬁned as follows.

Original: Deﬁnition 1 (Regularized Bellman Operators) .
Revised: Deﬁnition 1 (Regularized Bellman Operators) .

Original: For V ∈ R S , let us deﬁne Q ( s, a ) =
Revised: For V ∈ R S , let us deﬁne Q ( s, a ) =

Original: r ( s, a ) + γ E s (cid:48) ∼ P
Revised: r ( s, a ) + γ E s (cid:48) ∼ P

Original: [Equation]
Revised: [Equation]

Original: ·| s
Revised: ·| s

Original: [Equation]
Revised: [Equation]

Original: a ) [ V ( s (cid:48) )] .
Revised: a ) [ V ( s (cid:48) )] .

Original: The regularized Bellman evaluation operator is deﬁned as
Revised: The regularized Bellman evaluation operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and T π V =
Revised: and T π V =

Original: V is called the regularized Bellman equation.
Revised: V is called the regularized Bellman equation.

Original: Also, the regularized Bellman optimality operator is deﬁned as
Revised: Also, the regularized Bellman optimality operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and T ∗ V =
Revised: and T ∗ V =

Original: V is called the regularized Bellman optimality equation.
Revised: V is called the regularized Bellman optimality equation.

Original: Deﬁnition 2 (Regularized value functions) .
Revised: Deﬁnition 2 (Regularized value functions) .

Original: The unique ﬁxed point V π of the operator T π is called the state value function, and Q π ( s, a ) =
Revised: The unique ﬁxed point V π of the operator T π is called the state value function, and Q π ( s, a ) =

Original: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a )
Revised: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a )

Original: [ V π ( s (cid:48) )] is called the state-action value function.
Revised: [ V π ( s (cid:48) )] is called the state-action value function.

Original: Also, the unique ﬁxed point V ∗ of the operator T ∗ is called the optimal state value function, and Q ∗ ( s, a ) =
Revised: Also, the unique ﬁxed point V ∗ of the operator T ∗ is called the optimal state value function, and Q ∗ ( s, a ) =

Original: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ( s (cid:48) )] is called the optimal state-action value function.
Revised: r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ( s (cid:48) )] is called the optimal state-action value function.

Original: It should be noted that Proposition 1 in Geist et al.
Revised: It should be noted that Proposition 1 in Geist et al.

Original: (2019) tells us ∇ Ω ∗ ( Q ( s, · )) is a policy that uniquely maximizes Eq.(15).
Revised: (2019) tells us ∇ Ω ∗ ( Q ( s, · )) is a policy that uniquely maximizes Eq.(15).

Original: For example, when Ω( π ( ·| s ))
Revised: For example, when Ω( π ( ·| s ))

Original: = (cid:80) a ∼ π ( ·| s ) log π ( a | s ) (negative Shannon entropy), ∇ Ω ∗ ( Q ( s, · )) is a softmax policy, i.e., ∇ Ω ∗ ( Q ( s, · ))
Revised: = (cid:80) a ∼ π ( ·| s ) log π ( a | s ) (negative Shannon entropy), ∇ Ω ∗ ( Q ( s, · )) is a softmax policy, i.e., ∇ Ω ∗ ( Q ( s, · ))

Original: =
Revised: =

Original: exp( Q ( s, · )) (cid:80) a (cid:48)∈A exp( Q ( s,a (cid:48) )) .
Revised: exp( Q ( s, · )) (cid:80) a (cid:48)∈A exp( Q ( s,a (cid:48) )) .

Original: Due to this property, the optimal policy π ∗ of a regularized MDP is uniquely found for the optimal state-action value function Q ∗ which is also uniquely deﬁned as the ﬁxed point of T ∗ .
Revised: Due to this property, the optimal policy π ∗ of a regularized MDP is uniquely found for the optimal state-action value function Q ∗ which is also uniquely deﬁned as the ﬁxed point of T ∗ .

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Original: Let us deﬁne π s = π ( ·| s ) .
Revised: Let us deﬁne π s = π ( ·| s ) .

Original: For r ( s, a ) =
Revised: For r ( s, a ) =

Original: t ( s, a ; π E ) , the RL objective Eq.(1) satisﬁes
Revised: t ( s, a ; π E ) , the RL objective Eq.(1) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (cid:111)(cid:35)
Revised: (cid:111)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: (cid:34) ∞ (cid:88) i
Revised: (cid:34) ∞ (cid:88) i

Original: = γ i (cid:110) E a ∼ π si Ω (cid:48)
Revised: = γ i (cid:110) E a ∼ π si Ω (cid:48)

Original: ( s i , a ; π E ) − E a ∼ π siE Ω (cid:48) ( s i , a ; π E ) + Ω( π s iE ) − Ω( π s i ) (cid:111)(cid:35) ( iv ) =
Revised: ( s i , a ; π E ) − E a ∼ π siE Ω (cid:48) ( s i , a ; π E ) + Ω( π s iE ) − Ω( π s i ) (cid:111)(cid:35) ( iv ) =

Original: E π (cid:34) ∞ (cid:88) i
Revised: E π (cid:34) ∞ (cid:88) i

Original: = γ i (cid:8) (cid:104)∇ Ω( π s iE ) , π s i (cid:105)
Revised: = γ i (cid:8) (cid:104)∇ Ω( π s iE ) , π s i (cid:105)

Original: A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)
Revised: A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)

Original: A + Ω( π s iE ) − Ω( π s i ) (cid:9)(cid:35) =
Revised: A + Ω( π s iE ) − Ω( π s i ) (cid:9)(cid:35) =

Original: E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s iE ) − Ω( π s i )
Revised: E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s iE ) − Ω( π s i )

Original: + (cid:104)∇ Ω( π s iE ) , π s i (cid:105)
Revised: + (cid:104)∇ Ω( π s iE ) , π s i (cid:105)

Original: A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105) A (cid:9)(cid:35) =
Revised: A − (cid:104)∇ Ω( π s iE ) , π s iE (cid:105) A (cid:9)(cid:35) =

Original: − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) , π s i (cid:105)
Revised: − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) , π s i (cid:105)

Original: A
Revised: A

Original: + (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)
Revised: + (cid:104)∇ Ω( π s iE ) , π s iE (cid:105)

Original: A (cid:9) (cid:35) =
Revised: A (cid:9) (cid:35) =

Original: − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) ,
Revised: − E π (cid:34) ∞ (cid:88) i = γ i (cid:8) Ω( π s i ) − Ω( π s iE ) − (cid:104)∇ Ω( π s iE ) ,

Original: π s i − π s iE (cid:105) A (cid:9)(cid:35) ( v ) =
Revised: π s i − π s iE (cid:105) A (cid:9)(cid:35) ( v ) =

Original: − E π (cid:34) ∞ (cid:88) i =
Revised: − E π (cid:34) ∞ (cid:88) i =

Original: γ i D A Ω (
Revised: γ i D A Ω (

Original: π s i
Revised: π s i

Original: || π s iE ))
Revised: || π s iE ))

Original: (cid:35) , (16) where (i) follows from the assumption r ( s, a )
Revised: (cid:35) , (16) where (i) follows from the assumption r ( s, a )

Original: =
Revised: =

Original: t ( s, a ; π E ) in Lemma 1 , (ii) follows from the deﬁnition of t ( s, a ; π ) in Eq.(7), (iii) follows since taking the inner expectation ﬁrst does not change the overall expectation, (iv) follows from the deﬁnition of Ω (cid:48) in Lemma 1 and (cid:80) a ∈A p ( a )[ ∇ Ω( p )]( a ) =
Revised: t ( s, a ; π E ) in Lemma 1 , (ii) follows from the deﬁnition of t ( s, a ; π ) in Eq.(7), (iii) follows since taking the inner expectation ﬁrst does not change the overall expectation, (iv) follows from the deﬁnition of Ω (cid:48) in Lemma 1 and (cid:80) a ∈A p ( a )[ ∇ Ω( p )]( a ) =

Original: (cid:104)∇ Ω( p ) , p (cid:105) A , and (v) follows from the deﬁnition of Bregman divergence, i.e., D A Ω ( p 1 || p 2 )
Revised: (cid:104)∇ Ω( p ) , p (cid:105) A , and (v) follows from the deﬁnition of Bregman divergence, i.e., D A Ω ( p 1 || p 2 )

Original: =
Revised: =

Original: Ω( p 1 ) − Ω
Revised: Ω( p 1 ) − Ω

Original: ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105) A .
Revised: ( p 2 ) − (cid:104)∇ Ω( p 2 ) , p 1 − p 2 (cid:105) A .

Original: Due to the non-negativity of D A Ω ,
Revised: Due to the non-negativity of D A Ω ,

Original: Eq.(16) is less than or equal to zero which can be achieved when π = π E .
Revised: Eq.(16) is less than or equal to zero which can be achieved when π = π E .

Original: C P ROOF OF C OROLLARY
Revised: C P ROOF OF C OROLLARY

Original: Let a ∈ { 1 , ..., |A|} and π a = π ( a ) for simplicity.
Revised: Let a ∈ { 1 , ..., |A|} and π a = π ( a ) for simplicity.

Original: For
Revised: For

Original: [Equation]
Revised: [Equation]

Original: with f φ ( x ) =
Revised: with f φ ( x ) =

Original: xφ ( x ) , we have
Revised: xφ ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: for f (cid:48) φ ( x ) =
Revised: for f (cid:48) φ ( x ) =

Original: ∂∂x ( xφ ( x )) .
Revised: ∂∂x ( xφ ( x )) .

Original: Therefore, for π s = π ( ·| s ) we have
Revised: Therefore, for π s = π ( ·| s ) we have

Original: [Equation]
Revised: [Equation]

Original: ( π s ) , π s
Revised: ( π s ) , π s

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: sa
Revised: sa

Original: [Equation]
Revised: [Equation]

Original: a (cid:48)
Revised: a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: π sa (cid:48)
Revised: π sa (cid:48)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: sa (cid:48) )
Revised: sa (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: D P ROOF OF O PTIMAL R EWARDS ON C ONTINUOUS C ONTROLS
Revised: D P ROOF OF O PTIMAL R EWARDS ON C ONTINUOUS C ONTROLS

Original: Note that for two continuous distributions P 1 and P 2 having probability density functions p 1 ( x ) and p 2 ( x ) , respectively, the Bregman divergence can be deﬁned as (Guo et al., 2017; Jones & Byrne, 1990)
Revised: Note that for two continuous distributions P 1 and P 2 having probability density functions p 1 ( x ) and p 2 ( x ) , respectively, the Bregman divergence can be deﬁned as (Guo et al., 2017; Jones & Byrne, 1990)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ω (cid:48) ( x ) :
Revised: where ω (cid:48) ( x ) :

Original: = ∂∂x ω ( x ) and the divergence is measured point-wisely on x ∈ X .
Revised: = ∂∂x ω ( x ) and the divergence is measured point-wisely on x ∈ X .

Original: Let us assume
Revised: Let us assume

Original: [Equation]
Revised: [Equation]

Original: for a probability density function π on A and deﬁne
Revised: for a probability density function π on A and deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: 20) where (i) follows from r ( s, a ) =
Revised: 20) where (i) follows from r ( s, a ) =

Original: t ( s, a ; π E ) , (ii) follows from Eq.(18) and Eq.(19), and (iii) follows from the deﬁnition of Bregman divergence in Eq.(17).
Revised: t ( s, a ; π E ) , (ii) follows from Eq.(18) and Eq.(19), and (iii) follows from the deﬁnition of Bregman divergence in Eq.(17).

Original: Due to the non-negativity of D ω , Eq.(24) is less than or equal to zero which can be achieved when π = π E .
Revised: Due to the non-negativity of D ω , Eq.(24) is less than or equal to zero which can be achieved when π = π E .

Original: Also, π = π E is a unique solution since Eq.(1) has a unique solution for arbitrary reward functions.
Revised: Also, π = π E is a unique solution since Eq.(1) has a unique solution for arbitrary reward functions.

Original: E P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Original: Since Lemma 2 was mentioned but not proved in Geist et al.
Revised: Since Lemma 2 was mentioned but not proved in Geist et al.

Original: (2019), we derive the rigorous proof for
Revised: (2019), we derive the rigorous proof for

Original: Lemma 2 in this subsection.
Revised: Lemma 2 in this subsection.

Original: Note that we follow the proof idea in Ng et al. (1999).
Revised: Note that we follow the proof idea in Ng et al. (1999).

Original: First, let us assume an MDP M r with a reward r and corresponding optimal policy π ∗ .
Revised: First, let us assume an MDP M r with a reward r and corresponding optimal policy π ∗ .

Original: From
Revised: From

Original: Deﬁnition 1 , the optimal state-value function V ∗ ,r and its corresponding state-action value function
Revised: Deﬁnition 1 , the optimal state-value function V ∗ ,r and its corresponding state-action value function

Original: Q ∗ ,r ( s, a ) :
Revised: Q ∗ ,r ( s, a ) :

Original: = r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ,r ( s (cid:48) )] should satisfy the regularized Bellman optimality equation
Revised: = r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V ∗ ,r ( s (cid:48) )] should satisfy the regularized Bellman optimality equation

Original: [Equation]
Revised: [Equation]

Original: (21) where we explicitize the dependencies on r .
Revised: (21) where we explicitize the dependencies on r .

Original: Also, the optimal policy π ∗ is a unique maximizer for the above maximization.
Revised: Also, the optimal policy π ∗ is a unique maximizer for the above maximization.

Original: Now, let us consider the shaped rewards r ( s, a )+Φ( s (cid:48) ) − Φ( s ) and r ( s, a )+ E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) )
Revised: Now, let us consider the shaped rewards r ( s, a )+Φ( s (cid:48) ) − Φ( s ) and r ( s, a )+ E s (cid:48) ∼ P ( ·| s,a ) Φ( s (cid:48) )

Original: − Φ( s ) .
Revised: − Φ( s ) .

Original: Please note that for both rewards, the expectation over s (cid:48) for given s, a is equal to
Revised: Please note that for both rewards, the expectation over s (cid:48) for given s, a is equal to

Original: [Equation]
Revised: [Equation]

Original: and thus, it is sufﬁcient to only consider the optimality for ˜ r .
Revised: and thus, it is sufﬁcient to only consider the optimality for ˜ r .

Original: By subtracting Φ( s ) from the regularized optimality Bellman equation for r in Eq.(21), we have
Revised: By subtracting Φ( s ) from the regularized optimality Bellman equation for r in Eq.(21), we have

Original: [Equation]
Revised: [Equation]

Original: That is, V ∗ ,r − Φ is the ﬁxed point of the regularized Bellman optimality operator T ∗ , ˜ r associated with the shaped reward ˜ r .
Revised: That is, V ∗ ,r − Φ is the ﬁxed point of the regularized Bellman optimality operator T ∗ , ˜ r associated with the shaped reward ˜ r .

Original: Also, a maximizer for the above maximization is π ∗ since subtracting Φ( s ) from Eq.(21) does not change the unique maximizer π ∗ .
Revised: Also, a maximizer for the above maximization is π ∗ since subtracting Φ( s ) from Eq.(21) does not change the unique maximizer π ∗ .

Original: F C OMPARISON BETWEEN OUR SOLUTION AND EXISTING SOLUTION IN G EIST ET AL . (2019) F. I SSUES WITH THE SOLUTIONS IN G EIST ET AL . (2019)
Revised: F C OMPARISON BETWEEN OUR SOLUTION AND EXISTING SOLUTION IN G EIST ET AL . (2019) F. I SSUES WITH THE SOLUTIONS IN G EIST ET AL . (2019)

Original: In Proposition 5 of Geist et al.
Revised: In Proposition 5 of Geist et al.

Original: (2019), a solution of regularized IRL is given, and we rewrite the relevant theorem in this subsection.
Revised: (2019), a solution of regularized IRL is given, and we rewrite the relevant theorem in this subsection.

Original: Let us consider a regularized IRL problem, where π E ∈ ∆ AS is an expert policy.
Revised: Let us consider a regularized IRL problem, where π E ∈ ∆ AS is an expert policy.

Original: Assuming that both the model (dynamics, discount factor and regularizer) and the expert policy are known, Geist et al.
Revised: Assuming that both the model (dynamics, discount factor and regularizer) and the expert policy are known, Geist et al.

Original: (2019) proposed a solution of regularized IRL as follows:
Revised: (2019) proposed a solution of regularized IRL as follows:

Original: Lemma 4 (A solution of regularized IRL from Geist et al.
Revised: Lemma 4 (A solution of regularized IRL from Geist et al.

Original: Let Q E ∈ R
Revised: Let Q E ∈ R

Original: S×A be a function such that π E ( ·| s ) =
Revised: S×A be a function such that π E ( ·| s ) =

Original: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S .
Revised: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S .

Original: Also, deﬁne
Revised: Also, deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then, in the Ω -regularized MDP with the reward r E , π E is an optimal policy.
Revised: Then, in the Ω -regularized MDP with the reward r E , π E is an optimal policy.

Original: Proof.
Revised: Proof.

Original: Although the brief idea of the proof is given in Geist et al.
Revised: Although the brief idea of the proof is given in Geist et al.

Original: (2019), we rewrite the proof in a more rigorous way as follows.
Revised: (2019), we rewrite the proof in a more rigorous way as follows.

Original: Let us deﬁne V E ( s )
Revised: Let us deﬁne V E ( s )

Original: = (cid:104) π E ( ·| s ) , Q E ( s, · ) (cid:105)
Revised: = (cid:104) π E ( ·| s ) , Q E ( s, · ) (cid:105)

Original: A − Ω( π E ( ·| s )) .
Revised: A − Ω( π E ( ·| s )) .

Original: Then, r E ( s, a ) =
Revised: Then, r E ( s, a ) =

Original: Q E ( s, a ) − γ E s (cid:48) ∼ P ( ·| s,a ) [ V E ( s (cid:48) )] , and thus, Q E ( s, a ) =
Revised: Q E ( s, a ) − γ E s (cid:48) ∼ P ( ·| s,a ) [ V E ( s (cid:48) )] , and thus, Q E ( s, a ) =

Original: r E ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V E ( s (cid:48) )] .
Revised: r E ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ V E ( s (cid:48) )] .

Original: By using this and regularized Bellman optimality operator, we have
Revised: By using this and regularized Bellman optimality operator, we have

Original: [Equation]
Revised: [Equation]

Original: where (i) and (ii) follow from Deﬁnition 1 , and (iii) follows since π E is a unique maximizer.
Revised: where (i) and (ii) follow from Deﬁnition 1 , and (iii) follows since π E is a unique maximizer.

Original: Thus,
Revised: Thus,

Original: V E is the ﬁxed point of T ∗ , and π E ( ·| s ) =
Revised: V E is the ﬁxed point of T ∗ , and π E ( ·| s ) =

Original: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S , becomes an optimal policy.
Revised: ∇ Ω ∗ ( Q E ( s, · )) , s ∈ S , becomes an optimal policy.

Original: For example, when negative Shannon entropy is used as a regularizer, we can get r E ( s, a )
Revised: For example, when negative Shannon entropy is used as a regularizer, we can get r E ( s, a )

Original: = log π E ( a | s ) by setting Q E ( s, a )
Revised: = log π E ( a | s ) by setting Q E ( s, a )

Original: = log π E ( a | s ) .
Revised: = log π E ( a | s ) .

Original: However, a solution proposed in Lemma 4 has two crucial issues:
Revised: However, a solution proposed in Lemma 4 has two crucial issues:

Original: Issue 1.
Revised: Issue 1.

Original: It requires the knowledge on the model dynamics, which is generally intractable.
Revised: It requires the knowledge on the model dynamics, which is generally intractable.

Original: Issue 2.
Revised: Issue 2.

Original: We need to ﬁgure out Q E that satisﬁes π E ( ·| s ) =
Revised: We need to ﬁgure out Q E that satisﬁes π E ( ·| s ) =

Original: ∇ Ω ∗ ( Q E ( s, · )) , which comes from the relationship between the optimal value function and optimal policy (Geist et al., 2019).
Revised: ∇ Ω ∗ ( Q E ( s, · )) , which comes from the relationship between the optimal value function and optimal policy (Geist et al., 2019).

Original: In the following subsection, we show how our solution in Lemma 1 is related to the solution from
Revised: In the following subsection, we show how our solution in Lemma 1 is related to the solution from

Original: Geist et al.
Revised: Geist et al.

Original: (2019) in Lemma 4 .
Revised: (2019) in Lemma 4 .

Original: F. R ELATION BETWEEN THE SOLUTION OF G EIST ET AL . (2019) AND OUR TRACTABLE SOLUTION
Revised: F. R ELATION BETWEEN THE SOLUTION OF G EIST ET AL . (2019) AND OUR TRACTABLE SOLUTION

Original: Let us consider the expert policy π E and functions Q E and r E satisfying the conditions in Lemma 4 .
Revised: Let us consider the expert policy π E and functions Q E and r E satisfying the conditions in Lemma 4 .

Original: From Lemma 2 , a regularized MDP with the shaped reward
Revised: From Lemma 2 , a regularized MDP with the shaped reward

Original: [Equation]
Revised: [Equation]

Original: for Φ ∈ R S has its optimal policy as π E .
Revised: for Φ ∈ R S has its optimal policy as π E .

Original: Since Φ can be arbitrarily chosen, let us assume Φ( s )
Revised: Since Φ can be arbitrarily chosen, let us assume Φ( s )

Original: =
Revised: =

Original: Ω ∗ ( Q E ( s, · )) .
Revised: Ω ∗ ( Q E ( s, · )) .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Note that the reward in Eq.
Revised: Note that the reward in Eq.

Original: (22) does not require the knowledge on the model dynamics, which addresses Issue 1 in Appendix F.1.
Revised: (22) does not require the knowledge on the model dynamics, which addresses Issue 1 in Appendix F.1.

Original: Also, by using V E ( s ) =
Revised: Also, by using V E ( s ) =

Original: Ω ∗ ( Q E ( s, · )) in the proof of Lemma 4 , the reward in Eq.
Revised: Ω ∗ ( Q E ( s, · )) in the proof of Lemma 4 , the reward in Eq.

Original: (22) can be written as
Revised: (22) can be written as

Original: [Equation]
Revised: [Equation]

Original: which means ˜ r E ( s, a ) is an advantage function for the optimal policy π E in the Ω -regularized MDP.
Revised: which means ˜ r E ( s, a ) is an advantage function for the optimal policy π E in the Ω -regularized MDP.

Original: However, we still have Issue 2 in Appendix F.1 since Ω ∗ in Eq.
Revised: However, we still have Issue 2 in Appendix F.1 since Ω ∗ in Eq.

Original: (22) is generally intractable (Geist et al., 2019), which prevents us from ﬁnding the appropriate Q E ( s, a ) .
Revised: (22) is generally intractable (Geist et al., 2019), which prevents us from ﬁnding the appropriate Q E ( s, a ) .

Original: Interestingly, we show that for all s ∈ S and Q E ( s, · ) =
Revised: Interestingly, we show that for all s ∈ S and Q E ( s, · ) =

Original: ∇ Ω( π E ( ·| s )) ,
Revised: ∇ Ω( π E ( ·| s )) ,

Original: [Equation]
Revised: [Equation]

Original: where the last equality holds since the Bregman divergence
Revised: where the last equality holds since the Bregman divergence

Original: D A Ω ( π ( ·| s )
Revised: D A Ω ( π ( ·| s )

Original: ||
Revised: ||

Original: π E ( ·| s )) is greater than or equal to zero and its lowest value is achieved when π ( ·| s )
Revised: π E ( ·| s )) is greater than or equal to zero and its lowest value is achieved when π ( ·| s )

Original: =
Revised: =

Original: π E ( ·| s ) .
Revised: π E ( ·| s ) .

Original: This means that when
Revised: This means that when

Original: Q E ( s, · ) = ∇ Ω( π E ( ·| s )) is used, the condition π E ( ·| s )
Revised: Q E ( s, · ) = ∇ Ω( π E ( ·| s )) is used, the condition π E ( ·| s )

Original: = ∇ Ω ∗ ( Q E ( s, · )) in Lemma 4 is satisﬁed without knowing the tractable form of Ω ∗ or ∇ Ω ∗ .
Revised: = ∇ Ω ∗ ( Q E ( s, · )) in Lemma 4 is satisﬁed without knowing the tractable form of Ω ∗ or ∇ Ω ∗ .

Original: Thus, Issue 2 in Appendix F.1 is addressed; we instead require the knowledge on the gradient ∇ Ω of the policy regularizer Ω , which is practically more tractable.
Revised: Thus, Issue 2 in Appendix F.1 is addressed; we instead require the knowledge on the gradient ∇ Ω of the policy regularizer Ω , which is practically more tractable.

Original: Finally, by substituting Q E ( s, · ) =
Revised: Finally, by substituting Q E ( s, · ) =

Original: Ω (cid:48)
Revised: Ω (cid:48)

Original: ( s, · ; π E ) for Ω (cid:48) ( s, · ; π ) :=
Revised: ( s, · ; π E ) for Ω (cid:48) ( s, · ; π ) :=

Original: ∇ Ω( π ( ·| s )) , s ∈ S , to Eq.(22), we have
Revised: ∇ Ω( π ( ·| s )) , s ∈ S , to Eq.(22), we have

Original: [Equation]
Revised: [Equation]

Original: where t ( s, a ; π E ) is our proposed solution in Lemma 1 .
Revised: where t ( s, a ; π E ) is our proposed solution in Lemma 1 .

Original: G P ROOF OF L EMMA
Revised: G P ROOF OF L EMMA

Original: RL objective in Regularized MDPs w.r.t.
Revised: RL objective in Regularized MDPs w.r.t.

Original: normalized visitation distributions.
Revised: normalized visitation distributions.

Original: For a reward function r ∈ R S×A and a strongly convex function Ω : ∆ A → R , the RL objetive J Ω ( r, π ) in Eq.(1) is equivalent to
Revised: For a reward function r ∈ R S×A and a strongly convex function Ω : ∆ A → R , the RL objetive J Ω ( r, π ) in Eq.(1) is equivalent to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where for a set D of normalized visitation distributions (Syed et al., 2008)
Revised: where for a set D of normalized visitation distributions (Syed et al., 2008)

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: we deﬁne ¯Ω( d ) :
Revised: we deﬁne ¯Ω( d ) :

Original: =
Revised: =

Original: E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] and ¯ π d ( ·| s ) :
Revised: E ( s,a ) ∼ d [Ω(¯ π d ( ·| s ))] and ¯ π d ( ·| s ) :

Original: = d ( s, · ) (cid:80) a (cid:48)
Revised: = d ( s, · ) (cid:80) a (cid:48)

Original: d ( s,a (cid:48) ) ∈
Revised: d ( s,a (cid:48) ) ∈

Original: ∆ AS for d ∈ D and use ¯ π d π ( ·| s )
Revised: ∆ AS for d ∈ D and use ¯ π d π ( ·| s )

Original: =
Revised: =

Original: π ( ·| s ) for all s ∈ S .
Revised: π ( ·| s ) for all s ∈ S .

Original: For ¯Ω : D → R , its convex conjugate ¯Ω ∗ is
Revised: For ¯Ω : D → R , its convex conjugate ¯Ω ∗ is

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from using the one-to-one correspondence between policies and visitation distributions (Syed et al., 2008; Ho & Ermon, 2016).
Revised: where (i) follows from using the one-to-one correspondence between policies and visitation distributions (Syed et al., 2008; Ho & Ermon, 2016).

Original: Note that Eq.(24) is equal to the optimal discounted average return in regularized MDPs.
Revised: Note that Eq.(24) is equal to the optimal discounted average return in regularized MDPs.

Original: IRL objective in regularized MDPs w.r.t.
Revised: IRL objective in regularized MDPs w.r.t.

Original: normalized visitation distributions.
Revised: normalized visitation distributions.

Original: By using the RL objective in Eq.(23), we can rewrite the IRL objective in Eq.(6) w.r.t.
Revised: By using the RL objective in Eq.(23), we can rewrite the IRL objective in Eq.(6) w.r.t.

Original: the normalized visitation distributions as the maximization of the following objective over r ∈ R S×A :
Revised: the normalized visitation distributions as the maximization of the following objective over r ∈ R S×A :

Original: [Equation]
Revised: [Equation]

Original: Note that if ∇ ¯Ω( d ) is well-deﬁned and r = ∇ ¯Ω( d π E ) for any strictly convex ¯Ω , Eq.(25) is equal to
Revised: Note that if ∇ ¯Ω( d ) is well-deﬁned and r = ∇ ¯Ω( d π E ) for any strictly convex ¯Ω , Eq.(25) is equal to

Original: [Equation]
Revised: [Equation]

Original: where the equality comes from the deﬁnition of Bregman divergence.
Revised: where the equality comes from the deﬁnition of Bregman divergence.

Original: Proof of t ( s, a ; π d ) = ∇ [¯Ω( d )]( s, a ) .
Revised: Proof of t ( s, a ; π d ) = ∇ [¯Ω( d )]( s, a ) .

Original: For simpler notation, we use matrix-vector notation for the proof when discrete state and action spaces S
Revised: For simpler notation, we use matrix-vector notation for the proof when discrete state and action spaces S

Original: = { 1 , ..., |S|} and A = { 1 , ..., |A|} are considered.
Revised: = { 1 , ..., |S|} and A = { 1 , ..., |A|} are considered.

Original: For a normalized visitation distribution d ∈ D , let us deﬁne
Revised: For a normalized visitation distribution d ∈ D , let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: where 111 A = [1 , ..., 1] T ∈
Revised: where 111 A = [1 , ..., 1] T ∈

Original: R
Revised: R

Original: A is an |A| -dimensional all-one vector.
Revised: A is an |A| -dimensional all-one vector.

Original: By using these notations
Revised: By using these notations

Original: [Equation]
Revised: [Equation]

Original: the original ¯Ω can be rewritten as
Revised: the original ¯Ω can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: The gradient of ¯Ω w.r.t.
Revised: The gradient of ¯Ω w.r.t.

Original: DDD (using denominator-layout notation) is
Revised: DDD (using denominator-layout notation) is

Original: [Equation]
Revised: [Equation]

Original: where each element of ∇ DDD ¯Ω( DDD ) satisﬁes
Revised: where each element of ∇ DDD ¯Ω( DDD ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: 26) for
Revised: 26) for

Original: [Equation]
Revised: [Equation]

Original: Note that each element of ∂ ¯ π a ( ddd s ) ∂ddd s satisﬁes
Revised: Note that each element of ∂ ¯ π a ( ddd s ) ∂ddd s satisﬁes

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: By substituting Eq.(27) into Eq
Revised: By substituting Eq.(27) into Eq

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: If we use the function notation, Eq.(28) can be written as
Revised: If we use the function notation, Eq.(28) can be written as

Original: [Equation]
Revised: [Equation]

Original: for t of Eq.(7) in Lemma 1 .
Revised: for t of Eq.(7) in Lemma 1 .

Original: H D ERIVATION OF B REGMAN -D IVERGENCE -B ASED M EASURE IN C ONTINUOUS C ONTROLS
Revised: H D ERIVATION OF B REGMAN -D IVERGENCE -B ASED M EASURE IN C ONTINUOUS C ONTROLS

Original: In Eq.(17), the Bregman divergence in the control task is deﬁned as
Revised: In Eq.(17), the Bregman divergence in the control task is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that we consider Ω( p ) =
Revised: Note that we consider Ω( p ) =

Original: (cid:82) X ω ( p ( x )) dx =
Revised: (cid:82) X ω ( p ( x )) dx =

Original: (cid:82) X [ − f φ ( p ( x ))] dx for f φ ( x ) =
Revised: (cid:82) X [ − f φ ( p ( x ))] dx for f φ ( x ) =

Original: xφ ( x )
Revised: xφ ( x )

Original: [Equation]
Revised: [Equation]

Original: which makes Eq.(29) equal to
Revised: which makes Eq.(29) equal to

Original: [Equation]
Revised: [Equation]

Original: Thus, by considering a learning agent’s policy π s = π ( ·| s ) , expert policy π s E = π E ( ·| s )
Revised: Thus, by considering a learning agent’s policy π s = π ( ·| s ) , expert policy π s E = π E ( ·| s )

Original: [Equation]
Revised: [Equation]

Original: and the objective in Eq.(8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:
Revised: and the objective in Eq.(8) characterized by the Bregman divergence, we can think of the following measure between expert and agent policies:

Original: [Equation]
Revised: [Equation]

Original: I T SALLIS ENTROPY AND ASSOCIATED B REGMAN DIVERGENCE AMONG MULTI - VARIATE G AUSSIAN DISTRIBUTIONS
Revised: I T SALLIS ENTROPY AND ASSOCIATED B REGMAN DIVERGENCE AMONG MULTI - VARIATE G AUSSIAN DISTRIBUTIONS

Original: Based on the derivation in Nielsen & Nock (2011)
Revised: Based on the derivation in Nielsen & Nock (2011)

Original: [Equation]
Revised: [Equation]

Original: we derive the Tsallis entropy and associated
Revised: we derive the Tsallis entropy and associated

Original: Bremgan divergence as follows.
Revised: Bremgan divergence as follows.

Original: We ﬁrst consider the distributions in the exponential family
Revised: We ﬁrst consider the distributions in the exponential family

Original: [Equation]
Revised: [Equation]

Original: Note that for
Revised: Note that for

Original: [Equation]
Revised: [Equation]

Original: we can recover the multi-variate Gaussian distribution (Nielsen & Nock, 2011):
Revised: we can recover the multi-variate Gaussian distribution (Nielsen & Nock, 2011):

Original: [Equation]
Revised: [Equation]

Original: For two distributions with k ( x ) = 0 ,
Revised: For two distributions with k ( x ) = 0 ,

Original: [Equation]
Revised: [Equation]

Original: that share t, F , and k , it can be shown that
Revised: that share t, F , and k , it can be shown that

Original: [Equation]
Revised: [Equation]

Original: since
Revised: since

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: dx = (cid:90) exp (cid:16) (cid:104)
Revised: dx = (cid:90) exp (cid:16) (cid:104)

Original: αθ +
Revised: αθ +

Original: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) exp (cid:16) F ( αθ + β ˆ θ )
Revised: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) exp (cid:16) F ( αθ + β ˆ θ )

Original: − αF ( θ ) − βF
Revised: − αF ( θ ) − βF

Original: (ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) (cid:90) exp (cid:16) (cid:104)
Revised: (ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) (cid:90) exp (cid:16) (cid:104)

Original: αθ +
Revised: αθ +

Original: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) .
Revised: β ˆ θ, t ( x ) (cid:105) − F ( αθ + β ˆ θ ) (cid:17) dx = exp (cid:16) F ( αθ + β ˆ θ ) − αF ( θ ) − βF (ˆ θ ) (cid:17) .

Original: I. T SALLIS E NTROPY
Revised: I. T SALLIS E NTROPY

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) and k = 1 , the Tsallis entropy of π can be written as
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) and k = 1 , the Tsallis entropy of π can be written as

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: If π is a multivariate Gaussian distribution, we have
Revised: If π is a multivariate Gaussian distribution, we have

Original: [Equation]
Revised: [Equation]

Original: For Σ = diag { σ 21 , ..., σ 2 d } , we have
Revised: For Σ = diag { σ 21 , ..., σ 2 d } , we have

Original: [Equation]
Revised: [Equation]

Original: I. T RACTABLE FORM OF BASELINE
Revised: I. T RACTABLE FORM OF BASELINE

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) , we have
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the baseline can be rewritten as
Revised: Therefore, the baseline can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: For a multivariate Gaussian distribution π , the tractable form of E x ∼ π [ − f (cid:48) φ ( x )+ φ ( x )] can be derived by using that of Tsallis entropy T q ( π ) of π .
Revised: For a multivariate Gaussian distribution π , the tractable form of E x ∼ π [ − f (cid:48) φ ( x )+ φ ( x )] can be derived by using that of Tsallis entropy T q ( π ) of π .

Original: I. B REGMAN DIVERGENCE WITH T SALLIS ENTROPY REGULARIZATION
Revised: I. B REGMAN DIVERGENCE WITH T SALLIS ENTROPY REGULARIZATION

Original: In Eq.(30)
Revised: In Eq.(30)

Original: [Equation]
Revised: [Equation]

Original: we consider the following form of the Bregman divergence:
Revised: we consider the following form of the Bregman divergence:

Original: [Equation]
Revised: [Equation]

Original: For φ ( x ) = kq − 1 (1 − x q − 1 ) , f (cid:48) φ ( x ) = kq −
Revised: For φ ( x ) = kq − 1 (1 − x q − 1 ) , f (cid:48) φ ( x ) = kq −

Original: [Equation]
Revised: [Equation]

Original: 1 − qx q − 1 ) =
Revised: 1 − qx q − 1 ) =

Original: qφ ( x ) − k
Revised: qφ ( x ) − k

Original: [Equation]
Revised: [Equation]

Original: and k = 1 , the above form is equal to
Revised: and k = 1 , the above form is equal to

Original: [Equation]
Revised: [Equation]

Original: For multivariate Gaussians
Revised: For multivariate Gaussians

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: J E XPERIMENT S ETTING J. P OLICY R EGULARIZERS IN E XPERIMENTS
Revised: J E XPERIMENT S ETTING J. P OLICY R EGULARIZERS IN E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: J. D ENSITY - BASED MODEL
Revised: J. D ENSITY - BASED MODEL

Original: By exploiting the knowledge on the reward in Corollary
Revised: By exploiting the knowledge on the reward in Corollary

Original: [Equation]
Revised: [Equation]

Original: we consider the density-based model (DBM) which is deﬁned by
Revised: we consider the density-based model (DBM) which is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for θ = ( θ 1 , θ 2 ) .
Revised: for θ = ( θ 1 , θ 2 ) .

Original: Here, f (cid:48) φ is a function that can be known priorly, and π θ 1 is a neural network which is deﬁned separately from the policy neural network π ψ .
Revised: Here, f (cid:48) φ is a function that can be known priorly, and π θ 1 is a neural network which is deﬁned separately from the policy neural network π ψ .

Original: The DBM for discrete control problems are depicted in Figure 6 ( Left ).
Revised: The DBM for discrete control problems are depicted in Figure 6 ( Left ).

Original: The model outputs rewards over all actions in parallel, where softmax is used for π θ 1 ( ·| s ) and − f (cid:48) φ is elementwisely applied to those softmax outputs followed by elementwisely adding B θ 2 ( s ) .
Revised: The model outputs rewards over all actions in parallel, where softmax is used for π θ 1 ( ·| s ) and − f (cid:48) φ is elementwisely applied to those softmax outputs followed by elementwisely adding B θ 2 ( s ) .

Original: For continuous control (Figure 6, Right ), we use the network architecture similar to that in discrete control, where a multivariate Gaussian distribution is used instead of a softmax layer.
Revised: For continuous control (Figure 6, Right ), we use the network architecture similar to that in discrete control, where a multivariate Gaussian distribution is used instead of a softmax layer.

Original: J. E XPERT IN B ERMUDA W ORLD ENVIRONMENT
Revised: J. E XPERT IN B ERMUDA W ORLD ENVIRONMENT

Original: We assume a stochastic expert deﬁned by
Revised: We assume a stochastic expert deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for s =
Revised: for s =

Original: ( x, y ) , ¯ s (1) = (¯ x (1) , ¯ y (1) ) =
Revised: ( x, y ) , ¯ s (1) = (¯ x (1) , ¯ y (1) ) =

Original: ( − 5 , 10) , ¯ s (2) =
Revised: ( − 5 , 10) , ¯ s (2) =

Original: (¯ x (2) , ¯ y (2) ) =
Revised: (¯ x (2) , ¯ y (2) ) =

Original: (0 , 10) , ¯ s (3) =
Revised: (0 , 10) , ¯ s (3) =

Original: (¯ x (3) , ¯ y (3) ) =
Revised: (¯ x (3) , ¯ y (3) ) =

Original: (5 , 10) , (cid:15) =
Revised: (5 , 10) , (cid:15) =

Original: 10 − 4 and an operator Proj( θ ) : R →
Revised: 10 − 4 and an operator Proj( θ ) : R →

Original: A that maps θ to the closest angle in A .
Revised: A that maps θ to the closest angle in A .

Original: In Figure 7, we depicted the expert policy.
Revised: In Figure 7, we depicted the expert policy.

Original: [Figure]
Revised: [Figure]

Original: M U J O C O EXPERIMENT SETTING
Revised: M U J O C O EXPERIMENT SETTING

Original: Instead of directly using MuJoCo environments with tanh -squashed policies proposed in Soft ActorCritic (SAC) (Haarnoja et al., 2018), we move tanh to a part of the environment—named hyperbolized environments in short—and assume Gaussian policies.
Revised: Instead of directly using MuJoCo environments with tanh -squashed policies proposed in Soft ActorCritic (SAC) (Haarnoja et al., 2018), we move tanh to a part of the environment—named hyperbolized environments in short—and assume Gaussian policies.

Original: Speciﬁcally, after an action a is sampled from the policies, we pass tanh( a ) to the environment.
Revised: Speciﬁcally, after an action a is sampled from the policies, we pass tanh( a ) to the environment.

Original: We then consider multi-variate Gaussian policy
Revised: We then consider multi-variate Gaussian policy

Original: [Equation]
Revised: [Equation]

Original: with µµµ ( s ) =
Revised: with µµµ ( s ) =

Original: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } , where − arctanh(0 . 99) ≤ µ i ( s ) ≤ arctanh(0 . 99) , log(0 . 01)
Revised: [ µ 1 ( s ) , ..., µ d ( s )] T , ΣΣΣ( s ) = diag { ( σ 1 ( s )) 2 , ..., ( σ d ( s )) 2 } , where − arctanh(0 . 99) ≤ µ i ( s ) ≤ arctanh(0 . 99) , log(0 . 01)

Original: ≤ log σ i ( s ) ≤ log(2) for all i
Revised: ≤ log σ i ( s ) ≤ log(2) for all i

Original: = 1 , ..., d .
Revised: = 1 , ..., d .

Original: Instead of using clipping, we use tanh -activated outputs and scale them to ﬁt in the above ranges, which empirically improves the performance.
Revised: Instead of using clipping, we use tanh -activated outputs and scale them to ﬁt in the above ranges, which empirically improves the performance.

Original: Also, instead of using potential-based reward shaping used in AIRL (Fu et al., 2018), we update the moving mean of intermediate reward values and update the value network with mean-subtracted rewards—so that the value network gets approximately mean-zero reward—to stabilize the RL part of RAIRL.
Revised: Also, instead of using potential-based reward shaping used in AIRL (Fu et al., 2018), we update the moving mean of intermediate reward values and update the value network with mean-subtracted rewards—so that the value network gets approximately mean-zero reward—to stabilize the RL part of RAIRL.

Original: Note that this is motivated by
Revised: Note that this is motivated by

Original: Lemma 2 from which we can guarantee that any constant shift of reward functions does not change optimality.
Revised: Lemma 2 from which we can guarantee that any constant shift of reward functions does not change optimality.

Original: J. H YPERPARAMETERS
Revised: J. H YPERPARAMETERS

Original: Table 2, Table 3 and Table 4 list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.
Revised: Table 2, Table 3 and Table 4 list the parameters used in our Bandit, Bermuda World, and MuJoCo experiments, respectively.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]