R OBUST I MITATION VIA M IRROR D ESCENT I NVERSE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial imitation learning techniques are based on modeling statistical divergences using agent and expert demonstration data.
However, unbiased minimization of these divergences is not usually guaranteed due to the geometry of the underlying space.
Furthermore, when the size of demonstrations is not sufficient, estimated reward functions from the discriminative signals become uncertain and fail to give informative feedback.
Instead of formulating a global cost at once, we consider reward functions as an iterative sequence in a proximal method.
In this paper, we show that rewards dervied by mirror descent ensures minimization of a Bregman divergence in terms of a rigorous regret bound of O (1 /T ) for a particular condition of step sizes { η t } Tt =1 .
The resulting mirror descent adversarial inverse reinforcement learning (MD-AIRL) algorithm gradually advances a parameterized reward function in an associated reward space, and the sequence of such functions provides optimization targets for the policy space.
We empirically validate our method in discrete and continuous benchmarks and show that
MD-AIRL outperforms previous methods in various settings.
I NTRODUCTION
Inverse reinforcement learning (IRL) is an algorithm of learning ground-truth rewards from demonstrations of an expert that acts optimally with respect to an unknown reward function.
Traditional
IRL approaches (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) solve the imitation problem based on iterative algorithms, alternating between the reward estimation process and a reinforcement learning (RL) algorithm.
Recent adversarial imitation learning (AIL) studies (Ho & Ermon, 2016; Fu et al., 2017) focus on formulating a statistical divergence minimization with a fine-tuned representation of the target expert probability distribution (Ghasemipour et al., 2020).
One method to approximate the distance between probability distributions is using Bregman divergences, a family of metric-like functions induced by a strongly convex function.
Jeon et al.
(2020) generalized the AIL framework to an optimization problem with respect to a Bregman divergence between policy distributions where the method shares on the essence of regularized Markov decision processes (Geist et al., 2019).
Through the lens of differential geometries, this implies that a policy distribution and a reward function can be associated with geometric constraints specified by a convex regularizer (Shima, 2007).
Apparently, the limitation of regularized IRL studies naturally comes from a geometric property in which minimizing a Bregman divergence does not guarantee unbiased progression due to the constraints of the underlying space (Butnariu & Resmerita, 2006).
The success of machine learning has been the key to learning such divergences with high precision.
When the demonstration size is sufficiently large, the discriminator approximation of AIL can model the entire expert policy distribution internally using nonlinear approximators such as neural networks (Finn et al., 2016; Fu et al., 2017).
However, there are challenging tasks that require long action sequences to be solved.
When states can only be rarely visited by imitation learning agents, the uncertainty of estimated expert policy often substantially affects the discrimination quality.
Therefore, the development of a robust IRL method that is tolerant to unreliable discriminative signals induced by imperfect demonstrations is necessary.
The reasonable question is how to learn with cost estimations that are not precisely determined throughout the learning process.
There have been considerable achievements in dealing with temporal costs in the online learning domain (Fiat & Woeginger, 1998; Hazan, 2019).
An online learning algorithm predicts a sequence of parameters { w t } Tt =1 ⊂ W that is incurred by a cost function F t ( · ) at each iteration.
The most ordinary approach in online learning is stochastic gradient descent (SGD): w t +1 =
w t − η t ∇ F t ( w t ) .
SGD is a desirable algorithm when the set W is the Euclidean geometry since it can provide unbiased progression for minimizing F t (Raskutti & Mukherjee, 2015).
However, policies appear in manifolds of probability distributions; a gradient may not be the direction of the steepest descent in this case due to geometric constraints (Amari, 1998; 2016).
In this paper, we propose an iterative imitation learning framework that interprets the RL-IRL scheme similar to proximal optimization methods (Boyd et al., 2004; Amari, 2016).
We identify two issues in the AIL studies that have characteristics of unconstrained updates: (1) a divergence does not guarantee informative global directions to match the expert policy due to the geometry, and (2) a representation of divergence often cannot be accurately obtained due to insufficient data.
Our method is motivated by a template of optimization algorithms specified by a convex function, called mirror descent (MD; Nemirovsky & Yudin 1983).
For parameters sequences, cost functions, and step sizes, an MD update for a strongly convex function Ω is derived as
[Equation]
where ∇ Ω is a bijective transformation that links the primal space and the dual space of gradients.
Different from these standard MD formulations, our methodology draws a sequence of functions on a space formulated by a regularized reward operator which is an alternative to the dual space.
Consequently, the reward functions are projected optimization targets for the space of policies.
Our Contributions.
We propose a novel IRL algorithm which facilitates the agent in robustly imitating the expert. Our work is complementary to previous regularized IRL studies; we introduce
NEW R1 , R a geometric perspective for optimizing rewards and derive solutions with theoretical guarantees.
• Instead of a monolithic estimation process of a global solution, we derive a sequence of reward functions that provides local optimization targets for the space of policies (Section 3).
• We prove that rewards derived by an MD algorithm guarantee convergent divergence minimization performance along with a rigorous regret bound (Section 4).
• We propose mirror descent adversarial inverse reinforcement learning (MD-AIRL), a novel IRL algorithm that can be easily implemented on top of the standard AIL framework (Section 5).
• We validate the outperforming performance of MD-AIRL on benchmarks with large discrete action spaces and continuous action spaces (Section 6).
P RELIMINARIES
Notation.
For finite sets X and Y , we define Y X as a set of functions from X to Y .
∆ X ( ∆ YX ) is a set of (conditional) probabilities over X (conditioned on Y ).
For a function f ∈ R S×A and a policy π ∈ ∆ SA , f s and π s denote shorthand notation of f ( s, · ) and π ( ·| s ) , respectively.
A Markov decision process (MDP) is defined as a tuple ( S , A , P, r, γ ) with the state space S , the action space A , the transition kernel P ∈ ∆ S×A S , the reward function r ∈ R S × A and the discount factor γ ∈ [0 , 1) .
A Bregman divergence with respect to a convex function Ω for a state s is defined as D Ω ( π s ∥ ˆ π s )
=
Ω( π s ) − Ω(ˆ π s ) − ⟨∇
Ω(ˆ π s ) , π s − ˆ π s ⟩ A .
We write Ω( π ) =
[Ω( π s )] s ∈S and similarly for ∇ Ω .
Regularized RL & IRL.
We consider the RL-IRL framework in regularized MDPs (Geist et al., 2019), where the policy is optimized along with a causal convex regularizer.
The objective is to find π which maximizes the expected discounted sum of rewards with a strongly convex regularizer Ω :
[Equation]
where the subscript π on the expectation denotes samples generated from the MDP and policy π .
Consider the convex conjugate of q s E = ∇ Ω( π s E ) : Ω ∗ ( q s E ) = max π s ∈ ∆
A ⟨ π s , q s E ⟩ A − Ω( π s ) .
Differentiating both sides with respect to q s E , the gradient of conjugate ∇ Ω ∗ maps q E to the unique optimal policy π E .
Let Π be a bounded, open set for ∆ SA ⊂ Π and R : Π → R S×A be a reward operator which maps a policy to a reward function.
The output ˆ r = R ( π E ) is a representation of the expert’s behavior; but finding such operator is an ill-posed problem because every function ˆ r that makes the state-action value function q E is a valid solution of regularized IRL (Geist et al., 2019).
Recently,
Jeon et al.
(2020) proposed a tractable solution, which we refer to the function as the regularized reward function.
We rewrite the previous notation by defining the regularized reward operator Ψ Ω .
Definition 1 (Regularized reward operators) .
Define the regularized reward operator Ψ Ω : π (cid:55)→ ψ π
[Equation]
for Ω ′ ( s, · ; π ) := ∇ Ω( π s ) =
[ ∇ p Ω( p )] p = π ( ·| s ) .
By using the operator in discriminative networks, Jeon et al.
proposed regularized adversarial
IRL (RAIRL) as a generalization of AIL in terms of minimizing a specific Bregman divergence.
Mirror Descent.
Let w be a parameter on a set W and F t : W → R be a convex cost function from a class of functions F .
Replacing the L2 proximity term of proximal gradient descent with the
Bregman divergence (Gutman & Pe˜na, 2018), the proximal form of the MD update is given as
[Equation]
(4) where α t := 1 / η t denotes an inverse of the step size η t .
MD was developed as a generalization of
SGD, where the local geometry is specified by a Bregman divergence (Gunasekar et al., 2020).
I TERATIVE RL-IRL AS A P ROXIMAL O PTIMIZATION M ETHOD
We interpret the RL-IRL framework as a variant of proximal optimization methods and consider two sequences { π t } ∞ t =1 and { ψ t } ∞ t =1 that denote the learning policies and reward functions, respectively.
Associated reward function.
An updated point in MD can be uniquely projected to the desired space ( ∆ SA in our case) using a Bregman projection operator P Ω that locates iterative points to the feasible region, i.e. π t +1 =
P Ω
(˜ π t +1 ) :
= argmin π ∈ ∆ SA [ D Ω ( π ∥ ˜
π t +1 )] s ∈S for ˜ π t +1 ∈ Π .
To avoid these computations, we preemptively constrain a reward operator to satisfy the projection invariance with respect to regularized RL process of the associated state-action value function ˆ q : P Ω ( ∇ Ω ∗ (ˆ q )) =
∇ Ω ∗ (ˆ q ) .
According to Lemma 1 of Jeon et al.
(2020), a regularized reward function ψ π can replace ˆ q , since the induced Bregman divergence allows the learning in a greedy manner.
FIX R The projection invariance of the operator Ψ Ω can be shown by a bijective relation with ∇ Ω ∗ in the space of ∆ SA by the following lemma.
Lemma 1 (Natural isomorphism) .
Let ψ ∈ Ψ Ω (∆ SA ) for Ψ Ω (Π) =
{ ψ | ψ ( s, a ) =
ψ π ( s, a ) , ∀ s ∈
S , a ∈A , π ∈ Π } .
Then, ∇ Ω ∗ ( ψ ) is unique with respect to ψ , and for every π = ∇ Ω ∗ ( ψ ) , π ∈ ∆ SA . [Figure] [Equation] [Figure]
[Equation]
[Figure]
The proof is in Appendix A. Figure 1 illustrates that there is unique ψ t for π t , hence the two functions are isomorphic objects.
Note that the transformed Ψ Ω ( π t ) is different from the form ∇ Ω( π t ) of the dual space that is used in classical MD algorithms.
The transformation is shifted by a vector 1 c with a constant c =
Ω ( π s t ) − ⟨ π s t , ∇ Ω( π s t ) ⟩ A for each s .
However, if the underlying space is ∆ SA , the operator ∇ Ω ∗ reconstructs the original point for both Ψ Ω and ∇ Ω , since the distributivity of Ω (Geist et al., 2019): Ω ∗ ( y + 1 c ) =
Ω ∗ ( y )+ c , holds (so ∇ Ω ∗ ( y + 1 c ) = ∇ Ω ∗ ( y ) ).
Consequently, we may omit the projection phase (the NEW R dotted angle in Figure 1) when the reward function satisfies the constraint of Ψ Ω (∆ SA ) .
As a result, we consider an updated reward function ψ t +1 ∈ Ψ Ω (∆ SA ) as a projected target of MD, FIX R which can be associated by parameterization of ∆ SA .
For instance, a bijective mapping of ψ t +1 can NEW R represent a softmax policy for a discrete space, or a Gaussian policy for a continuous space, using its parameters.
The subsequent RL process at t -th step becomes finding the next iteration π t +1 by
NEW R
FIX R3 maximize
[Equation]
The equation shows that an arbitrary regularized RL algorithm with the regularizer Ω forms a cumulative discounted sum of Bregman divergences, thus the mapping to π t +1 can be achieved.
[Figure]
entropy
Tsallis entropy
[Figure]
Online imitation learning.
The necessity of online learning setup comes from observing that our RL-IRL processes do not retain ψ E = Ψ Ω ( π E ) , a representation of π E , during training.
Instead, consider a random process { ¯ π E ,t } ∞ t =1 where the estimation ¯ π E ,t resides in a closed, convex neighborhood of π E , estimated by a separate estimation algorithm such as maximum likelihood methods.
Substituting ψ E to ψ ¯ π E,t in J Ω ( π, ψ E ) turns the RL objective into an online learning problem:
FIX R3 minimize
[Equation]
For a deeper understanding of our setup, we consider an example in Figure 2.
Assume that policies of the learning agent and the expert follow multivariate Gaussian distributions initially at π 1 =
N ([0 , 0] T , I) and π E = N ([5 , 3] T , Σ E ) for | Σ E | < 1 .
The policy π t is trained by a cost function
D Ω ( ·∥ ¯ π E ,t ) with the proximity constraint 1 η D Ω ( ·∥ π t ) where the Gaussian policy ¯ π E ,t is also fitted using stochastic samples from π E at each iteration, starting from ¯ π E , 1 = π 1 .
We first observe that the step size constant η affects the training speed in the early phase.
Notably, the performance of certain cases exceeds the baselines of D Ω (¯ π E ,t ∥ π E ) by choosing the step size effectively low as η < 1 .
This suggests that there is a clear advantage of the online imitation learning setup for unreliable ¯ π E ,t .
MD update rules.
Plugging each divergence of the cumulative cost F t to the template of Equation (4), the optimization process for the subsequent ψ t +1 of the MD formulation is derived as [Equation] [Table]
[Equation]
where the gradient of D Ω is taken with respect to its first argument π st .
The objective of reward NEW R learning is analogous to finding an interpolation at each iteration where the point is controlled by the step size η t .
Figure 3 shows that the region of π t (defined by a norm) gradually decreases when η t > η t +1 .
Note that solving the optimization of F t requires interaction between π t and the dynamics of the given environment; thus, the RL process in Equation (5) plays the essential role of sequential learning by the value measures of the reward function.
A NALYSES ON S TEP S IZES , C ONVERGENCE , AND R EGRETS Online IRL.
We define an online cost f t ( π t , τ t ) which involves a policy and a trajectory as inputs.
The trajectory τ t = { s i } ∞ i =0 is available at the t -th step by executing the policy π t in the environment.
An online cost function for reward ¯ ψ E ,t = Ψ Ω (¯ π E ,t ) can be expressed as
[Equation]
We refer the objective of the learning is to find a unique fixed point π ∗ ∈ Π that minimizes E [ f t ( π, τ )] , where the expectation is taken over the entire steps (i.e. lim t →∞ E τ 1 , ··· ,τ t [ f ( π t , τ t )] ). NEW R
Taking (stepwise) gradient for each π ( ·| s ) , π ∗ is found by E [ ∇ Ω( π ∗ ( ·| s ))
− ∇ Ω(¯ π E ,t ( ·| s ))]
=when t → ∞ , hence ∇ Ω( π ∗ ) =
lim t →∞ E [ ∇ Ω(¯ π E ,t ( ·| s ))] .
The assumption of π E = π ∗ allows the particular situation when the estimation algorithm of ¯ π E ,t is actually convergent with t → ∞ .
This also allows general situations where the estimated expert policy is not stationary; the algorithm finds the unique point according to the expectation of gradient by scheduling the step size
[Equation] [Figure]
sequence
Theoretical Analyses.
We state two conditions of { η t } ∞ t =1 to guarantee convergence properties which is explained in the following analyses.
• Convergent sequence & divergent series:
[Equation]
• Divergent series & convergent series of squared terms:
[Equation]
Under the satisfaction of both cases, a sequence that is divergent in its series (e.g. harmonic series (cid:80) ∞ n =1 1 n ) is suitable as in Figure 4.
In our arguments, a policy conditioned by a state is in a Banach space called the L p space ( R A , ∥·∥ p ) , where ∥·∥ p denotes a p -norm, where we assume 1 < p ≤for the convergence in the dual L q space ( 1 / p + 1 / q = 1 ).
The proofs are in Appendix A.
Theorem 1 (Stepsize considerations) .
Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences are bounded.
Assume inf π ∈ ∆ SA E [ f t ( π, τ )] > 0 .
Then lim
[Equation]
[-= 
-]0 if and only if Eq.
(9) is satisfied.
(a) If lim t →∞ η t = 0 , then T ∈ N , n < T , and c > 0 exist s.t.
E τ 1 , ···
[Equation]
τ T
[Equation]
f T ( π T
[Equation]
τ T ) (cid:3) ≥ [Equation] n
[Equation]
(b) If the step size takes the form η t = 4 t +1 , then E τ 1 , ··· ,τ T (cid:2)(cid:80) ∞ i =
γ i D Ω (cid:0) π
s i ∗ (cid:13)(cid:13)
π s iT (cid:1)(cid:3) =
O (cid:0) 1 /T ) .
Theorem 2 addresses the convergence of the algorithm in a specific case when π E can be achieved.
Theorem 2 (Convergence in optimal cases) .
Let Ω be strongly convex, ∇ Ω be Lipschitz continuous, and the associated Bregman divergences are bounded.
Assume π 1 ̸ = π E and inf π ∈ ∆ SA E [ f t ( π, τ )]
= 0 .
Then, lim t →∞ E τ 1 , ··· ,τ t (cid:2)
f t ( π t , τ t ) (cid:3) =
if and only if (cid:80) ∞ t =1 η t = ∞ .
If η t ≡ η 1 , then there exist c 1 , c 2 ∈ (0 , 1) such that c 1 · A 1 ≤ A T ≤ c 2 · A 1 , where { A t } ∞ t =1 denote a sequence of A t = FIX R3 E τ 1 ,
[Equation]
D
[Equation]
( π [Equation]
(cid:3) .
Proposition 1 provides a sufficient condition for the almost certain convergence of an MD algorithm by imposing the stronger condition of step size in Equation (10).
Proposition 1 (Convergence in general cases) .
Assume inf π ∈ ∆ SA E [ f t ( π, τ )] > 0 .
If the step size sequence satisfies Eq.
(10), then we have lim t →∞ (cid:80) ∞ i =
γ i D Ω ( π s i ∗ ∥
π s i t ) = 0 almost surely.
Regrets.
Define the regret at t -th iteration for a sequence of cost functions { f t } t ∈ N as
[Equation]
In the optimal case of inf π ∈ ∆ SA E [ f t ( π, τ t )] = 0 , R t is bounded by O (1 /T )
since f t inherits the property of Bregman divergence so that the infimum is achieved by 0 at π E .
By Proposition 1, the updates regarding the policy converge when the step size sequence abides by Equation (10).
Therefore, even for the general case of inf π ∈ ∆ SA E [ f t ( π, τ )] > 0 , R T is bounded to O (1 /T ) .
M IRROR D ESCENT A DVERSARIAL I NVERSE R EINFORCEMENT L EARNING
In this section, parameters θ , ϕ , and ν are presented representing agent policy, reward, and expert policy functions respectively parameterized with neural networks.
On top of the standard AIL algorithm, we propose the MD-AIRL method, learning with a dual discriminator architecture motivated by previous studies regarding multiple discriminators (Nguyen et al., 2017; Chongxuan et al., 2017).
AIL methods have dealt divergences between joint densities of states and actions (Ghasemipour et al., 2020).
To clearly distinguish matching overall state densities and imitating specific behavior, we aim to disentangle these concepts and propose two structured discriminators: [Equation] and D b
[Equation]
, where b : S → R is a neural network regarding states and σ ( · ) denotes the sigmoid function.
The discriminators D b and D θ,ν are trained with binary cross-entropy losses regarding trajectories:
[Equation]
Note that b and θ are not trained for learning D θ,ν .
Let ρ π ∈ ∆ S denote normalized state visitation distribution of π , which is defined as ρ π ( s ) :
= (1 − γ )
E π (cid:2)(cid:80) ∞
i =
γ i
I { s i = s } (cid:3) where I {·} is an indicator function.
Since x (cid:55)→ α log σ ( x ) + β log(1 − σ ( x )) attains its maximum at σ ( x )
=
αα + β (Goodfellow et al., 2014), optimality is achieved when π ν = π E and b ( s )
= log
ρ πE ( s ) /
ρ π ( s ) .
Let ψ ϕ denote the regularized reward function where the parameter ϕ is trained using a step size η t , the agent policy π θ , and the estimation of expert policy π ν :
[Equation]
14) where π ϕ denotes the transformed policy ∇ Ω( ψ ϕ ) and ¯ τ t denotes mini-batches of states using the both agent and expert trajectories.
The algorithm adjusts the term η t = 1 /α t with a harmonic progression by linearly increasing α t by the range of [ α 1 , α T ] , which is derived from our analyses.
For a hyperparameter λ > 0 , the MD-AIRL reward function is defined by a linear combination:
[Equation]
By using arbitrary regularized RL which uses λ Ω( · ) as the regularization function, the reward learning of ψ λϕ regarding the agent π θ is decomposed into the following two terms:
[Equation]
Minimizing the first term of E π θ (cid:2) D Ω ( π sθ ∥ π sϕ ) (cid:3) represents the online learning of MD formulation.
However, ψ ϕ that is trained on ¯ τ t cannot cover the entire reachable states since the state visitation is heavily misaligned in challenging sequential decision problems.
Therefore, we propose the second auxiliary term of generalized KL divergence D KL ( ρ π θ ∥ ρ π E ) , playing an additional role of facilitating the supports of state visitation densities to be matched properly.
Algorithm 1 summarizes the entire procedure of MD-AIRL.
See Appendix C for detailed implementation.
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
E XPERIMENTAL R ESULTS
For the RL algorithm, we implemented a RAC (Yang et al., 2019) method that is a generalization of SAC (Haarnoja et al., 2018) in terms of the regularization choice.
We considered the class of separable regularizers Ω( p ) =
− E a ∼ p
[ φ ( p ( a ))] : (1) Shannon entropy ( φ ( x ) = log( x ) ), (2) Tsallis entropy T q ( φ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) , q = 2 by default), (3) exp regularizer ( φ ( x ) =
e − e x ) , (4) cos regularizer ( φ ( x ) = cos( π 2 x ) ), and (5) sin regularizer ( φ ( x ) = [-
-]1 − sin π 2 x ).
We evaluated our approach on three topics (bandits, multiple goals, and MuJoCo environments).
The main comparative method was RAIRL with a density-based model (RAIRL-DBM) since this model shares the identical level of expressiveness as our method when the parameterization of ∆ SA is specified.
6. L ARGE -S CALE M ULTI - ARMED B ANDITS
We first considered multi-armed bandit problems, where the cardinality of action spaces is varied.
Learning the optimal distribution of π E becomes challenging as |A| increases, because the frequency of each sample becomes sparse due to the curse of dimensionality (Bellman et al., 1957).
For each experiment, a stateless expert distribution π E was generated by the parameters of softmax distribution π E ( i ) =
exp( z i ) / (cid:80) j exp( z j ) where the logits
z
i were randomly initialized to a uniform distribution.
We set the action size to |A| =
10 2 , 10 3 , 10 4 and restricted the sample size to 16 .
Figure 5 shows that the Bregman divergence was large for MD-AIRL at the early training phase, because we chose the initial step size η 1 to be greater than 1 ( α 1 = 0 . 5 ).
MD-AIRL exceeded the discriminative performance of RAIRL after certain steps, while the progression of RAIRL mostly stopped at local minima.
Table 1 shows that MD-AIRL achieved overall lower Bregman divergence on average when three different cardinalities and five regularizers were considered.
MD-AIRL outperformed RAIRL in four cases by choosing effectively low step size at the η T to be less than( α T = 2 ).
These results match properties of MD algorithms and our convergence analyses.
Therefore, we argue that a constrained update rule with appropriate step sizes is necessary for robust reward acquisition and imitation for the situations when the total number of data samples is limited.
6. C ONTINUOUS M ULTI -G OAL E NVIRONMENT
We then considered an environment with a two-dimensional continuous state space.
In this envioronment, an agent is a point mass initialized at the origin, and the four goals are located in the four cardinal directions.
To draw meaningful reward surface, we considered multivariate Gaussian dis-
[Figure]
tribution policies with full covariance matrices.
We parameterized a covariance matrix of Gaussian policy using lower triangular matrix which is an outcome of LDL decomposition (see Appendix C).
Figure 6 (a) shows trajectories generated by the trained agent.
Figure 6 (b) shows that MD-AIRL achieved higher entropy for reaching the goals.
Figure 6 (c) shows reward surfaces with regularizers, which was calculated by ψ ϕ ( s, a )
+ φ ( π θ ( a | s )) for each point of a ∈ A and s = (5 , − 1) .
During the training, the MD reward was similar to the estimated ground truth using adversarial training.
However, the surface of MD-AIRL became flatter than the ground-truth estimation when π t was sufficiently close to the expert behavior.
As a result, we claim that drastic changes in the target distribution, which are one of the typical characteristics of adversarial frameworks, are prevented.
We argue that these characteristics mitigate overfitting caused by unreliable discriminatve signals.
[Figure]
6. C ONTINUOUS C ONTROL : M U J O C O
We validated MD-AIRL on MuJoCo continuous control tasks (Brockman et al., 2016).
We assumed diagonal Gaussian policies for both learner’s policy π and expert policy π E .
Instead of the tanh -squashed policy (Haarnoja et al., 2018), we used the hyperbolized environment assumption of
RAIRL
[Figure]
which means that tanh is regarded as a part of the environment.
For each tasks
[Figure]
we considered two different numbers of episodes collected by an expert policy.
In
Figure [Figure] the performance of MD-AIRL, RAIRL
[Figure]
and behavior cloning (bc; Pomerleau, 1991) al-
[Figure]
gorithms are shown with the expert and random agent performance. In terms of sample efficiency,
MD-AIRL outperformed RAIRL on Hopper-v3 , Walker-v3 , and Ant-v3 .
The performance gaps between MD-AIRL and RAIRL were more prominent in 4 episodes of expert demonstrations.
The training curves in Figure 8 indicate that MD-AIRL showed lower variance than RAIRL especially after early phase of training.
It can be concluded that MD-AIRL inherits the sample efficiency of AIL algorithms in challenging RL tasks benchmarks.
Additionally, the algorithm is highly stable with respect to limited sample sizes, which is in alignment with our theoretical analyses.
R ELATED W ORKS
Statistical manifolds.
The Hessian of strongly convex function forms a metric tenser of a manifold called Hessian geometries (Shima, 2007).
Bregman divergences are similar to these metrics, providing useful metric-like properties (Butnariu & Resmerita, 2006).
Using the Bregman divergence allows to solve many optimization problems, generalizing traditional approaches such as least squres (Boyd et al., 2004; Hiriart-Urruty & Lemar ´ echal, 2004).
Probability distributions can be considered as points in a geometric space.
A representative statistical manifold is the information geometry induced by the Fisher-Rao metric (Amari, 2016; Bauer et al., 2016; Nielsen, 2020).
Regularized IRL.
Energy-based policies (i.e. Boltzmann distributions) have been appeared in early IRL researches (Ramachandran & Amir, 2007; Neu & Szepesv ´ ari, 2007; Babes¸-Vroman et al., 2011).
Notably, MaxEnt IRL (Ziebart et al., 2008; 2010) is a representative IRL algorithm based on information theory.
Other statistical entropies have also been applied to the imitation learning problem, such as the Tsallis entropy derived from Tsallis statistics (Lee et al., 2018). Compared to
RAIRL which also can use various convex regularizers (Jeon et al., 2020), our work allows more
NEW R1 , R realistic situations where the expert policy cannot be precisely attained due to insufficient data, thanks to theoretical foundations originated from optimization studies.
Mirror descent.
MD is closely related to algorithms regarding non-Euclidean geometries with discretization of steps such as natural gradients (Amari, 1998; Raskutti & Mukherjee, 2015; Gunasekar et al., 2020).
On the primal space, the infinitesimal limit of MD step corresponds to a
Riemannian gradient flow (Do Carmo, 2016; Gunasekar et al., 2020).
The online MD algorithms possess rigorous regret bounds (Srebro et al., 2011; Lei & Zhou, 2020); thus they can be highly efficient in terms of the number of evaluations until convergence.
D ISCUSSION AND C ONCLUSIONS
In this paper, we presented a novel IRL framework.
We provided an mirror descent solution on reward functions and corresponding theoretical arguments.
We proposed MD-AIRL, a practical FIX R adversarial IRL framework that can solve challenging imitation learning tasks.
We verified that the proposed method has clear advantages over previous AIL methods in terms of robustness.
As the reward hypothesis is grounded in obtainining a robust representation of the expert policy using IRL, we argued that current IRL studies lack robustness and theoretical guarantees for practical situations.
Considering RL and its inverse problem with geometric perspectives is vital for achieving desired goals in realistic situations.
Although our work covers various online imitation learning methods with MD, it does not include some other cases when the proximity term is of other statistical divergence families such as f-divergence (Amari, 2016).
Additionally, the boundedness and continuity assumptions on Bregman divergences in our analyses are usually justified, but outliers exist such as NEW R
KL divergences. More sophisticated analyses on these general cases remain as future works.
A P ROOFS
We define the space of policy as an object ∆ SA , which is a vector space formed by a collection of |S| elements of unit ( |A| − 1) -simplexes: ∆ A = (cid:8) x 1 e 1 + · · · + x |A| e |A| | (cid:80) |A| i =1 θ i =
and θ i ≥ 0 for
i ∈ A (cid:9) .
We assume that each space of simplex is a subset of a specific Banach space called
L p space ( R A , ∥·∥ ) , where ∥·∥ is a p -norm on A .
The dual space of L p space for 1 < p < ∞ is L q space ( R A , ∥·∥ ∗ ) , where ∥·∥ ∗ is defined as a q -norm ( 1 / p + 1 / q = 1 ).
We assume 1 < p ≤ 2 for the convergence property in the dual L q space.
We start with following definitions.
Definition 2 (Lipschitz constants) .
Given two metric spaces ( X, d X ) and ( Y, d Y ) where d X denotes the metric on set X and d Y is the metric on set Y , a function f : X → Y is called Lipschitz continuous if there exists a real constant k ≥ 0 such that, for all x 1 and x 2 in X ,
[Equation]
In particular, a function f is called Lipschitz continuous if there exists a constant k ≥ 0 such that,
[Equation]
) where norms ∥·∥ and ∥·∥ ∗ are endowed with each space X and Y respectively.
For the smallest
L that substitutes k , L is called the Lipschitz constant and f is called the L -Lipschitz continuous function.
Definition 3 (Discrete-time martingale) .
If a stochastic process { Z t } t ≥ 1 satisfies E [ | Z n | ] ≤ ∞ and 1) E [ Z [Equation] [Equation]
[Equation]
n ]
[Equation]
then, the stochastic process { Z t } t ≥ 1 is called a 1) submartingale, 2) martingale, and 3) supermartingale, respectively, with respect to a filtration { X t } t ≥ 1 .
The following analyses and proofs follow the results appeared in previous literatures for general aspects (Nemirovsky & Yudin, 1983; Gunasekar et al., 2020;
Srebro et al., 2011; Lei & Zhou, 2020;
Beck & Teboulle, 2003; Raskutti & Mukherjee, 2015).
Our analyses extend existing results to imitation learning, and they are also highly general to cover various online methods for sequential decision problems.
A. P ROOF OF L EMMA
Proof of Lemma 1.
The conjugate operator of ψ sπ satisfies the following identity (Lemma 1 of Jeon et al.
[Equation]
for every state s ∈ S .
By the property of Bregman divergence, and the convexity of D Ω (˜ π s ∥ π s ) with respect to ˜ π s (Geist et al., 2019; Acharyya et al., 2013), the optimal condition is obtained by the unique maximizing argument ˜ π ( ·| s ) =
π ( ·| s ) .
By taking gradient to both sides with respect to ψ sπ we yield π = ∇ Ω ∗ ( ψ π ) .
If there is another ˜ π ̸ = π that makes ψ ˜ π = ψ π , this contradicts the property of unique maximizing arguments for conjugates.
Therefore, ψ π is uniquely defined for each π and ∇ Ω ∗ ( ψ ) ∈ ∆ SA .
We model the reward approximator regarding actions as ψ ∈ Ψ Ω (∆ SA ) =
{ ψ | ψ ( s, a ) =
ψ π ( s, a ) , ∀ s ∈ S , a ∈ A , π ∈ ∆ SA } , where ψ π is defined ψ π ( s, a ) :
=
Ω ′ ( s, a ; π ) −
E a ′ ∼ π ( ·| s ) (cid:2) Ω ′ ( s, a ′ ; π ) (cid:3) + Ω( π ( ·| s )) .
Note that Ψ Ω ( π ) is different from ∇ Ω( π ) , but it is shifted by the amount of − E a ′ ∼ π ( ·| s ) (cid:2) Ω ′ ( s, a ′ ; π ) (cid:3) + Ω( π ) for every element of ψ π ( s, · ) .
Therefore, the operators Ψ Ω and ∇ Ω are link functions that form natural isomorphisms of functions.
A. P ROOF OF T HEOREM
We consider the unique fixed point of π ∗ as the solution of inf π ∈ ∆ SA lim t →∞ E [ f t ( π, τ t )] where the expectation indicates that we take consideration for all τ t and f t .
By equating derivatives to zero, we write ∇ Ω( π ∗ ) =
lim t →∞ E [ ∇ Ω(¯ π E ,t )] ).
We provide more general results than previous imitation learning, as some of following statements include the case of inf π ∈ ∆ SA E [ f t ( π, τ )] > 0 , which means the estimates { ¯ π E ,t } ∞ t =1 do not converge to the fixed point of π ∗ , so lim t →∞ ∥ π ∗ − ¯ π E ,t ∥ > 0 .
Thus, MD-AIRL and other online MD algorithms allow realistic settings such as scarcity of data or imperfect demonstrations.
We first introduce a key relationship regarding cumulative gradients in our online MD setting.
Lemma 2.
Let { π t } ∞ t =1 , { ¯ π E ,t } ∞ t =1 , and { η t } ∞ t
=1 be policy, estimate, and step size sequences, respectively.
The subsequent policy π t +1 in Equation (5) is obtained by an RL algorithm using the derivation of ψ t +1 in Equation (7), resulting to the following equation:
[Equation]
We have for t ∈ N ,
[Equation]
18) where we write ∇ [Equation] = [Equation]
∈S
Proof of Lemma 2.
Since the optimization is convex with respect to each π s , we equate the derivatives to 0 at π t +1 as
[Equation]
Finally, we derive Equation (18) as
[Equation]
Therefore, the proof is complete.
Lemma 2 indicates that the distances between dual maps are equivalent to η t ∥∇ Ω(¯ π s E ,t ) −∇ Ω( π st ) ∥ ∗
Therefore, if the step size decreases as lim t →∞ η t = 0 , lim t →∞ ∥∇ Ω( π t ) − ∇ Ω( π t +1 ) ∥ ∗ = 0 ; thus, our argument in Section 3 is reasonable when Ω is strongly smooth.
Next, we reintroduce the three-point identity as follows.
Lemma 3 (Three-point identity) .
Let π a , π b , and π c be any policies with a given state.
We have the following identity:
[Equation]
Proof of Lemma 3.
This can be derived using the definition of divergence as follows.
[Equation]
Therefore, the proof is complete.
We now introduce basic identities regarding a Bregman divergence Lemmas 4 and 5 that are used to address progress of the algorithm.
Lemma 4.
Let π a , π b , and π c be any policies with a given state.
The following identity holds.
[Equation]
Proof of Lemma 4.
By Lemma 3, we have
[Equation]
Utilizing the identity of two Bregman divergences
[Equation]
we separate π c − π b into π c − π a and π a − π b and write the rest of derivation as followings.
[Equation]
Therefore
[Equation]
we obtain the desired identity.
Lemma 5.
Let π a , π b [Equation] and π c be any policies with a given state.
The following identity holds.
[Equation]
Proof of Lemma 5.
By Lemma 3, we have
[Equation]
We separate ∇ Ω( π a ) − ∇ Ω( π b ) into ∇ Ω( π a ) − ∇ Ω( π c ) and ∇ Ω( π c ) − ∇ Ω( π b ) and write the rest of derivation as followings.
[Equation]
Therefore, we obtain the desired identity.
We show the key arguments to prove Theorem 1 in the following lemma.
Lemma 6.
Assume inf π ∈ ∆ SA E [ f t ( π, τ )] > 0 .
Assume also that Ω is ω -strongly convex and ∇ Ω is
L -Lipschitz continuous for ω ≥ 0 and L ≥ 0 .
If lim t →∞ E {+τ 1 , ··· ,τ t (cid:2)(cid:80) ∞ i =
+}
γ i D Ω (cid:0)
π s i t (cid:13)(cid:13)
π s iE (cid:1)(cid:3) = 0 for π E , then { η t } ∞ t =1 satisfies Equation (9).
Furthermore, if Ω is strongly smooth, then Theorem 1 (a) holds with some constants n ∈ N and c > 0 .
Proof of Lemma 6.
First, we show the condition of lim t →∞ η t = 0 .
Assuming all states are reachable, the condition lim t →∞ E τ 1 , ··· ,τ t (cid:2)(cid:80) ∞ i =
γ i D Ω (cid:0)
π s i t (cid:13)(cid:13)
π s iE (cid:1)(cid:3) = 0 , implies lim t →∞ E τ 1 , ··· ,τ t (cid:2) ∥ π t − π E ∥ (cid:3) = 0 , where ∥·∥ is a matrix norm induced by the p -norm on A .
Then, our aim is to show that the gradient of strong convex function for π t converges to ∇ Ω( π E ) as
[Equation]
To prove this point, we use the continuity of ∇ Ω at π E ; for any ε > 0 , there exists some 0 < δ ≤such that ∥∇ Ω( π ) − ∇ Ω( π E ) ∥ ∗ < ε whenever ∥ π − π E ∥ < δ .
When ∥ π − π E ∥ ≥ δ , we apply the L -Lipschitz continuity assumption to find
[Equation]
(23) where ∥·∥ ∗ is a matrix norm induced by the q -norm.
Combining Equation (22) and Equation (23), we know that
[Equation]
However, lim t →∞ E τ 1 , ··· ,τ t (cid:2) ∥ π E − π t ∥ (cid:3) = 0 ensures the existence of some n ∈ N such that for t > n , there holds E τ 1 , ··· ,τ t (cid:2) ∥ π E − π t ∥ (cid:3) <
ε / L .
Applying this inequality to Equation (24), we have
E τ 1 , ··· ,τ t (cid:2) ∥∇ Ω( π t ) − ∇ Ω( π E ) ∥ ∗ (cid:3) < 2 ε for some t > n .
For temporal estimation, let us denote the infimum of the expectation ℓ = inf π ∈ ∆ SA E [ ∇ Ω( π t ) − ∇ Ω(¯ π E ,t )] > 0 .
From Lemma 2, we have η t ( ∇ Ω( π t ) −∇ Ω(¯ π E ,t ))
= ∇ Ω( π t ) −∇ Ω( π t +1 ) .
Taking the expectations on both sides of this equation yields
[Equation]
Hence the convergence of the point ∇ Ω( π t ) is confirmed by taking the limit: lim t →∞ η t = 0 .
Next, we show (cid:80) ∞ t =1 η t = ∞ .
By the ω -strong convexity by the L -Lipschitz continuity of Ω , we can find inequalities as
[Equation]
We note that ∥ π t +1 − ¯ π E ,t ∥ ≤ ∥ π t − ¯ π E ,t ∥ so that there is a constant ε that satisfies E [ ∥ π t +1 − ¯ π E ,t +1 ∥ ] ≥ E [ ∥ π t +1 − ¯ π E ,t ∥ ]+ ε .
Therefore, taking expectations in Equation (21) (setting π a = ¯ π E ,t , π b = π t +1 , and π c = π t ), for lim t →∞ η t = 0 and the strongly convex Ω , we can find
[Equation]
for some t and 0 < ε ′ < ε ′′ .
The positive constant a = 2 L / ω is derived by the inequalities in
Equation (25).
We omit the conditioned states (superscripts) for simplicity of the derivation.
Since lim t →∞ η t =
[Equation]
we can also find a constant n ∈ N such that η t ≤ (3 a ) − 1 for t ≥ n .
Applying the inequality 1 − x > exp( − 2 x ) for x ∈ (0 , 1 / 3] [Equation] we derive another inequality
[Equation]
Applying this for t = T − [Equation]
[Equation]
n yields
[Equation]
Using Equation (28), we conclude E τ 1 , ··· ,τ n (cid:2) D Ω ( π n +1 ∥ ¯ π E ,n +1 ) (cid:3) > 0 .
Otherwise, we have
[Equation]
according to Equation (27), which leads to E τ 1 , ··· ,τ n (cid:2) ∥ π n − ¯ π E ,n ∥ 2 (cid:3) =
E τ 1 , ··· ,τ n +1 (cid:2) ∥ π n +1 − ¯ π E ,n +1 ∥ 2 (cid:3) = 0 .
This implies π n = ¯ π E ,n = π n +1 almost surely, leading to E [ f t ( π t , τ t )]
= 0 .
This is a contradiction to the previous assumption inf π ∈ ∆ SA E [ f t ( π, τ t )] > 0 , thus E τ 1 , ··· ,τ n +1 (cid:2) D Ω ( π n +1 ∥ ¯ π E ,n +1 ) (cid:3) > 0 .
Let us suppose the ideal case that the estimation process learns the exact π E in t → ∞ .
In order to satisfy the limit lim T →∞ E [Equation] T (cid:2) D Ω ( π T ∥ ¯ π E , T ) (cid:3) {+= 
+}
[-= 
-]
we see from Equation (28) that (cid:80) ∞
t =1 η t = ∞ .
Now, we show that Theorem 1 (a) holds.
Since Ω is ω -strongly convex, so Ω ∗ is ( ω − 1 ) -strongly smooth with respect to ∥·∥ ∗ .
Additionally, the L -Lipschitz continuity of ∇ Ω implies L -strong smoothness of Ω ; thus, naturally, Ω ∗ is L -strongly convex.
Combining these, for ∀ t ≥ n , the condition η t ≤ (3 a ) − 1 induces
[Equation]
and by Lemma 2, we get
[Equation]
Using the Cauchy-Schwarz inequality
[Equation]
we obtain a lower bound of the last term as
[Equation]
Thus
[Equation]
we obtain the final inequality as
[Equation]
Applying this inequality from t =
T ≥ n + 1 to t = n +
[Equation]
we obtain
[Equation]
By the Cauchy-Schwarz inequality and our bound 0 < 1 − aη k ≤ 1 for k ≥ n
[Equation]
we have
[Equation]
Hence
T
[Equation]
Therefore, we obtain the lower bound of
[Equation]
Since the Bregman divergence is bounded for all states, the sequence { γ i D Ω ( π s i t ∥
π s i t ) } will converge as i → ∞ .
Applying the monotone convergence theorem, we can interchange expectation and summation, which yields
[Equation]
This verifies Theorem 1 (a) with the constant c = η 2 n +1 (2 L − 2 Lγ ) − 1 ℓ 2 .
Last, we show convergence to a unique fixed point of π ∗ using the particular form of η t in Equation (9).
Lemma 7.
If { η t } ∞ t =1 satisfies Eq.
(9), lim t →∞ E τ 1 , ··· ,τ t (cid:2)(cid:80) ∞ i =
γ i D Ω ( π s ∗ ∥ π st ) (cid:3)
= 0 .
Furthermore, if the step size takes the form η t = 4 t +1 , then E τ 1 , ··· ,τ T (cid:2)(cid:80) ∞ i =
γ i D Ω (cid:0)
π s i ∗ (cid:13)(cid:13)
π s iT (cid:1)(cid:3) =
O (cid:0) 1 /T ) .
Proof of Lemma 7.
According to Lemma 4 and the Bregman divergence of the conjugate D Ω ∗ , the one step progress regarding ¯ π E ,t can be written as
[Equation]
where we omit a given state.
As ω -strong convexity of Ω implies the ( ω − 1 )
-strong smoothness of Ω ∗ , we have
[Equation]
We bound ∥∇ Ω
[Equation]
+2 ∥∇ Ω( π ∗ ) −∇ Ω(¯ π E ,t ) ∥ 2 ∗ , following the work of Lei & Zhou (2020).
Since ∇ Ω is cocoercive with 1 L bye the Lipschitz continuity, we obtain
[Equation]
thus
[Equation]
By taking expectation, it follows that there exists n ∈ N such that η t ≤ ω 2 L for t ≥ n holds
[Equation]
[Equation]
where z is the constant z = 1 ω E [ ∥∇ Ω( π ∗ ) − Ω(¯ π E ,t ) ∥ 2 ∗ ] .
Let { A t } ∞ t =1 denote a sequence of A t =
E τ 1 , ··· ,τ t (cid:2) D Ω ( π ∗ ∥ π t ) (cid:3) .
Then we have
[Equation]
[Equation] For a constant h > 0 , we claim that A t 1 < h for some t 1 > n ′ .
Assume that this is not true, and we find some t 2 ≥ t 1 such that A t > h , ∀ t ≥ t 2 .
Since lim t →∞ η t = 0 , there are some t > t 3 > tthat η t ≤ h 4 b .
However, Equation (32) tells us that for t ≥ t 3 ,
[Equation]
This is a contradiction, which verifies A t < h for t > n ′ .
Since lim t →∞ η t = 0 , we can find some η t that makes A t to be monotone decreasing.
Then, we can conclude that the nonnegative sequence { A t } ∞ t =1 converges by iteratively applying the upper bounds.
We now prove Theorem 1 (b) under the condition and the choice η t = 4 t +1 of the step size sequence.
The estimate becomes
[Equation]
It follows the recurrence relation as
[Equation]
Applying this relation iteratively
[Equation]
we obtain the general form of inequality.
[Equation]
therefore we get the inequality as follows:
[Equation]
By applying the monotone convergence theorem similar to Lemma
[Equation]
we conclude that E
[Equation]
A. P ROOF OF T HEOREM
Necessity.
We rewrite the inequality in Equation (26) as
[Equation]
Since we assume that η t converge to 0 from previous arguments
[Equation]
onsider the step size sequence FIX R [Equation] ω (2+ κ ) L for κ > 0 and t ≥ n where n ∈ N .
Denote a constant ˜ a = 2+ κ 2 log 2+ κκ and apply the elementary inequality
[Equation]
From Equation (33)
[Equation]
it can be obtained that
[Equation]
Applying this inequality iteratively for t = n
[Equation] [Equation]
T −
[Equation]
then gives
[Equation]
t (cid:41) D Ω ( π n ∥ ¯ π E ,n ) .
From the assumption π E ̸ = π n , we have D Ω ( π n ∥ ¯ π E ,n ) > 0 .
The convergence lim t →∞ E τ 1 , ··· ,τ t (cid:2) D Ω ( π t ∥ ¯ π E ,t ) (cid:3) = 0 then implies (cid:80) ∞ t =1 η t = ∞ .
Sufficiency.
Here we use the estimate (32) derived in the proof of Lemma 7.
However, in the optimal case, z = 1 ω E [ f t ( π ∗ , τ )]
= 0 , so (32) takes the form (we can choose n = 1 by Equation (31) [Equation]
[Equation]
This implies that for any 0 < h , there must exist some integer t 1 ∈ N such that A t ≤ γ for t ≥ t 1 .
Otherwise, A t > γ for every t ≥ t 2 with t 2 ≥ t 1 , which leads to a contradiction:
[Equation]
Equation (34) also tells us that the sequence { A t } ∞ t =1 of nonnegative numbers is monotone decreasing.
Hence A t ≤ h for every t ≥ t 1 , which proves the convergence of A t
[Equation]
We now prove the second point in Theorem 2 which is under the special choice of constant step size sequence η t ≡ η 1 .
It follows from Equation (33) that A T ≥ (1 − 2 Lω − 1 η 1 ) T − 1 A 1 .
Hence, Eq (34) translates to
A
[Equation]
≤ [Equation]
A t , from which we find A T ≤ (1 − η 1 / 2) T − 1 A 1 by iteration starting from t = 1 .
This verifies the theorem with c 1 = (1 − 2 Lη 1 ω ) and c 2 = (1 − η 1 2 ) .
A. P ROOF OF P ROPOSITION
The proof is based on Doob’s forward convergence theorem.
Theorem 3 (Doob’s forward convergence theorem) .
Let { X t } t ∈
N be a sequence of nonnegative random variables and let {F t } t ∈ N be a filtration with F t ⊂ F t +1 for every t ∈ N .
Assume that E (cid:2)
X t +1 |F t (cid:3) ≤ X t almost surely for every t ∈ N .
Then the sequence { X t } converges to a nonnegative random variable X ∞ almost surely.
We follow the proof of Lemma 7 and apply Eq.
Since ⟨ π ∗ − π t , ∇ Ω( π ∗ ) − ∇ Ω( π t ) ⟩ ≥ 0 ,
Equation (30) implies
[Equation]
The condition (cid:80) ∞ t =1 η 2 t < ∞ enables us to define a stochastic process { X t } by
[Equation]
By (35), we know that E τ t [ X t +1 ] ≤ X t for t ≥ n .
Additionally, X t ≥ 0 .
Therefore, the stochastic process { X t } t − n +1 ≥ 1 is a submartingale (equivalently, {− X t } t − n +1 ≥ 1 is a supermartingale).
Then by Lemma 3, the sequence { X t } t ≥ 1 converges to a nonnegative random variable X ∞ almost surely.
According to Fatou’s Lemma and the convergence lim t →∞ E (cid:2)(cid:80) ∞ i [-=
 γ i-]=
[Equation]
γ
[Equation]
[-π t ) (cid:3) = 
 proved by Lemma 7, we obtain-]D Ω
[Equation]
π ∗ ∥
π
[Equation]
) (cid:3) = 0 for any states a proved by Lemma 7, we obtain NEW
[Equation]
{+It follows that the sequence of discounted sum (cid:8)(cid:80) ∞ i =
+}
γ i D Ω
( π ∗ ( ·| s i ) ∥
π t ( ·| s i )) (cid:9) t ∈ N converges toalmost surely.
B T SALLIS E NTROPY AND A SSOCIATED B REGMAN D IVERGENCE AMONG F ULL C OVARIANCE M ULTIVARIATE G AUSSIAN D ISTRIBUTIONS
In this section, we briefly reintroduce the tractable method to derive Bregman divergences and regularized reward functions proposed by Nielsen & Nock (2011) and Jeon et al.
Then, we identify specific paremeterization to model Gaussian distributions with full covariance matrices.
The standard form of the exponential family is represented as
[Equation]
The parameterization is as follows:
[Equation]
where we can analytically recover the multivariate Gaussian distribution (Nielsen & Nock, 2011)
[Equation]
[Equation]
For two distributions π and ˆ π with k ( x ) = 0 , Nielsen & Nock (2011) proposed the function I ( · ) :
[Equation]
where the detailed derivation is as follows:
[Equation]
B. T SALLIS ENTROPY OF FULL COVARIANCE G AUSSIAN DISTRIBUTIONS
For φ ( x ; q ) = 1 q − 1 ( x q − 1 − 1) , the Tsallis entropy can be written as
[Equation]
If π is a multivariate Gaussian distribution, we have
[Equation]
Since the covariance matrix is positive-definite, Cholesky decomposition can be applied, which separates the matrix to lower- and upper-triangle matrices.
Likewise, we can apply LDL decomposition.
Let us factorize the covariance Σ
= L diag { σ 21 , . . .
, σ 2 d } L T where L denotes a unit lower triangular matrix produced by LDL decomposition.
Then we have
[Equation]
B. T RACTABLE F ORM OF ψ π
For separable Ω , ψ π is written as (Jeon et al., 2020)
[Equation]
where φ ( x ) =
kq − 1 (1 − x q − 1 ) and accordingly f ( x ) =
xφ ( x ) .
For the gradient of f ( · ) , we have
[Equation]
Taking the expectation yields Tsallis entropy as follows.
[Equation]
For a multivariate Gaussian distribution π , the tractable form of E x ∼ π (cid:2) − f ′ ( x ) + φ ( x ) (cid:3) can be derived by using that of Tsallis entropy T kq ( π ) of π .
Thus ψ π can be rewritten as
[Equation]
In the special case of q = 1 and k = 1 , we have ψ π ( s, a ) = log π ( a | s ) .
B. B REGMAN D IVERGENCE WITH T SALLIS E NTROPY R EGULARIZATION
We consider the following form of the Bregman divergence:
[Equation]
For ω ( x ) = kq − 1 (1 − x q − 1 ) , f ′ ( x ) = kq − 1 (1 − qx q − 1 ) = qω ( x ) − k
[Equation]
and k = 1 , the above form is equal to
[Equation]
Let us define two multivariate Gaussians as follows:
[Equation]
where L and ˆ L denote unit lower triangular matrix.
We have
[Equation]
where
[Equation]
We can replace some difficult computations using LDL decomposition such as Σ −=
L − 1 diag(1 /σ )
( L − 1 ) T and | Σ | = (cid:80) di
=1 ln σ i .
C I MPLEMENTATION D ETAILS
Unnormalized rewards of the IRL algorithm often mislead the agent to take unnecessary awareness of termination in finite-horizon MDPs (Kostrikov et al., 2018).
To solve the issue IRL algorithms need to remove the difference between regarding steps depending on MDP’s time.
Doob’s optimal stopping theorem formally states that the expected value of a martingale at a stopping time is equal to its initial expectation.
Assume a martingale makes the entire procedure as a fair game on average, which means nothing can be gained by stopping the play.
Theorem 4 (Doob’s optimal stopping theorem) .
Let a process { X t } ∞ t =1 be a martingale and τ be a stopping time with respect to filtration {F t } t ≥ 1 .
Assume that one of the conditions holds:
(a) τ is almost surely bounded, i
e., there exists a constant c ∈ N such that τ ≤ c .
(b) τ has finite expectation and the conditional expectations of the absolute value of the martingale increments almost surely bounded, more precisely, E [ τ ] < ∞ and there exists a constant c such that E (cid:2) | X t +1 − X t | (cid:12)(cid:12) F t (cid:3)
≤ c almost surely on the event { τ > t } for all t ≥ 0 .
(c) There exists a constant c such that | X min { t,τ } |
≤ c almost surely for all t ≥ 0 .
Then X τ is an almost surely well-defined random variable and E
[ X τ ] =
E [ X 0 ] .
Then X ∞ is integrable and E [ X ∞ ]
=
E [ X 0 ]
Doob’s optimal stopping theorem states one of the necessary conditions of IRL reward of normalizing the reward measures and making them as a martingale even for finite-horizon benchmarks.
C. N ETWORK A RCHITECTURES
For all networks, we use networks with 2-layer MLP with 100 hidden units, respectively.
We consider the reward model with two separate neural networks ( ψ ϕ , b ( · )) for the proposed reward function for λ > 0 :
[Equation]
Motivated by RAIRL-DBM, we consider the reward models in Figure 9.
The model outputs reward for proximal updates trained by mirror descent and state-only discriminator network.
Discriminating state visitation by b ( · ) is required because the reward function needs to consider every state (especially the state that cannot be visited by π E ) until D KL ( ρ π ∥ ρ π E ) ≈ 0 .
Figure 9 (a) shows logits of the softmax distribution involved when calculating rewards when the action space is discrete.
For continuous control (Figure 9 (b)), the architecture is similar, where the mean and covariance are used to compute a reward for a certain action.
[Equation]
[Figure]
Let the 2D coordinate denote the position of a point mass on the environment.
In the multi-goal environment, the agent is initially located according to the normal distribution N ( 0 , (0 . 1) 2 I) .
The four goals are located at (6 , 0) , ( − 6 , 0) , (0 , 6) , and (0 , − 6) , where the agent can move a maximum of
1 unit per timestep for each coordinate.
The ground-truth reward is given by the difference between successive values of a Gaussian mixture depicted as Figure 10.
We used the full covariance Gaussian distribution in this experiment (as well as the conceptual experiment in Figure 2.
Note that the covariance matrix is positive-definite and symmetric.
To achieve numerically stable computation, we applied LDL decomposition to covariance matrix involving unit lower- and upper-triangle matrices, and a diagonal matrix.
As a result, the policy network outputs a vector ( µ ( s ) T , σ ( s ) T , l ( s ) T ) T where the additional part l ( s ) denotes d ( d − 1)entries of unit lower triangular matrix.
Denote L ( s ) as a unit lower triangular matrix from l ( s ) .
For example, the covariance matrix can be reconstructed by
[Equation]
Additionally, the action samples can be calculated by
[Equation]
Computing inverses, determinants and multiplications with unit triangular matrices and diagonal matrices can be efficiently performed by numerical libraries.
Therefore, we can fully model the
Bregman divergence and reward using neural networks as provided in Appendix B.
C. M U J O C O E XPERIMENTS
Instead of directly using squashed policies proposed in SAC (Haarnoja et al., 2018), we assume the application of tanh as a part of the environment (known as hyperbolized environments of
RAIRL (Jeon et al., 2020)).
Specifically, after an action a is sampled from the policies, we pass tanh ( a/ 1 . 01) ∗ 1 .
01 to the environment.
We additionally clip the hyperbolized actions to 1 , if the environment is not tolerant to the excessive values of action.
Therefore, we can consider the standard diagonal Gaussian policy
[Equation]
where µ ( s ) denotes means [ µ 1 ( s ) , µ 1 ( s ) , . . . , µ d ( s )] T and σ ( s ) denotes standard variance [ σ 1 ( s ) , σ 2 ( s ) , . .
σ d ( s )] T .
We soft-clip the standard deviation as σ i ( s ) ∈ [ln 0 .
01 , ln 2] for the stability using tanh .
Last, we update the moving mean of intermediate values of regularized reward r ( s, a ) + λφ ( π ( a | s )) and update the RL algorithm with mean-subtracted rewards.
In addition to the analyses of (Fu et al., 2017; Jeon et al., 2020) regarding reward shaping and normalization, meanzero rewards for training agents have the additional property of preventing termination awareness, as stated by the optimal stopping theorem (Theorem 4).
C. H YPERPARAMETERS
Tables 2, 3, and 4 show the hyperparameters of the conducted experiments.
[Table]
[Table] [Table]