Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
Abstract
Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert.
Many algorithms for IRL have an inherent nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit.
For high dimensional environments such nested-loop structure entails a significant computational burden.
To reduce the computational burden of a nested loop, novel methods such as SQIL (Reddy et al., 2020) and IQ-Learn (Garg et al., 2021) emphasize policy estimation at the expense of reward estimation accuracy.
However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks.
In this paper we develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy.
In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization.
We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee.
If the reward is parameterized linearly, we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in Correspondence to: Siliang Zeng zeng umn edu
Chenliang
Li chenliangli
Alfredo Garcia alfredo garcia tamu Mingyi Decision Awareness in Reinforcement Learning Workshop at the 39th International Conference on Machine Learning (ICML) , Baltimore, Maryland, USA, 2022.
Copyright 2022 by the author(s).
Mujoco and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.
1. Introduction
Given observed trajectories of states and actions implemented by an expert, we consider the problem of estimating the reinforcement learning environment in which the expert was trained.
This problem is generally referred to as inverse reinforcement learning (IRL) (see (Osa et al., 2018) for a recent survey).
Assuming the environment dynamics are known (or available online), the IRL problem consists of estimating the reward function and the expert’s policy (optimizing such rewards) that best fits the data.
While there are limitations on the identifiability of rewards (Kim et al., 2021), the estimation of rewards based upon expert trajectories enables important counterfactual analysis such as the estimation of optimal policies under different environment dynamics and/or reinforcement learning of new tasks.
In the seminal work (Ziebart et al., 2008), the authors developed an IRL formulation, in which the model for the expert’s behavior is the policy that maximizes entropy subject to a constraint requiring that the expected features under such policy match the empirical averages in the expert’s observation dataset.
The algorithms developed for MaxEnt-IRL (Ziebart et al., 2008; 2010; Wulfmeier et al., 2015) have a nested loop structure, alternating between an outer loop with a reward update step, and an inner loop that calculates the explicit policy estimates.
The computational burden of this nested structure is manageable in tabular environments, but it becomes significant in high dimensional settings requiring function approximation.
Towards developing more efficient IRL algorithms, a number of works (Finn et al., 2016a;b; Ho & Ermon, 2016; Fu et al., 2017) propose to leverage the idea of adversarial training (Goodfellow et al., 2014).
These algorithms learn a non-stationary reward function through training a discriminator, which is then used to guide the policy to match the behavior trajectories from the expert dataset.
However, (Ni et al., 2020) pointed out that the resulting discriminator (hence the reward function) typically cannot be used in new learning tasks, since it is highly dependent on the corresponding policy and current environment dynamics.
Moreover, due to the brittle approximation techniques and sensitive hyperparameter choice in the adversarial training, these IRL algorithms can be unstable.
(Kurach et al., 2018; Kostrikov et al., 2019). More recent works (Reddy et al., 2020; Garg et al., 2021) have developed algorithms to alleviate the computational burden of the nested-loop training procedures.
In (Reddy et al., 2020), the authors propose to model the IRL using certain maximum entropy RL problem with specific reward function (which assigns r = +1 for matching expert demonstrations and r = 0 for all other behaviors).
Then a soft Q imitation learning (SQIL) algorithm is developed.
In (Garg et al., 2021), the authors propose to transform the standard formulation of IRL (discussed above) into a single-level problem, through learning a soft Q-function to implicitly represent the reward function and the policy.
An inverse soft-Q learning (IQ-Learn) algorithm is then developed, which is shown to be effective in estimating the policy for the environment that it is trained on.
Despite being computational efficient, IQ-Learn sacrifices the accuracy in estimating the rewards since it indirectly recovers rewards from a soft Q-function approximator which is highly dependent upon the environment dynamics and does not strictly satisfy the soft-Bellman equation.
Therefore it is not well-suited for counterfactual prediction or transfer learning setting.
In f -IRL (Ni et al., 2020) the authors consider an approach for estimating rewards based on the minimization of several measures of divergence with respect to the expert’s state visitation measure.
The approach is limited to estimating rewards that only depend on state.
Moreover, while the results reported are based upon a single-loop implementation, the paper does not provide a convergence guarantee to support performance.
In the end, we introduce several related works which focus on developing “robust” algorithms and enable the agent to adapt to new environment with different dynamics.
There is a line of works focusing on disentangling the reward function from the environment dynamics, so that the recovered reward functions could be transferred across environments with different dynamics.
In (Fu et al., 2017), the authors propose an algorithm called adversarial inverse reinforcement learning (AIRL).
Constructing the estimated reward as a function which depends on the current state and next state, AIRL enables the agent to learn policies in a new environment through leveraging the estimated reward function recovered from the training environment.
In (Kim et al., 2021), the authors prove necessary and sufficient conditions for reward identifiability in deterministic MDP models with the maximum entropy reinforcement learning objective.
In (Cao et al., 2021), the authors present a theoretical analysis to show the necessary and sufficient condition to identify an action-independent time-homogeneous reward function under the maximum entropy IRL problem.
A recent line of works consider a more challenging setting, where the learner has no access to the expert environment, and there is a transition dynamics mismatch between the expert and the learner.
In (Liu et al., 2020), the authors propose a state alignment based imitation learning method so that the imitator could follow the state sequences in expert demonstrations as much as possible.
Arguing that the expert actions are not efficient demonstrations under transition dynamics mismatch, (Gangwani & Peng, 2020) further develops a state-only imitation learning method.
In (Viano et al., 2021), the authors revisit the Maximum Causal Entropy IRL when there is a transition dynamics mismatch between the expert and the learner.
A theoretical analysis is further provided in (Viano et al., 2021) to show the upper bound on the learner’s performance degradation, which is measured in terms of the ℓ 1 -distance between the transition dynamics of the expert and the learner.
Our Contributions.
The goal of this work is to develop an algorithm for IRL which is capable of producing high-quality estimates of both rewards and behavior policies with finite-time guarantees.
The major contributions of this work are listed below.
• We consider a formulation of IRL based on maximum likelihood (ML) estimation over optimal (entropy-regularized) policies, and prove that a strong duality relationship with maximum entropy IRL holds if rewards are represented by a linear combination of features.
1 The ML formulation is a bi-level optimization problem, where the upper-level problem maximizes the likelihood function, while the lower-level finds the optimal policy under the current reward parameterization.
Such a bi-level structure is not only instrumental to the subsequent algorithm design, but is also flexible to incorporate the use of state-only, as well as the regular reward function (which depends on the state and action pair).
The former is suitable for transfer learning since it is insensitive to the changes of the environment dynamics, while the latter can be used to efficiently imitate the expert policy.
• Based on the ML-IRL formulation, we develop an efficient algorithm.
To avoid the computational burden of repeatedly solving the lower-level policy optimization problem, the proposed algorithm has a single-loop structure where the policy improvement step and reward optimization step are performed alternatingly so that each step can be performed relatively cheaply.
Further, we show that the algorithm has strong theoretical guarantees: to achieve certain ϵ -approximate stationary solution for a non-linearly parameterized problem, it requires O ( ϵ − 2 ) steps of policy and reward updates each.
To our knowledge, it is the first algorithm which has finite-time guarantee for the IRL problem under nonlinear parameterization of reward functions. We
conduct extensive experiments to demonstrate that the proposed algorithm outperforms many state-of-the-art IRL algorithms in both policy estimation and reward recovery.
In particular, when transferring to a new environment, RL algorithms using rewards recovered by the proposed algorithm outperform those that use rewards recovered from existing IRL and imitation learning benchmarks.
2. Preliminaries
In this section, we review the fundamentals of the maximum entropy inverse reinforcement learning (MaxEnt-IRL).
We consider an MDP defined by the tuple ( S , A , P , η, r, γ ) ; S and A denote the state space and the action space respectively; P ( s ′ | s, a ) : S ×
A × S → [0 , 1] denotes the transition probability; η ( · ) denotes the distribution for the initial state; r ( s, a ) : S ×
A → R is the reward function and γ is a discount factor.
Given a dataset D which contains trajectories τ = { ( s t , a t ) } ∞ t =0 sampled from an expert policy, the MaxEnt-IRL formulation (Ziebart et al., 2010; 2013;
Bloem & Bambos, 2014) consists of finding a policy maximizing entropy subject to the expected features under such policy matching the empirical averages in the expert’s observation dataset.
Specifically, the MaxEnt-IRL formulation is given by:
[Equation]
s .
t .
E τ ∼ π [Equation] [Equation] where ϕ ( s, a ) is the feature vector of the state-action pair ( s, a ) .
Let θ denote the dual variable for the linear constraint, then the Lagrangian of (MaxEnt-IRL) is given by L
[Equation]
In (Ziebart et al., 2010; 2013; Bloem & Bambos, 2014), the authors proposed a ”dual descent” algorithm, which alternates between i) solving max π L ( π, θ ) for fixed θ , and ii) a gradient descent step to optimize the dual variable θ .
It is shown that the optimizer π ∗ θ in step i) can be recursively defined as π ∗ θ ( a t | s t ) =
Z at | st,θ Z st,θ , where log Z a t | s t ,θ =
[Equation]
and
[Equation]
From a computational perspective, the above algorithm is not efficient: it has a nested-loop structure, which repeatedly computes the optimal policy π ∗ θ under each variable θ .
It is known that when the underlying MDP is of high-dimension, such an algorithm can be computationally prohibitive (Finn et al., 2016b; Ho & Ermon, 2016).
Recent work (Garg et al., 2021) proposed an algorithm called IQ-Learn to improve upon the MaxEnt-IRL by considering a saddle-point formulation:
[Equation]
The authors show that this problem can be transformed into an optimization problem only defined in terms of the soft Q-function, which implicitly represents both reward and policy.
IQ-Learn is shown to be effective in imitating the expert behavior while only relying on the estimation of the soft Q-function.
However, the implicit reward estimate obtained is not necessarily accurate since its soft Q-function estimate depends on the environment dynamics and does not strictly satisfy the soft-Bellman equation.
Hence, it is difficult to transfer the recovered reward function to new environments.
3. Problem Formulation
In this section, we consider a ML formulation of the IRL problem and formalize a duality relationship with maximum entropy-based formulation (MaxEnt-IRL).
Maximum Log-Likelihood IRL (ML-IRL)
Let D := { τ i } Ni =1 denote the dataset containing observed trajectories.
A model of the expert’s behavior is a randomized policy π θ ( ·| s ) where θ is a parameter vector.
Assuming the state dynamics P ( s t +1 | s t , a t ) are known, the discounted log-likelihood of observing a sample trajectory τ under model π θ can be written follows:
[Equation]
[Table]
Let L ( θ ) :=
E τ ∼D (cid:2) (cid:80) t ≥
γ t log π θ ( a t | s t ) (cid:3) .
We consider the following maximum log-likelihood IRL formulation:
[Equation]
where r ( s, a ; θ ) is the reward function and H ( π ( ·| s ))
= − (cid:80) a ∈A π ( a | s ) log π ( a | s ) denotes the entropy of policy π ( ·| s ) .
We now make some remarks about ML-IRL.
First, the problem takes the form of a bi-level optimization problem, where the upper-level problem (ML-IRL) optimizes the reward parameter θ , while the lower-level problem describes the expert’s policy as the solution to an entropy-regularized
MDP ((Haarnoja et al., 2017; 2018)).
In what follows we will leverage recently developed (stochastic) algorithms for bi-level optimization (Hong et al., 2020; Ji et al., 2021; Khanduri et al., 2021), that avoid the high complexity resulted from nested loop algorithms.
Second, it is reasonable to use the ML function as the loss, because it searches for a reward function which generates a behavior policy that can best fit the expert demonstrations.
While the ML function has been considered in (Jain et al., 2019; Sanghvi et al., 2021), they rely on heuristic algorithms with nested-loop computations to solve their IRL formulations, and the theoretical properties are not studied.
Finally, the lower-level problem has been well-studied in the literature (Haarnoja et al., 2017; 2018; Cen et al., 2021; Cayci et al., 2021; Nachum et al., 2017).
The entropy regularization in (4a) ensures the uniqueness of the optimal policy π θ under the fixed reward function r ( s, a ; θ ) (Haarnoja et al., 2017; Cen et al., 2021).
Even when the underlying MDP is high-dimensional and/or complex, the optimal policy could still be obtained; see recent developments in (Haarnoja et al., 2017; 2018).
Subsequently, we will leverage these algorithms, together with recent advances in bi-level optimization to develop solution methods for (ML-IRL).
We close this section by formally establishing a connection between (MaxEnt-IRL) and (ML-IRL).
Theorem 3.1.
(Strong Duality) Suppose that the reward function is given as: r ( s, a ; θ ) :
= ϕ
( s, a ) T θ , for all s ∈ S and a ∈ A .
Then (ML-IRL) is the Lagrangian dual of (MaxEnt-IRL) .
Furthermore, strong duality holds, that is: L ( θ ∗ )
=
H ( π ∗ ) , where θ ∗ and π ∗ are the global optimal solutions for problems (ML-IRL) and (MaxEnt-IRL) , respectively.
The proof of Theorem 3.1 is relegated to Appendix F. To our knowledge this result which specifically addresses the (MaxEnt-IRL) formulation is novel.
A duality between ML estimation and maximum causal entropy is obtained in (Ziebart et al., 2013, Theorem 3).
However, the problem considered in that paper is not in RL nor IRL setting, therefore they cannot be directly used in the context of the present paper.
The above duality result reveals a strong connection between the two formulations under linear reward parameterization.
Of course, when non-linear parameterization is used strong duality may not hold.
However, this connection still suggests that ML-IRL formulation is a meaningful framework for IRL.
4. The Proposed Algorithm
In this section, we design algorithms for (ML-IRL).
Recall that one major drawback of algorithms for (MaxEnt-IRL) is that, they repeatedly solve certain policy optimization problem in the inner loop.
For example, the algorithm IQ-Learn (Garg et al., 2021) improves the computational efficiency through implicitly representing the reward function and the policy by a Q-function approximator, while the estimation accuracy of the recovered reward is sacrificed.
Therefore, one important goal of our design is to find provably efficient algorithms that can avoid high-complexity operations and accurately recover the reward function.
Specifically, it is desirable that the resulting algorithm only uses a finite number of reward and policy updates to reach certain high-quality solutions.
To proceed, we will leverage the special bi-level structure of the ML-IRL problem.
The idea is to alternate between one step of policy update to improve the solution of the lower-level problem, and one step of the parameter update which improves the upper-level loss function.
At each iteration k , given the current policy π k and the reward parameter θ k , a new policy π k +1 is generated from the policy improvement step, and θ k +1 is generated by the reward optimization step.
This kind of alternating update is efficient, because there is no need to completely solve the policy optimization subproblem, before updating the reward parameters.
It has been used in many other RL related settings as well.
For example, the well-known actor-critic (AC) algorithm for policy optimization (Konda & Tsitsiklis, 1999; Wu et al., 2020; Hong et al., 2020) alternates between one step of policy update, and one step of critic parameter update.
Below we present the details of our algorithm at a given iteration k .
Policy Improvement Step.
Given the reward parameter θ k is fixed, let us consider optimizing the lower-level problem.
Towards this end, define the so-called soft Q and soft value given policy π k and a reward parameter θ k :
[Equation]
We will adopt the well-known soft policy iteration (Haarnoja et al., 2017) to optimize the lower-level problem (4a).
Under the current reward parameter θ k and the policy π k , the soft policy iteration generates a new policy π k +1 as follows
[Equation]
Under a fixed reward function, it can be shown that the new policy π k +1 monotonically improves π k , and it converges linearly to the optimal policy; see (Haarnoja et al., 2017, Theorem 4) and (Cen et al., 2021, Thoerem 1).
Note that in practice, we usually do not have direct access to the exact soft Q-function in (5b).
In order to perform the policy improvement, a few stochastic update steps in soft
Q-learning (Haarnoja et al., 2017) or soft Actor-Critic (SAC) (Haarnoja et al., 2018) could be used to replace the one-step soft policy iteration (6).
In the appendix, we present Alg. 2 to demonstrate such alternative implementation of our proposed algorithm.
Reward Optimization Step.
We propose to use a stochastic gradient-type algorithm to optimize θ .
Towards this end, let us first derive the exact gradient ∇ L ( θ ) .
See Appendix C for detailed proof.
Lemma 4.1.
The gradient of the likelihood function L ( θ ) can be expressed as follows:
[Equation]
To obtain stochastic estimators of the exact gradient ∇ L ( θ k ) , we take two approximation steps: 1) approximate the optimal policy π θ k by π k +1 in (6), since the optimal policy π θ k is not available throughout the algorithm; 2) sample one expert trajectory τ Ek from the dataset D and one agent trajectory τ Ak from the current policy π k +1 so to approximate the expectation operators in (7).
Following the approximation steps mentioned above, we construct a stochastic estimator g k to approximate the exact gradient ∇ L ( θ k ) in (7) as follows:
[Equation]
where h ( θ ; τ ) = {+(cid:80) t ≥ 
+}
γ t ∇ θ r ( s t , a t ; θ ) .
With the stochastic gradient estimator g k , the reward parameter θ k is updated as:
[Equation]
where α is the stepsize in updating the reward parameter.
In summary, the proposed algorithm for solving the ML-IRL problem (ML-IRL) is given in Alg. 1.
5. Theoretical Analysis
In this section, we present finite-time guarantees for the proposed algorithm.
To begin with, first recall that in Sec. 3, we have mentioned that (ML-IRL) is a bi-level problem, where the upper level (resp.
the lower level) problem optimizes the reward parameter (resp.
the policy).
In order to solve (ML-IRL), our algorithm 1 has a singe-loop structure, which alternates between one step of policy update and one step of the reward parameter update.
Such a single-loop structure indeed has computational benefit, but it also leads to potential unstableness, since the lower level problem can stay far away from its true solutions.
Specifically, at each iteration k , the potential unstableness is induced by the distribution mismatch between the policy π k +1 and π θ k , when we use estimator g k (8) to approximate the exact gradient ∇ L ( θ k ) (7) in updating the reward parameter θ k .
Towards stabilizing the algorithm, we adopt the so-called two-timescale stochastic approximation (TTSA) approach (Borkar, 1997; Hong et al., 2020), where the lower-level problem updates in a faster time-scale (i.e., converges faster) compared with its upper-level counterpart.
Intuitively, the TTSA enables the π k +1 tracks the optimal π θ k , leading to a stable algorithm.
In the proposed Algorithm 1, the policy (lower-level variable) is continuously updated by the soft policy iteration (6), and it is ‘fast’ because it converges linearly to the optimal policy under a fixed reward function ( ? ) Theorem 1]cen2021fast.
On the other hand, the reward parameter update (9) does not have such linear convergence property, therefore it works in a ‘slow’ timescale.
To begin our analysis, let us first present a few technical assumptions.
Assumption 5.1 (Ergodicity) .
For any policy π , assume the Markov chain with transition kernel P is irreducible and aperiodic under policy π .
Then there exist constants κ >and ρ ∈ (0 , 1) such that
[Equation]
where ∥ · ∥ TV is the total variation (TV) norm; µ π is the stationary state distribution under π .
Assumption 5.1 assumes the Markov chain mixes at a geometric rate.
It is a common assumption in the iterature of RL (Bhandari et al., 2018; Zou et al., 2019; Wu et al., 2020), which holds for any time-homogeneous Markov chain with finite-state space or any uniformly ergodic Markov chain with general state space.
Assumption 5.2.
For any s ∈ S , a ∈ A and any reward parameter θ , the following holds:
[Equation]
∥ (10b) where L r and L g are positive constants.
Assumption 5.2 assumes that the parameterized reward function has bounded gradient and is Lipschitz smooth.
Such assumption in Lipschitz property are common in the literature of min-max / bi-level optimization (Jin et al., 2020;
Hong et al., 2020; Chen et al., 2021; Guan et al., 2021;
Khanduri et al., 2021).
Based on Assumptions 5.1 - 5.2, we next provide the following Lipschitz properties:
Lemma 5.3.
Suppose Assumptions 5.1 - 5.2 hold.
Given any reward parameters θ [-
-]1 and θ 2 , the following results hold for any s ∈ S and a ∈ A :
[Equation]
where Q soft r θ ,π θ ( · , · ) denotes the soft Q-function under the reward function r ( · , · ; θ ) and the policy π θ .
The positive constants L q and L c are defined in Appendix D.
The Lipschitz properties identified in Lemma 5.3 are vital for the convergence analysis.
Then we present the main results, which show the convergence speed of the policy { π k } k ≥ 0 and the reward parameter { θ k } k ≥ 0 in the Alg. 1.
Please see Appendix D for the detailed proof.
Theorem 5.4.
Suppose Assumptions 5.1 - 5.2 hold.
Selecting stepsize α :=
α 0 K σ for the reward update step (9) where α 0 > 0 and σ ∈ (0 , 1) are some fixed constants, and K is the total number of iterations to be run by the algorithm.
Then the following result holds:
[Equation]
where we denote
[Equation]
In particular, setting σ = 1 / 2 , then both quantities in (12a) and (12b) converge with the rate
O ( K − 1 / 2 ) .
In Theorem 5.4, we present the finite-time guarantee for the convergence of the Alg.1.
As a remark, we note that our theoretical guarantee is different from the existing results, such as (Cen et al., 2021) that showed the convergence rate of soft policy iteration under a fixed reward function.
Theorem 5.4 analyzes a more challenging setting where both the policy and reward parameter are kept changing.
To present the key steps in our analysis, we provide a proof sketch below.
The detailed proof is in Appendix G.
Proof sketch.
We outline our main steps in analyzing (12a) and (12b) respectively.
In order to show the convergence of policy estimates in (12a), there are several key steps.
First, we note that both policies π k +1 and π θ k are in the softmax parameterization, where π k +1 ( ·| s ) ∝ exp (cid:0) Q soft r θk ,π k ( s, · ) (cid:1) and
π θ k ( ·| s ) ∝ exp (cid:0) Q soft r θk ,π θk ( s, · ) (cid:1) .
Then, we can show a Lipschitz continuity property between the policy and the soft Q-function:
[Equation]
where the infinty norm ∥ · ∥ ∞ is defined over the state-action space S × A .
Moreover, by analyzing the contraction property of the soft policy iteration (6) [Equation] we
[Equation]
To ensure that the error term ∥ θ k − θ k − 1 ∥ is small, we select the stepsize of reward parameters as α :=
α 0 K σ , where K is the total number of iterations and σ > 0 .
Then, by combining previous two steps, we could further show the convergence rate of the policy estimates in (12a).
To prove the convergence of the reward parameters in (12b), we first leverage the Lipschitz smooth property of L ( θ ) in (11b).
However, one technical challenge in the convergence analysis is how to handle the bias between the gradient estimator g k defined in (8) and the exact gradient ∇ L ( θ k ) .
When we construct the gradient estimator g k in (8), we need to sample trajectories from the current policy π k +1 and the expert dataset D .
However, according to the expression of ∇ L ( θ k ) in (7), the trajectories are sampled from the optimal policy π θ k and the dataset D .
Hence, there is a distribution mismatch between π k +1 and π θ k .
Our key idea is to leverage (12a) to handle this distribution mismatch error, and thus show that the bias between g k and ∇ L ( θ k ) could be controlled.
6. A Discussion over State-Only Reward
In this section we consider the IRL problems modeled by using rewards that are only a function of the state.
A lower dimensional representation of the agent’s preferences (i.e. in terms only of states as opposed to states and actions) is more likely to facilitate counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks.
This is because the estimation of preferences which are only defined in terms of states is less sensitive to the specific environment dynamics in the expert’s demonstration dataset.
Moreover, in application such as healthcare (Yu et al., 2021) and autonomous driving (Kiran et al., 2021), where simply imitating the expert policy can potentially result in poor performance, since the learner and the expert may have different transition dynamics.
Similar points have also been argued in recent works (Gangwani & Peng, 2020; Ni et al., 2020; Viano et al., 2021).
Next, let us briefly discuss how we can understand (ML-IRL) and Alg.1, when the reward is parameterized as a state-only function.
Lemma 6.1.
Suppose the expert trajectories τ is sampled from a policy π E , and the reward is parameterized as a state-only function r ( s ; θ ) .
Then ML-IRL is equivalent to the following:
[Equation]
Please see Appendix E for the detailed derivation.
Intuitively, the above lemma says that, when dealing with the state-only IRL, (ML-IRL) minimizes the gap between the soft value functions of the optimal policy π θ and that of the expert policy π E .
Moreover, Alg.1 can also be easily implemented with the state-only reward.
In fact, the entire algorithm essentially stays the same, and the only change is that r ( s, a ; θ ) will be replaced by r ( s ; θ ) .
Therefore, even under the state-only IRL setting where the expert dataset only contains visited states, our formulation and the proposed algorithm still work if we parameterize the reward as a state-only function.
7. Numerical Results
In this section, we test the performance of our algorithm on a diverse collection of RL tasks and environments.
In each experiment set, we train algorithms until convergence and average the scores of the trajectories over multiple random seeds.
The hyperparameter settings and simulation details are provided in Appendix A.
Mujoco Tasks For Inverse Reinforcement Learning.
In this experiment set, we test the performance of our algorithm on imitating the expert behavior.
We consider several high-dimensional robotics control tasks in Mujoco (Todorov et al., 2012).
Two class of existing algorithms are considered as the comparison baselines: 1) imitation learning algorithms that only learn the policy to imitate the expert, including Behavior Cloning (BC)
(Pomerleau, 1988) and Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016); 2) IRL algorithms which learn a reward function and a policy simultaneously, including Adversarial
Inverse Reinforcement Learning (AIRL) (Fu et al., 2017), f -IRL (Ni et al., 2020) and IQ-Learn (Garg et al., 2021).
To ensure fair comparison, all imitation learning / IRL algorithms use soft Actor-Critic (Haarnoja et al., 2018) as the base RL algorithm.
For the expert dataset, we use the data provided in the official implementation 2 of f -IRL.
In this experiment, we implement two versions of our proposed algorithm: ML-IRL(State-Action) where the reward is parameterized as a function of state and action; ML-IRL(State-Only) which utilizes the state-only reward function.
In Table 1, we present the simulation results under a limited data regime where the expert dataset only contains a single expert trajectory.
The scores (cumulative rewards) reported in the table is averaged over 3 random seeds.
In each random seed, we train algorithm from initialization and collect 20 trajectories to average their cumulative rewards after the algorithms converge.
According to the results reported in Table 1, it shows that our proposed algorithms outperform the baselines on most tasks.
We observe that BC fails to imitate the expert’s behavior.
It [Table] [Table] is due to the fact that BC is based on supervised learning and thus could not learn a good policy under such a limited data regime.
Moreover, we notice the training of IQ-Learn is unstable, which may be due to its inaccurate approximation to the soft Q-function.
Therefore, in the Mujoco tasks where IQ-Learn does not perform well, so that we cannot match the results presented in the original paper (Garg et al., 2021), we directly report results from there (and mark them by ∗ in Table 1).
The results of AIRL are not presented in Table 1 since it performs poorly even after spending significant efforts in parameter tuning (similar observations have been made in in (Liu et al., 2020; Ni et al., 2020)).
Transfer Learning Across Changing Dynamics.
We further evaluate IRL algorithms on the transfer learning setting.
We follows the environment setup in (Fu et al., 2017), where two environments with different dynamics are considered: Custom-Ant vs Disabled-Ant .
We compare ML-IRL(State-Only) with several existing IRL methods: 1) AIRL (Fu et al., 2017), 2) f
-IRL (Ni et al., 2020); 3) IQ-Learn (Garg et al., 2021).
We consider two transfer learning settings: 1) data transfer; 2) reward transfer.
For both settings, the expert dataset / trajectories are generated in Custom-Ant .
In the data transfer setting, we train IRL agents in Disabled-Ant by using the expert trajectories, which are generated in Custom-Ant .
In the reward transfer setting, we first use IRL algorithms to infer the reward functions in Custom-Ant , and then transfer these recovered reward functions to Disabled-Ant for further evaluation.
In both settings, we also train SAC with the ground-truth reward in Disabled-Ant and report the scores.
The numerical results are reoprted in Table 2.
the proposed ML-IRL(State-Only) achieves superior performance compared with the existing IRL benchmarks in both settings.
We notice that IQ-Learn fails in both settings since it indirectly recovers the reward function from a soft Q-function approximator, which could be inaccurate and is highly dependent upon the environment dynamics.
Therefore, the reward function recovered by IQ-Learn can not be disentangled from the expert actions and environment dynamics, which leads to its failures in the transfer learning tasks.
8. Conclusion
In this paper, we present a maximum likelihood IRL formulation and propose a provably efficient algorithm with a single-loop structure.
To our knowledge, we provide the first non-asymptotic analysis for IRL algorithm under nonlinear reward parameterization.
As a by-product, when we parameterize the reward as a state-only function, our algorithm could work in state-only IRL setting and enable reward transfer to new environments with different dynamics.
Our algorithm outperforms existing IRL methods on high-dimensional robotics control tasks and corresponding transfer learning settings.
A limitation of our method is the requirement for online training, so one future direction of this work is to further extend our algorithm and the theoretical analysis to the offline IRL setting.
Potential Negative Social Impacts.
Since IRL methods aim to recover the reward function and the associated optimal policy from the observed expert dataset, potential negative social impacts may occur if there are bad demonstrations included in the expert dataset.
Thus, for sensitive applications such as autonomous driving and clinical decision support, additional care should be taken to avoid negative biases from the expert demonstrations and ensure safe adaptation.
Appendix A. Experiment Details A.1. Mujoco Tasks For Inverse Reinforcement Learning.
In all experiments, we test the performance of benchmark algorithms on Hopper, Half-Cheetah, Walker, Ant,
Humanoid environments from OpenAI Gym.
To ensure fair comparison, we use an open-source implementation 3 of SAC as the base RL algorithm for all imitation learning / IRL methods.
Moreover, Adam is used as the optimizer in SAC.
In SAC, both policy network and Q-network are (64 , 64) MLPs with ReLU activation function, and we set their stepsizes as 3 × 10 − 3 .
Moreover, in our proposed algorithms, we parameterize the reward function by a (64 , 64) MLPs with ReLU activation function.
For the reward network, we use Adam as the optimizer and the stepsize is set to be 1 × 10 − 4 .
We present the practical implementation procedure of our proposed algorithm in Table.2.
At each iteration, we first warm-start both policy network and Q-network in SAC by using the trained neural networks from the previous iteration.
Then, we run 10 episodes in the corresponding Mujoco environment to train the policy network and Q-network in SAC.
After that, we collect agent trajectory and expert trajectory to train the reward network by one gradient update.
For the imitation learning / IRL benchmark algorithms, we use their open-source implementations in our experiments.
The official implementations of f -IRL is provided in https://github.com/twni2016/f-IRL .
The offical code base for IQ-Learn is provided in https://github.com/Div99/IQ-Learn .
For the remaining benchmarks including BC, GAIL and AIRL, we refer to a open-source implementation: https://github.com/KamyarGh/rl_swiss .
A.2. Transfer Learning Across Changing Dynamics.
In this experiment, we follow the setup in (Fu et al., 2017).
A standard ant ( Custom-Ant ) and an ant with two disabled legs ( Disabled-Ant ) are simulated in Mujoco.
For all benchmark algorithms tested in this experiment, we follow same network structure and hyperparameter settings described in Section A.1.
Here, we provide a supplementary experiment result to show the performance of benchmark algorithms under different number of expert trajectory.
The performance of AIRL and IQ-Learn is not presented in Table 3 since we found their training is unstable (as we have mentioned in Sec.7).
The scores in Hopper are recorded after 1 × 10 6 environment steps and the scores in other environments are recorded after 2 × 10 6 environment steps.
The scores are reported after 3 independent
Monte Carlo (MC) trials for each algorithm.
[Table]
B. Auxiliary Lemmas
Throughout this section, we assume Assumptions 5.1 - 5.2 hold true.
Lemma B.1.
((Xu et al., 2020, Lemma 3)) Consider the initialization distribution η ( · ) and transition kernel P ( ·| s, a ) .
Under η ( · ) and P ( ·| s, a ) , denote d w ( · , · ) as the state-action visitation distribution of MDP with the Boltzman policy parameterized by parameter w .
Suppose Assumption 5.1 holds, for all policy parameter w and w ′ , we have
[Equation]
where C d is a positive constant.
Lemma B.2.
((Haarnoja et al., 2017, Theorem 4))
Under a reward function r ( · , · ) , given a policy π , we define a new policy ˜ π as
[Equation]
For any s ∈ S , a ∈ A , it holds that Q soft r, ˜ π ( s, a ) ≥ Q soft r,π ( s, a ) .
Next, in order to facilitate analysis for entropy-regularized MDPs, we introduce a “soft” Bellman optimality operator T : R |S|×|A| → R |S|×|A| as follows:
[Equation]
In the following lemma, the properties of entropy-regularized MDPs are characterized.
Lemma B.3.
((Cen et al., 2021, Lemma 2))
The operator T as defined in (15) satisfies the properties below:
• T has the following closed-form expression:
[Equation]
• T is a γ -contraction in the ℓ ∞ norm, namely, for any Q 1 , Q 2 ∈ R |S|×|A| , it holds that
[Equation]
• Under a given reward function r ( · , · ) , the corresponding optimal soft Q -function Q soft r,π ∗ is a unique fixed point of the operator T , namely,
[Equation]
Proof.
This Lemma is proved in (Cen et al., 2021, Lemma).
We refine its analysis as below.
We first show that
[Equation]
where (i) is from Jensen’s inequality.
Moreover [Equation] the equality between both sides of (i) holds when the policy π has the expression π ( ·| s )
∝ exp( Q ( s, · )) .
Therefore, through applying the inequality (19) to (15), it obtains that
[Equation]
which proves the equality (16).
We define ∥ Q 1 − Q 2 ∥ ∞ := max s ∈S ,a ∈A | Q 1 ( s, a ) − Q 2 ( s, a ) | and ϵ =
∥ Q 1 − Q 2 ∥ ∞ .
Then for any s ∈ S and a ∈ A , it follows that
[Equation]
Similarly
[Equation]
it is easy to obtain that log (cid:0)(cid:80)
[Equation]
exp
[Equation]
Hence, it leads to the contraction property that
[Equation]
which proves the contraction property (17) [Equation]
Moreover, we have
[Equation]
where (i) follows the equality (20).
Based on the defition of the soft Q-function Q soft r,π ∗ , we have
[Equation]
We prove the equality (ii) in (22) through combining (23) and the fact that the optimal soft policy has the closed form π ∗ ( ·| s ) ∝ exp (cid:0) Q soft r,π ∗ ( s, · ) (cid:1) .
Suppose two different fixed points of the soft Bellman operator exist, then it contradicts with the contraction property in (21).
Hence, we proved the uniqueness of the optimal soft Q -function Q soft r,π ∗ .
Moreover, the optimal soft Q -function Q soft r,π ∗ is a fixed point to the soft Bellman operator T in (18).
Lemma B.4.
Suppose Assumption 5.2 holds.
Under an arbitrary policy π , for any s ∈ S , a ∈ A and any reward parameters θ 1 and θ 2 , the following inequality holds:
[Equation]
where L q :
=
L r 1 − γ
and L r is the positive constant in Assumption 5.2.
Proof.
Based on the definition of soft-Q function, we have
[Equation]
Then it holds that
[Equation]
where (i) follows Jensen’s inequality; (ii) follows the mean value theorem; (iii) follows inequality (10a) in Assumption 5.2.
C. Proof of Lemma 4.
Proof.
First, we are able to express the objective function L ( θ ) in (ML-IRL) as below:
[Equation]
where (i) is due to the fact that the optimal policy has the closed form π θ ( ·| s ) ∝ exp (cid:0) Q soft r θ ,π θ ( s, · ) (cid:1) .
Therefore, we could express the objective function in this form:
[Equation]
where (i) and (ii) follows the fact that the the optimal soft value function could be expressed as V soft r θ ,π θ ( s )
= log (cid:0)
Based on (26)
[Equation]
we calculate the exact gradient of the objective function L ( θ ) as below:
[Equation]
Then we need to calculate the gradient ∇ θ Q soft r θ ,π θ ( s 0 , a 0 ) as follows.
[Equation]
where (i) and (iv) follows the definition of the soft Q-function; (ii) follows the fact that V soft r θ ,π θ ( s )
= log( (cid:80) a exp
[Equation] Q [Equation] θ ,π θ ( s, a ))) ; (iii) follows the fact that π θ ( a | s ) ∝ exp( Q soft r θ ,π θ ( s, a )) ; (v) is shown by recursively applying (i) - (iv).
Finally, plugging equation (28) into (27), the gradient of the maximum likelihood objective is:
[Equation]
D. Proof of Lemma 5.
To proof Lemma 5.3, we proof the equality (11a) and the equality (11b) respectively.
The constants L q and L c in Lemma 5.3 has the expression:
[Equation]
D.1. Proof of Inequality (11a)
In this subsection, we prove the inequality (11a) in Lemma 5.3.
Proof.
We show that Q soft r θ ,π θ has bounded gradient with respect to any reward parameter θ , then the inequality (11a) holds due to the mean value theorem.
According to the equality (28), we have shown the explicit expression of ∇ θ Q soft r θ ,π θ ( s, a ) for any s ∈ S and a ∈ A .
Using this expression, we have the following series of relations:
[Equation]
where (i) is from the equality (28) in the proof of Lemma 4.1, (ii) follows Jensen’s inequality and (iii) follows the inequality (10a) in Assumption 5.2.
To complete this proof, we use the mean value theorem to show that
[Equation]
where the last inequality follows (30) and we denote L q :
= L r 1 − γ .
Therefore, we have proved the Lipschitz continuous inequality in (11a).
D.2. Proof of Inequality (11b)
In this section, we prove the inequality (11b) in Lemma 5.3.
Proof.
According to Lemma 4.1, the gradient ∇ L ( θ ) is expressed as:
[Equation]
Using the above relation, we have
[Equation]
where (i) follows the exact gradient expression in equation (31).
Then we separately analyze term A and term B in (32). For term A, it follows that
[Equation]
where (i) follows Jensen’s inequality and (ii) is from (10b) in Assumption
[Equation]
[Equation]
For the term B, it holds that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
) where (i) follows the triangle inequality, (ii) is from Jensen’s inequality and the definition of the discounted state-action visitation measure
d ( s, a ; π ) :=
(1 − γ ) π ( a | s ) (cid:80) t ≥ 0 γ t P π ( s t = s ) ; (iii) is from (10b) in Assumption 5.2;(iv) is from (10a) and the definition of the total variation norm.
Plugging the inequalities (33), (34) to (32) [Equation] it holds that
[Equation]
Given the fact that π θ is a Boltzmann policy parameterized by Q soft r θ ,π θ where π θ ( a | s ) ∝ exp( Q soft r θ ,π θ ( s, a )) , we show the inequality (i) from the inequality (14) in Lemma B.1.
Moreover, the inequality (ii) follows the equivalence relation between Frobenius norm and infinity norm and (iii) is from the inequality (11a) in Lemma 5.3.
Define the constant L c :=[-
-]
[Equation] L q L r [Equation]
we have the following inequality:
[Equation]
Therefore, we complete the proof of the inequality (11b) in Lemma 5.3.
E. Proof of Lemma 6.
Proof.
Suppose the expert trajectories τ in ML-IRL is sampled from an expert policy π E .
Moreover, we parameterize the state-only reward as r ( s ; θ ) .
Then the objective function L ( θ ) in ML-IRL could be rewritten as follows.
[Equation]
where (i) follows (25) and the fact that the reward is a state-only function r ( s ; θ ) ; (ii) follows the definitions of the soft value function.
Ignoring the constant term H ( π E ) in (36), the maximum likelihood formulation (ML-IRL) is equivalent to the following bi-level problem:
[Equation]
Therefore, we complete the proof of Lemma 6.1.
As an alternative interpretation to (ML-IRL), the formulation above aims to minimize the gap between the soft value function of π θ and π E under the state-only IRL setting.
F. Proof of Theorem 3.
Proof.
Calculate the Lagrangian of MaxEnt-IRL, we obtain that
[Equation]
where θ is the dual variable to ensure the feature matching equality in (1), and C s t
= s is the dual variable to ensure that π is a well-defined policy satisfying (cid:80) a ∈A π ( a | s t = s )
= 1 .
Then we could calculate the gradient of (37) w.r.t.
π ( a | s t = s ) , and set it to 0 .
Then it holds that
[Equation]
Dividing γ t P π ( s t = s ) on both sides of (38) and further moving log π ( a | s t = s ) to the left side, then we have the equality as below:
[Equation]
Given that −
C [Equation]
) − 1 is independent of action a
[Equation]
we could express the closed form of π ( a | s t = s ) as below:
[Equation]
According to the closed form of the policy above
[Equation]
it shows that π ( a | s t = s ) is a stationary policy being independent of the time index t .
Therefore, it holds that π ( a | s t = s ) =
π ( a | s ) for any t ≥ 0 .
Denoting a linearly parameterized reward as r ( s, a ; θ ) :
= θ T ϕ ( s, a ) , it holds that
[Equation]
Here, the optimal π ( a | s ) is a function of the dual variables (reward parameters) θ .
In the maximum entropy reinforcement learning (Haarnoja et al., 2017), under a reward function r ( · , · ) and policy π , the soft value function and soft Q-function are defined as below:
[Equation]
Based on the definitions in (41a) - (41b), we could further express the closed form of the policy in (40) as below:
[Equation]
According to (Haarnoja et al., 2017), under a reward function r ( · , · ) , the optimal soft policy π satisfies π ( ·| s ) ∝ exp( Q softr,π ( s, · )) .
Hence, we have shown that the policy in (42) is the optimal policy under the reward function r ( · , · ; θ ) .
After denoting the optimal policy under r ( · , · ; θ ) as π θ , we have the following relation:
[Equation]
where (a) is due to the equality shown as below:
[Equation]
Rewriting the equality (38), we are able to show the expression of C s t =
s as below:
[Equation]
44) where (a) follows the definition of the soft Q-function in (41b), and (b) follows (43).
According to (44), we are able to show the exact expression of C s t =
s .
Plugging π θ and C s t =
s into (37), we have
[Equation]
where (a) is due to the fact that (cid:80) a ∈A π θ ( a | s t = s )
= 1 for all a ∈ A and s ∈ S ; (b) follows (43); (c) is due to the definition of the soft Q-function in (41b).
Here
[Equation]
we could further show the problem in (45) is equivalent to ML-IRL as below:
[Equation]
Finally, through combining (45) and (46), we are able to know that the maximum likelihood formulation ML-IRL is the dual form of MaxEnt-IRL.
G. Proof of Theorem 5.
In this section [Equation] we prove (12a) and (12b) respectively, to show the convergence of the lower-level problem and the upper-level problem.
G.1. Proof of (12a)
Proof.
In this proof, we first show the convergence of the lower-level variable { π k } k ≥ 0 .
Recall that we approximate the optimal policy π θ k by π k +1 at each iteration k .
We first analyze the approximation error between π θ k and π k +1 as follows.
For any s ∈ S and a ∈ A
[Equation]
we have the following relation:
[Equation]
where (i) follows (6) and the fact that π θ ( a | s ) ∝ exp( Q soft r θ ,π θ ( s, a )) ; (ii) follows the triangle inequality.
We further analyze the second term in (47)
[Equation]
We first denote the operator log( ∥ exp( v ) ∥ 1 ) :
= log( ∥ (cid:80) ˜ a ∈A exp( v ˜ a ) ∥ 1 ) , where the vector v ∈ R |A| and v = [ v 1 , v 2 , · · · , v |A| ] .
Then for any v ′ , v ′′ ∈ R |A| , we have the following relation:
[Equation]
[Equation]
where (i) follows the mean value theorem and v c is a convex combination of v ′ and v ′′ ; (ii) follows the following equalities:
[Equation]
v ∈ R |A| .
Through plugging (48) into (47), it holds that
[Equation]
Taking the infinity norm over R |S|·|A| , the following result holds:
[Equation]
where ∥ log π k +1 − log π θ k ∥ ∞ = max s ∈S ,a ∈A | log π k +1 ( a | s ) − log π θ k ( a | s ) | and ∥ Q soft r θk ,π k − Q soft r θk ,π θk ∥ ∞
= max s ∈S ,a ∈A | Q soft r θk ,π k ( s, a ) − Q soft r θk ,π θk ( s, a ) | .
Based on the inequality (50), we analyze ∥ Q soft r θk ,π k − Q soft r θk ,π θk ∥ ∞ to show the convergence of the policy estimates.
It leads to the following analysis:
[Equation]
∥ [Equation] where (i) is from (11a) in Lemma 5.3; (ii) follows Lemma B.4.
Based on (51), we further analyze the two terms in (51) as below.
Recall Lemma B.
[Equation]
we have the “soft” Bellman operator expressed as below:
[Equation]
According to the soft Bellman operator
[Equation]
it holds that
[Equation]
where (i) follows the policy improvement result in Lemma B.2, (ii) follows the definition π k +1 ( a | s ) :
=
[Equation]
in (6); (iii) follows the definition of the soft Bellman operator in (52).
For any s ∈ S and a ∈ A , it holds that
[Equation]
where (i) is due to the fact that π θ k is the optimal policy under reward parameter θ k ; (ii) is from (53).
Hence, it further leads to
[Equation]
55) where (i) is from (54); (ii) is from the fixed-point property in (18); (iii) is from the contraction property in (17).
Therefore, we have the following result:
[Equation]
where (i) is from (51); (ii) is from (55).
To show the convergence of the soft Q-function based on (56), we further analyze the error between the reward parameters θ k and θ k − 1 .
Recall in Alg.1, the updates in reward parameters follows (9):
[Equation]
where we denote τ = { ( s t , a t ) } ∞ t =0 , h ( θ, τ ) :
= (cid:80) t ≥
γ t ∇ θ r ( s t , a t ; θ ) and g k − 1 is the stochastic gradient estimator at iteration k − 1 .
Here, τ Ek − 1 denotes the trajectory sampled from the expert’s dataset D at iteration k − 1 and τ Ak − 1 denotes the trajectory sampled from the agent’s policy π k at time k − 1 .
Then according to the inequality (10a) in Assumption 5.2, we could show that
[Equation]
where the last equality follows the fact that we have defined the constant L q :
= L r 1 − γ .
Then we could further show that
[Equation]
(58) where (i) is from (56); (ii) follows the reward update scheme in (9); (iii) is from (57).
Summing the inequality (58) from k = 1 to k
= K , it holds that
[Equation]
Rearranging the inequality (59) and divided (59) by K on both sides, it holds that
[Equation]
Dividing the constant 1 − γ on both sides of (60)
[Equation]
it holds that
[Equation]
where we denote C 0 :
=
∥ Q soft r θ 0 ,π 0 − Q soft r θ 0 ,π θ 0 ∥ ∞ .
We could also write the inequality above as
[Equation]
Recall the stepsize is defined as α = α 0 T σ where σ > 0 .
Then we have the following result:
[Equation]
With the inequality (50)
[Equation]
it follows that
[Equation]
Therefore, we complete the proof of (12a) in Theorem 5.4.
G.2. Proof of (12b)
Proof.
In this part, we prove the convergence of reward parameters { θ k } k ≥ 0 .
We have the following result of the objective function L ( θ ) :
[Equation]
where (i) is from the Lipschitz smooth property in (11b) of Lemma 5.3; (ii) follows the update scheme (9); (iii) is from constant bound in (57).
Taking an expectation over the both sides of (62)
[Equation]
it holds that
[Equation]
where (i) follows (7) and (8); (ii) is due to the fact that ∥∇ L ( θ ) ∥ ≤ 2 L q .
Then we further analyze the term A as below:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where (i) follows the definition d ( s, a ; π )
= (1 − γ ) π ( a | s ) (cid:80) t ≥ 0 γ t P π
( s t = s ) ; (ii) is due to distribution mismatch between two visitation measures; (iii) follows the inequality (10a) in Assumption 5.2; the inequality (iv) follows Lemma B.1 and the fact that π θ k ( ·| s ) ∝ exp (cid:0) Q soft r θk ,π θk ( s, · ) (cid:1) , π k +1 ( ·
[Equation] s ) ∝ exp (cid:0) Q soft r θk ,π k ( s, · ) (cid:1) and the constant L q :=
L r 1 − γ ; (v) follows the conversion between Frobenius norm and infinity norm.
Through plugging the inequality (64) into (63), it leads to
E [ L ( θ k +1 )]
[Equation]
where (i) follows the inequality (64).
Rearranging the inequality above and denote C 1 := 4 C d L 2 q (cid:112) |S| · |A| , it holds that
[Equation]
Summing the inequality above from k = 0 to K − 1 and dividing both sides by αK , it holds that
[Equation]
Note that the log-likelihood function L ( θ K ) is negative and L ( θ 0 ) is a bounded constant.
Then we could plug (61) into (65), it holds that
[Equation]
which completes the proof for the inequality (12b).