Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
Abstract
Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best ﬁts observed sequences of states and actions implemented by an expert.
Many algorithms for IRL have an inherently nested structure: the inner loop ﬁnds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of ﬁt.
For high dimensional environments such nested-loop structure entails a signiﬁcant computational burden.
To reduce the computational burden of a nested loop, novel methods such as SQIL [1] and IQ-Learn [2] emphasize policy estimation at the expense of reward estimation accuracy.
However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks.
In this paper we develop a novel single-loop algorithm for
IRL that does not compromise reward estimation accuracy.
In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization.
We show that the proposed algorithm provably converges to a stationary solution with a ﬁnite-time guarantee.
If the reward is parameterized linearly, we show the identiﬁed solution corresponds to the solution of the maximum entropy IRL problem.
Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.
Introduction
Given observed trajectories of states and actions implemented by an expert, we consider the problem of estimating the reinforcement learning environment in which the expert was trained.
This problem is generally referred to as inverse reinforcement learning (IRL) (see [3] for a recent survey).
Assuming the environment dynamics are known (or available online), the IRL problem consists of estimating the reward function and the expert’s policy (optimizing such rewards) that best ﬁts the data.
While there are limitations on the identiﬁability of rewards [4], the estimation of rewards based upon expert trajectories enables important counterfactual analysis such as the estimation of optimal policies under different environment dynamics and/or reinforcement learning of new tasks.
In the seminal work [5], the authors developed an IRL formulation, in which the model for the expert’s behavior is the policy that maximizes entropy subject to a constraint requiring that the expected features under such policy match the empirical averages in the expert’s observation dataset.
The algorithms developed for MaxEnt-IRL [5–7] have a nested loop structure, alternating between an outer loop with a reward update step, and an inner loop that calculates the explicit policy estimates.
The computational burden of this nested structure is manageable in tabular environments, but it becomes signiﬁcant in high dimensional settings requiring function approximation.
Towards developing more efﬁcient IRL algorithms, a number of works [8–12] propose to leverage the idea of adversarial training [13].
These algorithms learn a non-stationary reward function through training a discriminator, which is then used to guide the policy to match the behavior trajectories from the expert dataset.
However, [14] pointed out that the resulting discriminator (hence the reward function) typically cannot be used in new learning tasks, since it is highly dependent on the corresponding policy and current environment dynamics.
Moreover, due to the brittle approximation techniques and sensitive hyperparameter choice in the adversarial training, these IRL algorithms can be unstable.
More recent works [1, 2] have developed algorithms to alleviate the computational burden of the nested-loop training procedures.
In [1], the authors propose to model the IRL using certain maximum entropy RL problem with speciﬁc reward function (which assigns r = +1 for matching expert demonstrations and r = 0 for all other behaviors).
Then a soft Q imitation learning (SQIL) algorithm is developed.
In [2], the authors propose to transform the standard formulation of IRL (discussed above) into a single-level problem, through learning a soft Q-function to implicitly represent the reward function and the policy.
An inverse soft-Q learning (IQ-Learn) algorithm is then developed, which is shown to be effective in estimating the policy for the environment that it is trained on.
Despite being computationally efﬁcient, IQ-Learn sacriﬁces the accuracy in estimating the rewards since it indirectly recovers rewards from a soft Q-function approximator which is highly dependent upon the environment dynamics and does not strictly satisfy the soft-Bellman equation.
Therefore it is not well-suited for counterfactual prediction or transfer learning setting.
Finally, in f -IRL [14] the authors consider an approach for estimating rewards based on the minimization of several measures of divergence with respect to the expert’s state visitation measure.
The approach is limited to estimating rewards that only depend on state.
Moreover, while the results reported are based upon a single-loop implementation, the paper does not provide a convergence guarantee to support performance.
We refer the readers to Appendix A for other related works.
Our Contributions.
The goal of this work is to develop an algorithm for IRL which is capable of producing high-quality estimates of both rewards and behavior policies with ﬁnite-time guarantees.
The major contributions of this work are listed below.
• We consider a formulation of IRL based on maximum likelihood (ML) estimation over optimal (entropy-regularized) policies, and prove that a strong duality relationship with maximum entropy IRL holds if rewards are represented by a linear combination of features.
1 The ML formulation is a bi-level optimization problem, where the upper-level problem maximizes the likelihood function, while the lower-level ﬁnds the optimal policy under the current reward parameterization.
Such a bi-level structure is not only instrumental to the subsequent algorithm design, but is also ﬂexible to incorporate the use of state-only, as well as the regular reward function (which depends on the state and action pair).
The former is suitable for transfer learning since it is insensitive to the changes of the environment dynamics, while the latter can be used to efﬁciently imitate the expert policy.
• Based on the ML-IRL formulation, we develop an efﬁcient algorithm.
To avoid the computational burden of repeatedly solving the lower-level policy optimization problem, the proposed algorithm has a single-loop structure where the policy improvement step and reward optimization step are performed alternatingly so that each step can be performed relatively cheaply.
Further, we show that the algorithm has strong theoretical guarantees: to achieve certain ✏ -approximate stationary solution for a non-linearly parameterized problem, it requires O ( ✏  2 ) steps of policy and reward updates each.
To our knowledge, it is the ﬁrst algorithm which has ﬁnite-time guarantee for the IRL problem under nonlinear parameterization of reward functions.
• We conduct extensive experiments to demonstrate that the proposed algorithm outperforms many state-of-the-art IRL algorithms in both policy estimation and reward recovery.
In particular, when transferring to a new environment, RL algorithms using rewards recovered by the proposed algorithm outperform those that use rewards recovered from existing IRL and imitation learning benchmarks.
Preliminaries
In this section, we review the fundamentals of the maximum entropy inverse reinforcement learning (MaxEnt-IRL).
We consider an MDP deﬁned by the tuple ( S , A , P , ⌘ , r,  ) ; S and A denote the state space and the action space respectively;[-P ( s 
 | s, a ) : S ⇥-]
[Equation]
[0 , 1] denotes the transition probability; ⌘ ( · ) denotes the distribution for the initial state; r [Equation] s, a [Equation]
S ⇥
[Equation]
R is the reward function and  is a discount factor.
The MaxEnt-IRL formulation [6, 18–20] consists of ﬁnding a policy maximizing entropy subject to the expected features under such policy matching the empirical averages in the expert’s observation dataset.
Speciﬁcally, the MaxEnt-IRL formulation is given by:
[Equation]
where ⌧ = { ( s t , a t ) } 1 t =0 denotes a trajectory,  ( s, a ) is the feature vector of the state-action pair ( s, a ) and ⇡ E denotes the expert policy.
Let ✓ denote the dual variable for the linear constraint, then the Lagrangian of (MaxEnt-IRL) is given by
[Equation]
In [6, 18, 19], the authors proposed a "dual descent" algorithm, which alternates between i) solving max ⇡ L ( ⇡ , ✓ ) for ﬁxed ✓ , and ii) a gradient descent step to optimize the dual variable ✓ .
It is shown that the optimizer ⇡ ⇤ ✓ in step i) can be recursively deﬁned as ⇡ ⇤ ✓ ( a t | s t ) =
Z at | st, ✓ Z st, ✓ , where log Z a t | s t , ✓ =  ( s t , a t ) T ✓ +  E s t +1 ⇠ P ( ·| s t ,a t ) ⇥ log
Z s t +1 , ✓ ⇤ and log Z s t , ✓ = log P a 2 A Z a | s t , ✓  .
From a computational perspective, the above algorithm is not efﬁcient: it has a nested-loop structure, which repeatedly computes the optimal policy ⇡ ⇤ ✓ under each variable ✓ .
It is known that when the underlying MDP is of high-dimension, such an algorithm can be computationally prohibitive [9, 10].
Recent work [2] proposed an algorithm called IQ-Learn to improve upon the MaxEnt-IRL by considering a saddle-point formulation:
[Equation]
where r ( s t , a t ) is the reward associated with state-action pair ( s t , a t ) .
The authors show that this problem can be transformed into an optimization problem only deﬁned in terms of the soft Q-function, which implicitly represents both reward and policy.
IQ-Learn is shown to be effective in imitating the expert behavior while only relying on the estimation of the soft Q-function.
However, the implicit reward estimate obtained is not necessarily accurate since its soft Q-function estimate depends on the environment dynamics and does not strictly satisfy the soft-Bellman equation.
Hence, it is difﬁcult to transfer the recovered reward function to new environments.
Problem Formulation
In this section, we consider a ML formulation of the IRL problem and formalize a duality relationship with maximum entropy-based formulation (MaxEnt-IRL).
Maximum Log-Likelihood IRL (ML-IRL)
A model of the expert’s behavior is a randomized policy ⇡ ✓ ( ·| s ) , where ⇡ ✓ is a speciﬁc policy corresponding to the reward parameter ✓ .
With the state dynamics P ( s t +1 | s t , a t ) , the discounted log-likelihood of observing the expert trajectory ⌧ under model ⇡ ✓ can be written follows:
[Equation]
Then we consider the following maximum log-likelihood IRL formulation:
[Equation]
where r ( s, a ; ✓ ) is the reward function and H ( ⇡ ( ·| s ))
:=
[Equation]
We now make some remarks about ML-IRL.
First, the problem takes the form of a bi-level optimization problem, where the upper-level problem (ML-IRL) optimizes the reward parameter ✓ , while the lower-level problem describes the corresponding policy ⇡ ✓ as the solution to an entropy-regularized MDP ([21, 22]).
In what follows we will leverage recently developed (stochastic) algorithms for bi-level optimization [23–25], that avoid the high complexity resulted from nested loop algorithms.
Second, it is reasonable to use the ML function as the loss, because it searches for a reward function which generates a behavior policy that can best ﬁt the expert demonstrations.
While the ML function has been considered in [26, 27], they rely on heuristic algorithms with nested-loop computations to solve their IRL formulations, and the theoretical properties are not studied.
Finally, the lower-level problem has been well-studied in the literature [21, 22, 28–30].
The entropy regularization in (3a) ensures the uniqueness of the optimal policy ⇡ ✓ under the ﬁxed reward function r ( s, a ; ✓ ) [21, 28].
Even when the underlying MDP is high-dimensional and/or complex, the optimal policy could still be obtained; see recent developments in [21, 22].
We close this section by formally establishing a connection between (MaxEnt-IRL) and (ML-IRL).
Theorem 1.
(Strong Duality) Suppose that the reward function is given as: r ( s, a ; ✓ ) :
=
( s, a ) T ✓ , for all s 2 S and a 2 A .
Then (ML-IRL) is the Lagrangian dual of (MaxEnt-IRL) .
Furthermore, strong duality holds, that is: L ( ✓ ⇤ )
=
H ( ⇡ ⇤ ) , where ✓ ⇤ and ⇡ ⇤ are the global optimal solutions for problems (ML-IRL) and (MaxEnt-IRL) , respectively.
The proof of Theorem 1 is relegated to Appendix G. To our knowledge this result which speciﬁcally addresses the (MaxEnt-IRL) formulation is novel.
Under ﬁnite horizon, a duality between ML estimation and maximum causal entropy is obtained in [18, Theorem 3].
However, the problem considered in that paper is not in RL nor IRL setting, therefore they cannot be directly used in the context of the present paper.
The above duality result reveals a strong connection between the two formulations under linear reward parameterization.
Due to the duality result, we know that (ML-IRL) is a concave problem under linear reward parameterization.
In this case, any stationary solution to (ML-IRL) is a global optimal estimator of the reward parameter.
The Proposed Algorithm
In this section, we design algorithms for (ML-IRL).
Recall that one major drawback of algorithms for (MaxEnt-IRL) is that, they repeatedly solve certain policy optimization problem in the inner loop.
Even though the recently proposed algorithm IQ-Learn [2] tries to improve the computational efﬁciency through implicitly representing the reward function and the policy by a Q-function approximator, it has sacriﬁced the estimation accuracy of the recovered reward.
Therefore, one important goal of our design is to ﬁnd provably efﬁcient algorithms that can avoid high-complexity operations and accurately recover the reward function.
Speciﬁcally, it is desirable that the resulting algorithm only uses a ﬁnite number of reward and policy updates to reach certain high-quality solutions.
To proceed, we will leverage the special bi-level structure of the ML-IRL problem.
The idea is to alternate between one step of policy update to improve the solution of the lower-level problem, and Algorithm Input: Initialize reward parameter ✓ 0 and policy ⇡ 0 . Set the reward parameter’s stepsize as ↵ .
for k = 0 , 1 , . . . , K  1 do Policy Evaluation: Compute Q soft r ✓ k , ⇡ k ( · , · ) under reward function r ( · , · ; ✓ k ) Policy Improvement: ⇡ k +1 ( ·| s ) / exp( Q soft r ✓ k , ⇡ k ( s, · )) , 8 s 2 S .
Data Sampling I: Sampling an expert trajectory ⌧ E k :=
{ s t , a t } t
Data Sampling II:
Sampling a trajectory ⌧ A k := { s t , a t } t  0 from the current policy ⇡ k +
Estimating Gradient: g k :
= h ( ✓ k ; ⌧ E k )  h ( ✓ k ; ⌧ A k ) where h ( ✓ ; ⌧ ) :
=
P t  0  t r ✓ r ( s t , a t ; ✓ )
Reward Parameter Update: ✓ k +1 :=
✓ k + ↵ g k end for one step of the parameter update which improves the upper-level loss function.
At each iteration k , given the current policy ⇡ k and the reward parameter ✓ k , a new policy ⇡ k +1 is generated from the policy improvement step, and ✓ k +1 is generated by the reward optimization step.
This kind of alternating update is efﬁcient, because there is no need to completely solve the policy optimization subproblem, before updating the reward parameters.
It has been used in many other
RL related settings as well.
For example, the well-known actor-critic (AC) algorithm for policy optimization [31, 32, 23] alternates between one step of policy update, and one step of critic parameter update.
Below we present the details of our algorithm at a given iteration k .
Policy Improvement Step.
Let us consider optimizing the lower-level problem, when the reward parameter ✓ k is held ﬁxed.
Towards this end, deﬁne the so-called soft Q and soft value functions for a given policy ⇡ k and a reward parameter ✓ k :
[Equation]
We will adopt the well-known soft policy iteration [21] to optimize the lower-level problem (3a).
Under the current reward parameter ✓ k and the policy ⇡ k , the soft policy iteration generates a new policy ⇡ k +1 as follows
[Equation]
Under a ﬁxed reward function, it can be shown that the new policy ⇡ k +1 monotonically improves ⇡ k , and it converges linearly to the optimal policy; see [21, Theorem 4] and [28, Thoerem 1].
Note that in practice, we usually do not have direct access to the exact soft Q-function in (4b).
In order to perform the policy improvement, a few stochastic update steps in soft Q-learning [21] or soft Actor-Critic (SAC) [22] could be used to replace the one-step soft policy iteration (5).
In the appendix, we present Alg. 2 to demonstrate such practical implementation of our proposed algorithm.
Reward Optimization Step.
We propose to use a stochastic gradient-type algorithm to optimize ✓ .
Towards this end, let us ﬁrst derive the exact gradient r L ( ✓ ) .
See Appendix D for detailed proof.
Lemma 1.
The gradient of the likelihood function L ( ✓ ) can be expressed as follows:
[Equation]
To obtain stochastic estimators of the exact gradient r L ( ✓ k ) , we take two approximation steps: 1) approximate the optimal policy ⇡ ✓ k by ⇡ k +1 in (5), since the optimal policy ⇡ ✓ k is not available throughout the algorithm; 2) sample one expert trajectory ⌧ E k which is already generated by the expert policy ⇡ E ; 3) sample one trajectory ⌧ A k from the current policy ⇡ k +1 .
Following the approximation steps mentioned above, we construct a stochastic estimator g k to approximate the exact gradient r L ( ✓ k ) in (6) as follows:
[Equation]
With the stochastic gradient estimator g k , the reward parameter ✓ k is updated as:
[Equation]
where ↵ is the stepsize in updating the reward parameter.
In summary, the proposed algorithm for solving the ML-IRL problem (ML-IRL) is given in Alg. 1.
Theoretical Analysis
In this section, we present ﬁnite-time guarantees for the proposed algorithm.
To begin with, ﬁrst recall that in Sec. 3, we have mentioned that (ML-IRL) is a bi-level problem, where the upper level (resp.
the lower level) problem optimizes the reward parameter (resp. the policy).
In order to solve (ML-IRL), our algorithm 1 has a singe-loop structure, which alternates between one step of policy update and one step of the reward parameter update.
Such a single-loop structure indeed has computational beneﬁt, but it also leads to potential unstableness, since the lower level problem can stay far away from its true solutions.
Speciﬁcally, at each iteration k , the potential unstableness is induced by the distribution mismatch between the policy ⇡ k +1 and ⇡ ✓ k , when we use estimator g k (7) to approximate the exact gradient r L ( ✓ k ) (6) in updating the reward parameter ✓ k .
Towards stabilizing the algorithm, we adopt the so-called two-timescale stochastic approximation (TTSA) approach [33, 23], where the lower-level problem updates in a faster time-scale (i.e., converges faster) compared with its upper-level counterpart.
Intuitively, the TTSA enables the ⇡ k +1 tracks the optimal ⇡ ✓ k , leading to a stable algorithm.
In the proposed Algorithm 1, the policy (lower-level variable) is continuously updated by the soft policy iteration (5), and it is ‘fast’ because it converges linearly to the optimal policy under a ﬁxed reward function [28, Theorem 1].
On the other hand, the reward parameter update (8) does not have such linear convergence property, therefore it works in a ‘slow’ timescale.
To begin our analysis, let us ﬁrst present a few technical assumptions.
Assumption 1 (Ergodicity) .
For any policy ⇡ , assume the Markov chain with transition kernel P is irreducible and aperiodic under policy ⇡ .
Then there exist constants  > 0 and ⇢ 2 (0 , 1) such that
[Equation]
where k · k TV is the total variation (TV) norm; µ ⇡ is the stationary state distribution under ⇡ .
Assumption 1 assumes the Markov chain mixes at a geometric rate.
It is a common assumption in the iterature of RL [34, 35, 32], which holds for any time-homogeneous Markov chain with ﬁnite-state space or any uniformly ergodic Markov chain with general state space.
Assumption 2.
For any s 2 S , a 2 A and any reward parameter ✓ , the following holds:
[Equation]
where L r and L g are positive constants.
Assumption 2 assumes that the parameterized reward function has bounded gradient and is Lipschitz smooth.
Such assumption in Lipschitz property are common in the literature of min-max / bi-level optimization [36, 23, 37, 25, 38].
Based on Assumptions 1 - 2, we next provide the following Lipschitz properties:
Lemma 2.
Suppose Assumptions 1 - 2 hold.
For any reward parameter ✓ 1 and ✓ 2 , the following results hold:
[Equation]
where Q soft r ✓ , ⇡ ✓ ( · , · ) denotes the soft Q-function under the reward function r ( · , · ; ✓ ) and the policy ⇡ ✓ .
The positive constants L q and L c are deﬁned in Appendix E.
The Lipschitz properties identiﬁed in Lemma 2 are vital for the convergence analysis.
Then we present the main results, which show the convergence speed of the policy { ⇡ k } k  0 and the reward parameter { ✓ k } k  0 in the Alg. 1.
Please see Appendix E for the detailed proof.
Theorem 2.
Suppose Assumptions 1 - 2 hold.
Selecting stepsize ↵ :=
↵ 0 K  for the reward update step (8) where ↵ 0 > 0 and  2 (0 , 1) are some ﬁxed constants, and K is the total number of iterations to be run by the algorithm.
Then the following result holds:
[Equation]
where we denote k log ⇡ k +1  log
[Equation] [Equation]
max
[Equation]
[Equation]
.
In particular, setting  = 1 / 2 , then both quantities in (11a) and (11b) converge with the rate O
( K  1 / 2 ) .
In Theorem 2, we present the ﬁnite-time guarantee for the convergence of the Alg.1.
Moreover, as a special case, when the reward is parameterized as a linear function, we know that (ML-IRL) is concave and Theorem 2 provides a stronger guarantee which identify the global optimal reward estimator in ﬁnite time.
We provide a proof sketch below to present the key steps.
The detailed proof is in Appendix H.
Proof sketch.
We outline our main steps in analyzing (11a) and (11b) respectively.
In order to show the convergence of policy estimates in (11a), there are several key steps.
First, we note that both policies ⇡ k +1 and ⇡ ✓ k are in the softmax parameterization, where ⇡ k +1 ( ·| s ) / exp
[Equation] [Equation]
and ⇡
[Equation]
[Equation]
[Equation] [Equation] s, · [Equation]
. Then, we can show a Lipschitz continuity property between the policy and the soft Q-function:
[Equation]
where the inﬁnty norm k · k 1 is deﬁned over the state-action space S ⇥ A .
Moreover, by analyzing the contraction property of the soft policy iteration (5), we bound
[Equation]
To ensure that the error term k ✓ k  ✓ k  1 k is small, we select the stepsize of reward parameters as ↵ :=
↵ K  , where K is the total number of iterations and  > 0 .
Then, by combining previous two steps, we could further show the convergence rate of the policy estimates in (11a).
To prove the convergence of the reward parameters in (11b), we ﬁrst leverage the Lipschitz smooth property of L ( ✓ ) in (10b).
However, one technical challenge in the convergence analysis is how to handle the bias between the gradient estimator g k deﬁned in (7) and the exact gradient r L ( ✓ k ) .
When we construct the gradient estimator g k in (7), we need to sample trajectories from the current policy ⇡
k +1 and the expert dataset D .
However, according to the expression of r L ( ✓ k ) in (6), the trajectories are sampled from the optimal policy ⇡ ✓ k and the dataset D .
Hence, there is a distribution mismatch between ⇡ k +1 and ⇡ ✓ k .
Our key idea is to leverage (11a) to handle this distribution mismatch error, and thus show that the bias between g k and r L ( ✓ k ) could be controlled.
To the best of our knowledge, Theorem 2 is the ﬁrst non-asymptotic convergence result for IRL with nonlinear reward parameterization.
A Discussion over State-Only Reward
In this section we consider the IRL problems modeled by using rewards that are only a function of the state.
A lower dimensional representation of the agent’s preferences (i.e. in terms only of states as opposed to states and actions) is more likely to facilitate counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks.
This is because the estimation of preferences which are only deﬁned in terms of states is less sensitive to the speciﬁc environment dynamics in the expert’s demonstration dataset.
Moreover, in application such as healthcare [39] and autonomous driving [40], where simply imitating the expert policy can potentially result in poor performance, since the learner and the expert may have different transition dynamics.
Similar points have also been argued in recent works [14, 41–43].
Next, let us brieﬂy discuss how we can understand (ML-IRL) and Alg.1, when the reward is parameterized as a state-only function.
First, it turns out that there is an equivalent formulation of (ML-IRL), when the expert trajectories only contain the visited states.
Lemma 3.
Suppose the expert trajectories ⌧ is sampled from a policy ⇡ E , and the reward is parameterized as a state-only function r ( s ; ✓ ) .
Then ML-IRL is equivalent to the following:
[Equation]
Please see Appendix F for the detailed derivation.
Intuitively, the above lemma says that, when dealing with the state-only IRL, (ML-IRL) minimizes the gap between the soft value functions of the optimal policy ⇡ ✓ and that of the expert policy ⇡ E .
Moreover, Alg.1 can also be easily implemented with the state-only reward.
In fact, the entire algorithm essentially stays the same, and the only change is that r ( s, a ; ✓ ) will be replaced by r ( s ; ✓ ) .
In this way, by only using the visited states in the trajectories, one can still compute the stochastic gradient estimator in (7).
Therefore, even under the state-only IRL setting where the expert dataset only contains visited states, our formulation and the proposed algorithm still work if we parameterize the reward as a state-only function.
Numerical Results
In this section, we test the performance of our algorithm on a diverse collection of RL tasks and environments.
In each experiment set, we train algorithms until convergence and average the scores of the trajectories over multiple random seeds.
The hyperparameter settings and simulation details are provided in Appendix B.
MuJoCo Tasks For Inverse Reinforcement Learning.
In this experiment set, we test the performance of our algorithm on imitating the expert behavior.
We consider several high-dimensional robotics control tasks in MuJoCo [44].
Two class of existing algorithms are considered as the comparison baselines: 1) imitation learning algorithms that only learn the policy to imitate the expert, including Behavior Cloning (BC) [45] and Generative Adversarial Imitation Learning (GAIL) [10]; 2) IRL algorithms which learn a reward function and a policy simultaneously, including
Adversarial Inverse Reinforcement Learning (AIRL) [11], f -IRL [14] and IQ-Learn [2].
To ensure fair comparison, all imitation learning / IRL algorithms use soft Actor-Critic [22] as the base RL algorithm.
For the expert dataset, we use the data provided in the ofﬁcial implementation 2 of f -IRL.
In this experiment, we implement two versions of our proposed algorithm: ML-IRL(State-Action) where the reward is parameterized as a function of state and action; ML-IRL(State-Only) which utilizes the state-only reward function.
In Table 1, we present the simulation results under a limited data regime where the expert dataset only contains a single expert trajectory.
The scores (cumulative rewards) reported in the table is averaged over 6 random seeds.
In each random seed, we train algorithm from initialization and collect 20 trajectories to average their cumulative rewards after the algorithms converge.
The results reported in Table 1 show that our proposed algorithms outperform the baselines.
The numerical results with conﬁdence intervals are in Table 3 (See Appendix).
We observe that BC fails to imitate the expert’s behavior.
It is due to the fact that BC is based on supervised learning and thus could not learn a good policy under such a limited data regime.
Moreover, we notice the training of IQ-Learn is unstable, which may be due to its inaccurate approximation to the soft Q-function.
Therefore, in the MuJoCo tasks where IQ-Learn does not perform well, so that we cannot match the results presented in the original paper [2], we directly report results from there (and mark them by ⇤ in Table 1).
The results of AIRL are not presented in Table 1 since it performs poorly even after spending signiﬁcant efforts in parameter tuning (similar observations have been made in in [46, 14]).
Transfer Learning Across Changing Dynamics.
We further evaluate IRL algorithms on the transfer learning setting.
We follows the environment setup in [11], where two environments with different dynamics are considered: Custom-Ant vs Disabled-Ant .
We compare ML-IRL(State-Only) with several existing IRL methods: 1) AIRL [11], 2) f
-IRL [14]; 3) IQ-Learn [2].
[Table]
We consider two transfer learning settings: 1) data transfer; 2) reward transfer.
For both settings, the expert dataset / trajectories are generated in Custom-Ant .
In the data transfer setting, we train IRL agents in Disabled-Ant by using the expert trajectories, which are generated in Custom-Ant .
In the reward transfer setting, we ﬁrst use IRL algorithms to infer the reward functions in Custom-Ant , and then transfer these recovered reward functions to Disabled-Ant for further evaluation.
In both settings, we also train SAC with the ground-truth reward in Disabled-Ant and report the scores.
The numerical results are reoprted in Table 2.
the proposed ML-IRL(State-Only) achieves superior performance compared with the existing IRL benchmarks in both settings.
We notice that IQ-Learn fails in both settings since it indirectly recovers the reward function from a soft Q-function approximator, which could be inaccurate and is highly dependent upon the environment dynamics.
Therefore, the reward function recovered by IQ-Learn can not be disentangled from the expert actions and environment dynamics, which leads to its failures in the transfer learning tasks
[Table]
Conclusion
In this paper, we present a maximum likelihood IRL formulation and propose a provably efﬁcient algorithm with a single-loop structure.
To our knowledge, we provide the ﬁrst non-asymptotic analysis for IRL algorithm under nonlinear reward parameterization.
As a by-product, when we parameterize the reward as a state-only function, our algorithm could work in state-only IRL setting and enable reward transfer to new environments with different dynamics.
Our algorithm outperforms existing IRL methods on high-dimensional robotics control tasks and corresponding transfer learning settings.
A limitation of our method is the requirement for online training, so one future direction of this work is to further extend our algorithm and the theoretical analysis to the ofﬂine IRL setting.
Potential Negative Social Impacts
Since IRL methods aim to recover the reward function and the associated optimal policy from the observed expert dataset, potential negative social impacts may occur if there are bad demonstrations included in the expert dataset.
Thus, for sensitive applications such as autonomous driving and clinical decision support, additional care should be taken to avoid negative biases from the expert demonstrations and ensure safe adaptation.
Acknowledgments
We thank the anonymous reviewers for their valuable comments and suggestions. M. Hong and S. Zeng are partially supported by NSF grants CIF-1910385, CMMI-1727757, and AFOSR grant 19RT0424.
A. Garcia would like to acknowledge partial support from grant FA9550-19-1-00347 by
AFOSR.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]