Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Inverse reinforcement learning (IRL) aims to recover the reward function andthe associated optimal policy that best ﬁts observed sequences of states andactions implemented by an expert.
Many algorithms for IRL have an inherentnested structure: the inner loop ﬁnds the optimal policy given parametrizedrewards while the outer loop updates the estimates towards optimizing a measureof ﬁt.
For high dimensional environments such nested-loop structure entailsa signiﬁcant computational burden.
To reduce the computational burden of anested loop, novel methods such as SQIL [1] and IQ-Learn [2] emphasize policyestimation at the expense of reward estimation accuracy.
However, withoutaccurate estimated rewards, it is not possible to do counterfactual analysis suchas predicting the optimal policy under different environment dynamics and/orlearning new tasks.
In this paper we develop a novel single-loop algorithm for
IRL that does not compromise reward estimation accuracy.
In the proposedalgorithm, each policy improvement step is followed by a stochastic gradientstep for likelihood maximization.
We show that the proposed algorithm provablyconverges to a stationary solution with a ﬁnite-time guarantee.
If the reward isparameterized linearly, we show the identiﬁed solution corresponds to the solutionof the maximum entropy IRL problem.
Finally, by using robotics control problemsin Mujoco and their transfer settings, we show that the proposed algorithm achievessuperior performance compared with other IRL and imitation learning benchmarks.
Introduction
Given observed trajectories of states and actions implemented by an expert, we consider the problemof estimating the reinforcement learning environment in which the expert was trained.
This problem isgenerally referred to as inverse reinforcement learning (IRL) (see [3] for a recent survey).
Assumingthe environment dynamics are known (or available online), the IRL problem consists of estimatingthe reward function and the expert’s policy (optimizing such rewards) that best ﬁts the data.
Whilethere are limitations on the identiﬁability of rewards [4], the estimation of rewards based upon experttrajectories enables important counterfactual analysis such as the estimation of optimal policies underdifferent environment dynamics and/or reinforcement learning of new tasks.
In the seminal work [5], the authors developed an IRL formulation, in which the model for the expert’sbehavior is the policy that maximizes entropy subject to a constraint requiring that the expectedfeatures under such policy match the empirical averages in the expert’s observation dataset.
Thealgorithms developed for MaxEnt-IRL [5–7] have a nested loop structure, alternating between anouter loop with a reward update step, and an inner loop that calculates the explicit policy estimates.
The computational burden of this nested structure is manageable in tabular environments, but itbecomes signiﬁcant in high dimensional settings requiring function approximation.
Towards developing more efﬁcient IRL algorithms, a number of works [8–11] propose to leveragethe idea of adversarial training [12].
These algorithms learn a non-stationary reward function throughtraining a discriminator, which is then used to guide the policy to match the behavior trajectoriesfrom the expert dataset.
However, [13] pointed out that the resulting discriminator (hence thereward function) typically cannot be used in new learning tasks, since it is highly dependent on thecorresponding policy and current environment dynamics.
Moreover, due to the brittle approximationtechniques and sensitive hyperparameter choice in the adversarial training, these IRL algorithms canbe unstable.
More recent works [1, 2] have developed algorithms to alleviate the computational burden of thenested-loop training procedures.
In [1], the authors propose to model the IRL using certain maximumentropy RL problem with speciﬁc reward function (which assigns r = +1 for matching expertdemonstrations and r = 0 for all other behaviors).
Then a soft Q imitation learning (SQIL) algorithmis developed.
In [2], the authors propose to transform the standard formulation of IRL (discussedabove) into a single-level problem, through learning a soft Q-function to implicitly represent thereward function and the policy.
An inverse soft-Q learning (IQ-Learn) algorithm is then developed,which is shown to be effective in estimating the policy for the environment that it is trained on.
Despite being computational efﬁcient, IQ-Learn sacriﬁces the accuracy in estimating the rewardssince it indirectly recovers rewards from a soft Q-function approximator which is highly dependentupon the environment dynamics and does not strictly satisfy the soft-Bellman equation.
Therefore itis not well-suited for counterfactual prediction or transfer learning setting.
Finally, in f -IRL [13] the authors consider an approach for estimating rewards based on theminimization of several measures of divergence with respect to the expert’s state visitation measure.
The approach is limited to estimating rewards that only depend on state.
Moreover, while the resultsreported are based upon a single-loop implementation, the paper does not provide a convergenceguarantee to support performance.
We refer the readers to Appendix A for other related works.
Our Contributions.
The goal of this work is to develop an algorithm for IRL which is capable ofproducing high-quality estimates of both rewards and behavior policies with ﬁnite-time guarantees.
The major contributions of this work are listed below.
• We consider a formulation of IRL based on maximum likelihood (ML) estimation over optimal(entropy-regularized) policies, and prove that a strong duality relationship with maximum entropyIRL holds if rewards are represented by a linear combination of features.
1 The ML formulation isa bi-level optimization problem, where the upper-level problem maximizes the likelihood function,while the lower-level ﬁnds the optimal policy under the current reward parameterization.
Such abi-level structure is not only instrumental to the subsequent algorithm design, but is also ﬂexible toincorporate the use of state-only, as well as the regular reward function (which depends on the stateand action pair).
The former is suitable for transfer learning since it is insensitive to the changes ofthe environment dynamics, while the latter can be used to efﬁciently imitate the expert policy.•
Based on the ML-IRL formulation, we develop an efﬁcient algorithm.
To avoid the computationalburden of repeatedly solving the lower-level policy optimization problem, the proposed algorithmhas a single-loop structure where the policy improvement step and reward optimization step areperformed alternatingly so that each step can be performed relatively cheaply.
Further, we show thatthe algorithm has strong theoretical guarantees: to achieve certain ✏ -approximate stationary solutionfor a non-linearly parameterized problem, it requires O ( ✏  2 ) steps of policy and reward updates each.
To our knowledge, it is the ﬁrst algorithm which has ﬁnite-time guarantee for the IRL problem undernonlinear parameterization of reward functions.
• We conduct extensive experiments to demonstrate that the proposed algorithm outperforms manystate-of-the-art IRL algorithms in both policy estimation and reward recovery.
In particular, whentransferring to a new environment, RL algorithms using rewards recovered by the proposed algorithmoutperform those that use rewards recovered from existing IRL and imitation learning benchmarks.
Preliminaries
In this section, we review the fundamentals of the maximum entropy inverse reinforcement learning(MaxEnt-IRL).
We consider an MDP deﬁned by the tuple ( S , A , P , ⌘ , r,  ) ; S and A denote thestate space and the action space respectively; P ( s
| s, a ) :
S ⇥
A ⇥ S !
[0 , 1] denotes the transitionprobability; ⌘ ( · ) denotes the distribution for the initial state; r ( s, a ) : S ⇥ A !
R is the rewardfunction and  is a discount factor.
Given a dataset D which contains trajectories ⌧ = { ( s t , a t ) } 1 t =0 sampled from an expert policy, the
MaxEnt-IRL formulation [6, 17, 18] consists of ﬁnding a policy maximizing entropy subject to theexpected features under such policy matching the empirical averages in the expert’s observationdataset.
Speciﬁcally, the MaxEnt-IRL formulation is given by:
[Equation]
where  ( s, a ) is the feature vector of the state-action pair ( s, a ) .
Let ✓ denote the dual variable forthe linear constraint, then the Lagrangian of (MaxEnt-IRL) is given by
[Equation]
In [6, 17, 18], the authors proposed a "dual descent" algorithm, which alternates between i) solvingmax ⇡ L ( ⇡ , ✓ ) for ﬁxed ✓ , and ii) a gradient descent step to optimize the dual variable ✓ .
It isshown that the optimizer ⇡ ⇤ ✓ in step i) can be recursively deﬁned as ⇡ ⇤ ✓ ( a t | s t ) =
Z at | st, ✓ Z st, ✓ , wherelog
[Equation]
, ✓
[Equation] [Equation]
From a computational perspective, the above algorithm is not efﬁcient: it has a nested-loop structure,which repeatedly computes the optimal policy ⇡ ⇤ ✓ under each variable ✓ .
It is known that when theunderlying MDP is of high-dimension, such an algorithm can be computationally prohibitive [9, 10].
Recent work [2] proposed an algorithm called IQ-Learn to improve upon the MaxEnt-IRL byconsidering a saddle-point formulation:
[Equation]
where r ( s t , a t ) is the reward associated with state-action pair ( s t , a t ) .
The authors show that thisproblem can be transformed into an optimization problem only deﬁned in terms of the soft Q-function,which implicitly represents both reward and policy.
IQ-Learn is shown to be effective in imitating theexpert behavior while only relying on the estimation of the soft Q-function.
However, the implicitreward estimate obtained is not necessarily accurate since its soft Q-function estimate depends on theenvironment dynamics and does not strictly satisfy the soft-Bellman equation.
Hence, it is difﬁcult totransfer the recovered reward function to new environments.
Problem Formulation
In this section, we consider a ML formulation of the IRL problem and formalize a duality relationshipwith maximum entropy-based formulation (MaxEnt-IRL).
Maximum Log-Likelihood IRL (ML-IRL)
Let D := { ⌧ i } Ni =1 denote the dataset containing observed trajectories.
A model of the expert’sbehavior is a randomized policy ⇡ ✓ ( ·| s ) where ✓ is a parameter vector.
Assuming the state dynamics P ( s t +1 | s t , a t ) are known, the discounted log-likelihood of observing a sample trajectory ⌧ undermodel ⇡ ✓ can be written follows:
[Equation]
Let L ( ✓ ) := E ⌧ ⇠ D ⇥ P t  0  t log ⇡ ✓ ( a t | s t )
⇤ . We consider the following maximum log-likelihood IRL formulation:
[Equation]
where r ( s, a ; ✓ ) is the reward function and H ( ⇡ ( ·| s ))
:=
P a 2 A ⇡ ( a | s ) log ⇡ ( a | s ) denotes theentropy of policy ⇡ ( ·| s ) .
We now make some remarks about ML-IRL.
First, the problem takes the form of a bi-leveloptimization problem, where the upper-level problem (ML-IRL) optimizes the reward parameter ✓ ,while the lower-level problem describes the expert’s policy as the solution to an entropy-regularized MDP ([19, 20]).
In what follows we will leverage recently developed (stochastic) algorithms forbi-level optimization [21–23], that avoid the high complexity resulted from nested loop algorithms.
Second, it is reasonable to use the ML function as the loss, because it searches for a reward functionwhich generates a behavior policy that can best ﬁt the expert demonstrations.
While the ML functionhas been considered in [24, 25], they rely on heuristic algorithms with nested-loop computations tosolve their IRL formulations, and the theoretical properties are not studied.
Finally, the lower-levelproblem has been well-studied in the literature [19, 20, 26–28].
The entropy regularization in (3a)ensures the uniqueness of the optimal policy ⇡ ✓ under the ﬁxed reward function r ( s, a ; ✓ ) [19, 26].
Even when the underlying MDP is high-dimensional and/or complex, the optimal policy could stillbe obtained; see recent developments in [19, 20].
Subsequently, we will leverage these algorithms,together with recent advances in bi-level optimization to develop solution methods for (ML-IRL).
We close this section by formally establishing a connection between (MaxEnt-IRL) and (ML-IRL).
Theorem 1.
(Strong Duality) Suppose that the reward function is given as: r ( s, a ; ✓ ) :
=
( s, a ) T ✓ ,for all s 2 S and a 2 A .
Then (ML-IRL) is the Lagrangian dual of (MaxEnt-IRL) .
Furthermore,strong duality holds, that is: L ( ✓ ⇤ )
=
H ( ⇡ ⇤ ) , where ✓ ⇤ and ⇡ ⇤ are the global optimal solutions forproblems (ML-IRL) and (MaxEnt-IRL) , respectively.
The proof of Theorem 1 is relegated to Appendix G. To our knowledge this result which speciﬁcallyaddresses the (MaxEnt-IRL) formulation is novel.
A duality between ML estimation and maximumcausal entropy is obtained in [17, Theorem 3].
However, the problem considered in that paper is notin RL nor IRL setting, therefore they cannot be directly used in the context of the present paper.
The above duality result reveals a strong connection between the two formulations under linear rewardparameterization.
Of course, when non-linear parameterization is used strong duality may not hold. However, this connection still suggests that ML-IRL formulation is a meaningful framework for IRL.
The Proposed Algorithm
In this section, we design algorithms for (ML-IRL).
Recall that one major drawback of algorithmsfor (MaxEnt-IRL) is that, they repeatedly solve certain policy optimization problem in the inner loop.
Initialize reward parameter ✓ 0 and policy ⇡ 0 .
Set the reward parameter’s stepsize as ↵ .
P t  0  t a
For example, the algorithm IQ-Learn [2] improves the computational efﬁciency through implicitlyrepresenting the reward function and the policy by a Q-function approximator, while the estimationaccuracy of the recovered reward is sacriﬁced.
Therefore, one important goal of our design is to ﬁndprovably efﬁcient algorithms that can avoid high-complexity operations and accurately recover thereward function.
Speciﬁcally, it is desirable that the resulting algorithm only uses a ﬁnite number ofreward and policy updates to reach certain high-quality solutions.
To proceed, we will leverage the special bi-level structure of the ML-IRL problem.
The idea is toalternate between one step of policy update to improve the solution of the lower-level problem, andone step of the parameter update which improves the upper-level loss function.
At each iteration k ,given the current policy ⇡ k and the reward parameter ✓ k , a new policy ⇡ k +1 is generated from thepolicy improvement step, and ✓ k +1 is generated by the reward optimization step.
This kind of alternating update is efﬁcient, because there is no need to completely solve the policyoptimization subproblem, before updating the reward parameters.
It has been used in many other
RL related settings as well.
For example, the well-known actor-critic (AC) algorithm for policyoptimization [29, 30, 21] alternates between one step of policy update, and one step of critic parameterupdate.
Below we present the details of our algorithm at a given iteration k .
Policy Improvement Step.
Let us consider optimizing the lower-level problem, when the rewardparameter ✓ k is held ﬁxed.
Towards this end, deﬁne the so-called soft Q and soft value functions for agiven policy ⇡ k and a reward parameter ✓ k :
[Equation]
We will adopt the well-known soft policy iteration [19] to optimize the lower-level problem (3a).
Under the current reward parameter ✓ k and the policy ⇡ k , the soft policy iteration generates a newpolicy ⇡ k +1 as follows
[Equation]
[Equation] Under a ﬁxed reward function, it can be shown that the new policy ⇡ k +1 monotonically improves ⇡ k ,and it converges linearly to the optimal policy; see [19, Theorem 4] and [26, Thoerem 1].
Note that in practice, we usually do not have direct access to the exact soft Q-function in (4b).
Inorder to perform the policy improvement, a few stochastic update steps in soft Q-learning [19] or soft
Actor-Critic (SAC) [20] could be used to replace the one-step soft policy iteration (5).
In the appendix,we present Alg. 2 to demonstrate such alternative implementation of our proposed algorithm.
Reward Optimization Step.
We propose to use a stochastic gradient-type algorithm to optimize ✓ .
Towards this end, let us ﬁrst derive the exact gradient r L ( ✓ ) .
See Appendix D for detailed proof.
Lemma 1.
The gradient of the likelihood function L ( ✓ ) can be expressed as follows:
[Equation]
To obtain stochastic estimators of the exact gradient r L ( ✓ k ) , we take two approximation steps: 1)approximate the optimal policy ⇡ ✓ k by ⇡ k +1 in (5), since the optimal policy ⇡ ✓ k is not availablethroughout the algorithm; 2) sample one expert trajectory ⌧ Ek from the dataset D and one agenttrajectory ⌧ Ak from the current policy ⇡ k
+1 so to approximate the expectation operators in (6).
Following the approximation steps mentioned above, we construct a stochastic estimator g k toapproximate the exact gradient r L ( ✓ k ) in (6) as follows:
[Equation]
where
[Equation]
With the stochastic gradient estimator g k , the reward parameter ✓ k is updated as:
[Equation]
where ↵ is the stepsize in updating the reward parameter.
In summary, the proposed algorithm for solving the ML-IRL problem (ML-IRL) is given in Alg. 1.
Theoretical Analysis
In this section, we present ﬁnite-time guarantees for the proposed algorithm.
To begin with, ﬁrst recall that in Sec. 3, we have mentioned that (ML-IRL) is a bi-level problem,where the upper level (resp.
the lower level) problem optimizes the reward parameter (resp. thepolicy).
In order to solve (ML-IRL), our algorithm 1 has a singe-loop structure, which alternatesbetween one step of policy update and one step of the reward parameter update.
Such a single-loopstructure indeed has computational beneﬁt, but it also leads to potential unstableness, since the lowerlevel problem can stay far away from its true solutions.
Speciﬁcally, at each iteration k , the potentialunstableness is induced by the distribution mismatch between the policy ⇡ k +1 and ⇡ ✓ k , when we useestimator g k (7) to approximate the exact gradient r L ( ✓ k ) (6) in updating the reward parameter ✓ k .
Towards stabilizing the algorithm, we adopt the so-called two-timescale stochastic approximation(TTSA) approach [31, 21], where the lower-level problem updates in a faster time-scale (i.e.,converges faster) compared with its upper-level counterpart.
Intuitively, the TTSA enables the⇡ k +1 tracks the optimal ⇡ ✓ k , leading to a stable algorithm.
In the proposed Algorithm 1, the policy(lower-level variable) is continuously updated by the soft policy iteration (5), and it is ‘fast’ because itconverges linearly to the optimal policy under a ﬁxed reward function [26, Theorem 1].
On the otherhand, the reward parameter update (8) does not have such linear convergence property, therefore itworks in a ‘slow’ timescale.
To begin our analysis, let us ﬁrst present a few technical assumptions.
Assumption 1 (Ergodicity) .
For any policy ⇡ , assume the Markov chain with transition kernel P isirreducible and aperiodic under policy ⇡ .
Then there exist constants  > 0 and ⇢ 2 (0 , 1) such that
[Equation]
where k · k TV is the total variation (TV) norm; µ ⇡ is the stationary state distribution under ⇡ .
Assumption 1 assumes the Markov chain mixes at a geometric rate.
It is a common assumption in theiterature of RL [32, 33, 30], which holds for any time-homogeneous Markov chain with ﬁnite-statespace or any uniformly ergodic Markov chain with general state space.
Assumption 2.
For any s 2 S , a 2 A and any reward parameter ✓ , the following holds:
[Equation]
9b) where L r and L g are positive constants.
Assumption 2 assumes that the parameterized reward function has bounded gradient and is Lipschitzsmooth.
Such assumption in Lipschitz property are common in the literature of min-max / bi-leveloptimization [34, 21, 35, 23].
Based on Assumptions 1 - 2, we next provide the following Lipschitz properties:
Lemma 2.
Suppose Assumptions 1 - 2 hold.
For any reward parameter ✓ 1 and ✓ 2 , the followingresults hold:
[Equation]
where Q soft r ✓ , ⇡ ✓ ( · , · ) denotes the soft Q-function under the reward function r ( · , · ; ✓ ) and the policy ⇡ ✓ .
The positive constants L q and L c are deﬁned in Appendix E.
The Lipschitz properties identiﬁed in Lemma 2 are vital for the convergence analysis.
Then wepresent the main results, which show the convergence speed of the policy { ⇡ k } k  0 and the rewardparameter { ✓ k } k  0 in the Alg. 1.
Please see Appendix E for the detailed proof.
Theorem 2.
Suppose Assumptions 1 - 2 hold.
Selecting stepsize ↵ :=
↵ 0 K  for the reward update step(8) where ↵ 0 > 0 and  2 (0 , 1) are some ﬁxed constants, and K is the total number of iterations tobe run by the algorithm.
Then the following result holds:
[Equation]
where we denote k log ⇡ k +1  log ⇡ ✓ k k 1 :
= max s 2 S ,a 2 A  log ⇡ k +1 ( a | s )
[Equation]
log ⇡ ✓ k ( a | s )  .
Inparticular, setting  = 1 / 2 , then both quantities in (11a) and (11b) converge with the rate O
( K  1 / 2 ) .
In Theorem 2, we present the ﬁnite-time guarantee for the convergence of the Alg.1.
As a remark, wenote that our theoretical guarantee is different from the existing results, such as [26] that showed theconvergence rate of soft policy iteration under a ﬁxed reward function. Theorem 2 analyzes a morechallenging setting where both the policy and reward parameter are kept changing.
To present the keysteps in our analysis, we provide a proof sketch below.
The detailed proof is in Appendix H.
Proof sketch.
We outline our main steps in analyzing (11a) and (11b) respectively.
In order to show the convergence of policy estimates in (11a), there are several key steps.
First, we note that both policies ⇡ k +1 and ⇡ ✓ k are in the softmax parameterization, where ⇡ k +1 ( ·| s ) / exp
[Equation]
and ⇡
[Equation]
Q soft r ✓ k , ⇡
[Equation]
.
Then, we can show a Lipschitz continuity property between the policy and the soft Q-function:
[Equation]
where the inﬁnty norm k · k 1 is deﬁned over the state-action space S ⇥ A .
Moreover, by analyzingthe contraction property of the soft policy iteration (5), we bound k Q soft r ✓ k , ⇡ k  Q soft r ✓ k , ⇡ ✓ k k 1 as:
[Equation]
To ensure that the error term k ✓ k  ✓ k  1 k is small, we select the stepsize of reward parameters as↵ :
= ↵ 0 K  , where K is the total number of iterations and  > 0 .
Then, by combining previous twosteps, we could further show the convergence rate of the policy estimates in (11a).
To prove the convergence of the reward parameters in (11b), we ﬁrst leverage the Lipschitz smoothproperty of L ( ✓ ) in (10b).
However, one technical challenge in the convergence analysis is howto handle the bias between the gradient estimator g k deﬁned in (7) and the exact gradient r L ( ✓ k ) .
When we construct the gradient estimator g k in (7), we need to sample trajectories from the currentpolicy
⇡ k +1 and the expert dataset D .
However, according to the expression of r L ( ✓ k ) in (6), thetrajectories are sampled from the optimal policy ⇡ ✓ k and the dataset D .
Hence, there is a distributionmismatch between ⇡ k +1 and ⇡ ✓ k .
Our key idea is to leverage (11a) to handle this distributionmismatch error, and thus show that the bias between g k and r L ( ✓ k ) could be controlled.
To the best of our knowledge, Theorem 2 is the ﬁrst non-asymptotic convergence result for IRL withnonlinear reward parameterization.
A Discussion over State-Only Reward
In this section we consider the IRL problems modeled by using rewards that are only a functionof the state.
A lower dimensional representation of the agent’s preferences (i.e. in terms only ofstates as opposed to states and actions) is more likely to facilitate counterfactual analysis such aspredicting the optimal policy under different environment dynamics and/or learning new tasks.
Thisis because the estimation of preferences which are only deﬁned in terms of states is less sensitive tothe speciﬁc environment dynamics in the expert’s demonstration dataset.
Moreover, in applicationsuch as healthcare [36] and autonomous driving [37], where simply imitating the expert policy canpotentially result in poor performance, since the learner and the expert may have different transitiondynamics.
Similar points have also been argued in recent works [38, 13, 39].
Next, let us brieﬂy discuss how we can understand (ML-IRL) and Alg.1, when the reward isparameterized as a state-only function.
First, it turns out that there is an equivalent formulation of(ML-IRL), when the expert trajectories only contain the visited states.
Lemma 3.
Suppose the expert trajectories ⌧ is sampled from a policy ⇡ E , and the reward isparameterized as a state-only function r ( s ; ✓ ) .
Then ML-IRL is equivalent to the following:
[Equation]
Please see Appendix F for the detailed derivation.
Intuitively, the above lemma says that, whendealing with the state-only IRL, (ML-IRL) minimizes the gap between the soft value functions of theoptimal policy ⇡ ✓ and that of the expert policy ⇡ E .
Moreover, Alg.1 can also be easily implementedwith the state-only reward.
In fact, the entire algorithm essentially stays the same, and the onlychange is that r ( s, a ; ✓ ) will be replaced by r ( s ; ✓ ) .
In this way, by only using the visited states in thetrajectories, one can still compute the stochastic gradient estimator in (7).
Therefore, even under thestate-only IRL setting where the expert dataset only contains visited states, our formulation and theproposed algorithm still work if we parameterize the reward as a state-only function.
Numerical Results
In this section, we test the performance of our algorithm on a diverse collection of RL tasks andenvironments.
In each experiment set, we train algorithms until convergence and average the scoresof the trajectories over multiple random seeds.
The hyperparameter settings and simulation detailsare provided in Appendix B.
Mujoco Tasks For Inverse Reinforcement Learning.
In this experiment set, we test the performanceof our algorithm on imitating the expert behavior.
We consider several high-dimensional roboticscontrol tasks in Mujoco [40].
Two class of existing algorithms are considered as the comparisonbaselines: 1) imitation learning algorithms that only learn the policy to imitate the expert, including Behavior Cloning (BC) [41] and Generative Adversarial Imitation Learning (GAIL) [10]; 2) IRLalgorithms which learn a reward function and a policy simultaneously, including Adversarial Inverse
Reinforcement Learning (AIRL) [11], f -IRL [13] and IQ-Learn [2].
To ensure fair comparison, allimitation learning / IRL algorithms use soft Actor-Critic [20] as the base RL algorithm.
For the expertdataset, we use the data provided in the ofﬁcial implementation 2 of f -IRL.
In this experiment, we implement two versions of our proposed algorithm: ML-IRL(State-Action)where the reward is parameterized as a function of state and action; ML-IRL(State-Only) whichutilizes the state-only reward function.
In Table 1, we present the simulation results under a limiteddata regime where the expert dataset only contains a single expert trajectory.
The scores (cumulativerewards) reported in the table is averaged over 3 random seeds.
In each random seed, we train algorithm from initialization and collect 20 trajectories to average their cumulative rewards afterthe algorithms converge.
According to the results reported in Table 1, it shows that our proposedalgorithms outperform the baselines on most tasks.
We observe that BC fails to imitate the expert’s behavior.
It is due to the fact that BC is based onsupervised learning and thus could not learn a good policy under such a limited data regime.
Moreover,we notice the training of IQ-Learn is unstable, which may be due to its inaccurate approximation tothe soft Q-function.
Therefore, in the Mujoco tasks where IQ-Learn does not perform well, so thatwe cannot match the results presented in the original paper [2], we directly report results from there(and mark them by ⇤ in Table 1).
The results of AIRL are not presented in Table 1 since it performspoorly even after spending signiﬁcant efforts in parameter tuning (similar observations have beenmade in in [42, 13]).
[Table]
Transfer Learning Across Changing Dynamics.
We further evaluate IRL algorithms on the transferlearning setting.
We follows the environment setup in [11], where two environments with differentdynamics are considered: Custom-Ant vs Disabled-Ant .
We compare ML-IRL(State-Only) withseveral existing IRL methods: 1) AIRL [11], 2) f
-IRL [13]; 3) IQ-Learn [2].
We consider two transfer learning settings: 1) data transfer; 2) reward transfer.
For both settings, theexpert dataset / trajectories are generated in Custom-Ant .
In the data transfer setting, we train IRLagents in Disabled-Ant by using the expert trajectories, which are generated in Custom-Ant .
Inthe reward transfer setting, we ﬁrst use IRL algorithms to infer the reward functions in Custom-Ant ,and then transfer these recovered reward functions to Disabled-Ant for further evaluation.
In bothsettings, we also train SAC with the ground-truth reward in Disabled-Ant and report the scores.
The numerical results are reoprted in Table 2.
the proposed ML-IRL(State-Only) achieves superiorperformance compared with the existing IRL benchmarks in both settings.
We notice that IQ-Learnfails in both settings since it indirectly recovers the reward function from a soft Q-functionapproximator, which could be inaccurate and is highly dependent upon the environment dynamics.
Therefore, the reward function recovered by IQ-Learn can not be disentangled from the expert actionsand environment dynamics, which leads to its failures in the transfer learning tasks.
[Table]
Conclusion
In this paper, we present a maximum likelihood IRL formulation and propose a provably efﬁcientalgorithm with a single-loop structure.
To our knowledge, we provide the ﬁrst non-asymptoticanalysis for IRL algorithm under nonlinear reward parameterization.
As a by-product, when weparameterize the reward as a state-only function, our algorithm could work in state-only IRL settingand enable reward transfer to new environments with different dynamics.
Our algorithm outperformsexisting IRL methods on high-dimensional robotics control tasks and corresponding transfer learningsettings.
A limitation of our method is the requirement for online training, so one future direction ofthis work is to further extend our algorithm and the theoretical analysis to the ofﬂine IRL setting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]