Receding Horizon Inverse Reinforcement Learning
Abstract
Inverse reinforcement learning (IRL) seeks to infer a cost function that explainsthe underlying goals and preferences of expert demonstrations.
This paper presentsreceding horizon inverse reinforcement learning (RHIRL), a new IRL algorithmfor high-dimensional, noisy, continuous systems with black-box dynamic models.
RHIRL addresses two key challenges of IRL: scalability and robustness.
Tohandle high-dimensional continuous systems, RHIRL matches the induced optimaltrajectories with expert demonstrations locally in a receding horizon manner and“stitches” together the local solutions to learn the cost; it thereby avoids the “curseof dimensionality”.
This contrasts sharply with earlier algorithms that match withexpert demonstrations globally over the entire high-dimensional state space.
To berobust against imperfect expert demonstrations and system control noise, RHIRLlearns a state-dependent cost function “disentangled” from system dynamics undermild conditions.
Experiments on benchmark tasks show that RHIRL outperformsseveral leading IRL algorithms in most instances.
We also prove that the cumulativeerror of RHIRL grows linearly with the task duration.
The code is available online 1 .
Introduction
Reinforcement learning (RL) has made exciting progress in a range of complex tasks, includingreal-time game playing [1], visuo-motor control of robots [2] and many other works.
The success,however, often hinges on a carefully crafted cost function [3, 4], which is a major impediment to thewide adoption of RL in practice.
Inverse reinforcement learning (IRL) [5] addresses this need bylearning a cost function that explains the underlying goals and preferences of expert demonstrations.
This work focuses on two key challenges in IRL, scalability and robustness .
Classic IRL algorithms commonly consist of two nested loops.
The inner loop approximates theoptimal control policy for a hypothesized cost function while the outer loop updates the cost functionby comparing the induced policy with expert demonstrations.
The inner loop solves the (forward)reinforcement learning or optimal control problem, which is in itself a challenge for complex highdimensional systems.
Many interesting ideas have been proposed for IRL, including, e.g., maximumentropy learning [6, 7], guided cost learning [8], and adversarial learning [9].
See Figure 1 forillustrations.
They try to match a globally optimal approximate policy with expert demonstrationsover the entire system state space or a sampled approximation of it.
This is impractical for highdimensional continuous systems and is a fundamental impediment to scalability.
Like RL, IRL alsosuffers from the “curse of dimensionality”.
To scale up, RHIRL computes locally optimal policieswith receding horizons rather than a globally optimal policy and then matches them with expertdemonstrations locally in succession (Figure 1 d ).
The local approximation and matching substantiallymitigate the impact of high-dimensional space and improve the sample efﬁciency of RHIRL, at thecost of a local rather than a global solution.
So RHIRL trades off optimality for scalability andprovides an alternative to the existing mainstream approaches.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Do not distribute.
[Figure]
Another important concern of IRL is noise in expert demonstrations and system control.
Humanexperts may be imperfect for various reasons and provide good, but still suboptimal demonstrations.
Further, the system may fail to execute the commanded actions accurately because of control noise.
We want to learn a cost function that captures the expert’s underlying intentions rather than theimperfectly executed actions.
We recognize that there are a number of IRL works on learningfrom sub-optimal/fail demonstrations [10, 11, 12].
There is a subtle, but key difference between (i)demonstrations intended to be suboptimal and (ii) optimal demonstrations corrupted by noise.
Theexisting works mentioned above address (i).
RHIRL addresses (ii).
To learn the true intentions fromthe noise corrupted demonstrations, RHIRL relies on a simplifying assumption: the cost functionis linearly separable with two components, one state-dependent and one control-dependent.
Manyinteresting systems in practice satisfy the assumption, at least, approximately [13].
RHIRL thenlearns the state-dependent cost component, which is disentangled from the system dynamics [9] andagnostic to noise.
Related Work
IRL can be viewed as an indirect approach to imitation learning.
It learns a cost function, whichinduces an optimal policy that matches with expert demonstrations.
In contrast, behavior cloning(BC) is a direct approach.
It assumes independence among all demonstrated state-action pairs andlearns a policy that maps states to actions through supervised learning of state-action pairs from expertdemonstrations.
The simplicity of BC is appealing.
However, it usually requires large amounts ofdata to learn well and suffers from covariate shift [14].
IRL is more data-efﬁcient.
Further, it producesa cost function, which explains the expert demonstrations and potentially transfers to other systemswith different dynamics.
These beneﬁts, however, come at the expense of greater computationalcomplexity.
Classic IRL algorithms learn a cost function iteratively in a double-loop: the outer loop updates ahypothesized cost function, and the inner loop solves the forward RL problem for an optimal policyand matches it with expert demonstrations.
Various methods have been proposed [6, 8, 7, 15], butthey all seek a globally optimal solution over the entire state space (Figure 1 a ) [6, 7] or the entire taskduration (Figure 1 b ) [8, 15].
As a result, they face signiﬁcant challenges in scalability and must makesimpliﬁcations, such as locally linear dynamics [8].
Recent methods use the generative adversarialnetwork (GAN) [16] to learn a discriminator that differentiates between the state or state-actiondistribution induced by the learned cost and that from expert demonstrations [9, 17, 18, 19, 20, 21, 22,23, 24].
We view this global matching as a major obstacle to scalability.
In addition, GAN training ischallenging and faces difﬁculties with convergence.
RHIRL stands in between BC and existing IRL algorithms as a novel midpoint by trading offoptimality for scalability.
BC performs local matching at each demonstrated state-action pair, treatingall of them independently.
Most of the existing IRL algorithms perform global matching over theentire state space or sampled trajectories from it.
Some works [8, 25, 26] use local optimizationto make the forward problem tractable, yet they [8, 25] either still still match over some globaldistributions, or it [26] only learns to the imitate the expert locally without any global performanceguarantee.
RHIRL follows the standard IRL setup.
To tackle the challenge of scalability for highdimensional continuous systems, RHIRL borrows ideas from receding horizon control [27].
Itsolves for locally optimal control sequences with receding horizons and learns the cost functionby “stitching” together a series of locally optimal solutions to match the global state distributionof expert demonstrations (Figure 1 d ).
Applying RHC to IRL has been explored in [28], yet this work relies on a manually selected feature vector and its recursive tree-search RHC limits itselfto discrete/low-dimensional tasks.
A concurrent work [29] shares similar idea to learn a costmapusing RHC, however it parameterizes the reward function as a linearly weighted combination ofstate features for simple IRL gradient derivation, hence it lacks the capacity to model other forms ofreward functions.
Another challenge to IRL is suboptimal expert demonstrations and system control noise.
Severalmethods learn an auxiliary score or ranking to reweigh the demonstrations, in order to approximatethe underlying optimal expert distribution [30, 31, 22].
RHIRL does not attempt to reconstruct theoptimal expert demonstrations.
It explicitly models the system noise and human error in controlactions as additive Gaussian noise and matches the noisy control with expert demonstrations, inorder to learn from the intended, rather than the executed expert actions.
Modelling the human erroras additive Gaussian noise is a natural choice technically, but compromises on realism.
Humanerrors in sequential decision making may admit other structural regularities, as a result of planning,resource constraints, uncertainty, and bounded rational pressures [32].
Studying the speciﬁc forms ofhuman errors in the context of IRL requires insights beyond the scope of our current work and it is apromising direction for future IRL works.
Receding Horizon Inverse Reinforcement Learning
3.1 Overview
Consider a continuous dynamical system with noisy control:
[Equation]
where x t 2 R n is the state, v t 2 R m is the control at time t , and the initial system state at t = 0 101 follows a distribution µ .
To account for noise in expert demonstrations and in system control, weassume that v t is a random variable following the Gaussian distribution
N ( v t | u t , ⌃ ) , with mean 103 u t and covariance ⌃ .
We can control u t directly, but not v t , because of noise.
The state-transition 104 function f captures the system dynamics.
RHIRL represents f as a black-box simulator and doesnot require its analytic form.
Thus, we can accommodate arbitrary complex nonlinear dynamics.
To simplify the presentation, we assume that the system dynamics is deterministic.
We sketch theextension to stochastic dynamics at the end of the section, the full proof is given in Appendix C.
In RL, we are given a cost function and want to ﬁnd a control policy that minimizes the expected totalcost over time under the dynamical system.
In IRL, we are not given the cost function, but instead, aset D of expert demonstrations.
Each demonstration is a trajectory of states visited by the expert over 111 time: ⌧ = ( x 0 , x 1 , x 2 , ..., x T  1 ) for a duration of T steps.
We assume that the expert chooses the actions to minimize an unknown cost function and want to rethis cost from the demonstrations.
Formally, suppose that the cost function is parameterized by ✓ .
RHIRL aims to learn a cost function that minimizes the loss L ( ✓ ; D ) , which measures the difference 115 between the demonstration trajectories and the optimal control policy induced by the cost functionwith parameters ✓ .
RHIRL performs this minimization iteratively, using the gradient @
L / @
✓ to update ✓ .
In iteration t , 118 let x t be the system state at time t and D t be the set of expert sub-trajectories starting at time t 119 and having a duration of maximum K steps.
We use the current cost to perform receding horizoncontrol (RHC) at x t , with time horizon K , and then update the cost by comparing the resulting statetrajectory distribution with the demonstrations in D t .
Earlier IRL work [9, 17, 15, 8, 33, 23] requires 122 the simulator to reset to the states from the initial state distribution.
To perform our receding horizonoptimization, RHIRL requires a simulator to reset to the initial state x t for the local control sequenceoptimization at each time step t = 0 , ...
T  1 .[-
-]
This is a common setting for the work such as[34, 35, 29] from the optimal control community.
See Algorithm 1 for a sketch.
First, sample M control sequences, each of length K (line 6).
We assume that the covariance ⌃is known.
If it is unknown, we set it to be identity by default.
For our experiment results reportedin Table 1 and Table 2, ⌃ is unknown and is approximated by a constant factor of the identitymatrix (grid search is performed to determine this constant factor).
We show through experimentsthat the learned cost function is robust over different noise settings (Section 4.3).
Next, we applymodel-predictive path integral (MPPI) control [34] at x t .
MPPI provides an analytic solution for the optimal control sequence distribution and the associated state sequence distribution, which allowus to estimate the gradient @
L / @✓ efﬁciently through importance sampling (lines 7–8) and update 134 cost function parameters ✓ (line 9).
Finally, we execute the computed optimal control (line 10–12)and update the mean control input for the next iteration (line 13).
We would like to emphasize thatwe only use the simulator to sample rollouts and evaluate the current cost function.
Fixed expertdemonstrations D are given upfront.
Unlike DAgger, we do not query the expert online during 138 training.
Our challenge is to uncover a cost function that captures the expert’s intended controls 140 ( u 0 , u 1 , . .
, u T  1 ) , even though they were not directly observed, because of noise, and do so in a 141 scalable and robust manner for high-dimensional, noisy systems.
We develop three ideas: structuring the cost function, matching locally with expert demonstrations,and efﬁcient computation of the gradient @
L / @✓ , which are described next.
3.2 Robust Cost Functions
To learn a cost function robust against noise, we make a simplifying assumption that linearly separatesthe one-step cost into two components: a state cost g ( x ; ✓ ) parameterized by ✓ and a quadratic controlcost u T ⌃  1 u .
Despite the simpliﬁcation, this cost function models a wide variety of interestingsystems in practice [13].
It allows RHIRL to learn a state cost g ( x ; ✓ ) , independent of control noise(Section 3.4), and thus generalize over different noise distributions (Section 4.3).
Suppose that V = ( v 0 , v 1 , . .
. , v K  1 ) is a control sequence of length K , conditioned on the input 151 U =
( u 0 , u 1 , . . .
, u K  1 ) .
We apply V to the dynamical system in (1) with start state x 0 and obtain a 152 state sequence ⌧ = ( x 0 , x 1 , . . .
, x K ) with x k ⇠ p ( x k | x k  1 , v k  1 ) for k = 1 , 2 , . . .
, K .
Deﬁne the 153 total cost of V as
[Equation]
where   0 is a constant weighting the relative importance between the state and control costs.
For 155 convenience, deﬁne also the total state cost of V as
[Equation]
[Equation]
While S is deﬁned in terms of the control sequence V , it only depends on the corresponding statetrajectory ⌧ .
This is very useful, as the training data contains state and not control sequences explicitly.
3.3 Local Control Sequence Matching
To minimize the loss L , each iteration of RHIRL applies RHC with time horizon K under the current 160 cost parameters ✓ and computes locally optimal control sequences of length K .
In contrast, classic
IRL algorithms, such as MaxEnt [6], perform global optimization over the entire task duration T inthe inner loop.
While RHC sacriﬁces global optimality, it is much more scalable and enables RHIRLto handle high-dimensional continuous systems.
We use the hyperparameter K to trade off optimalityand scalability.
Speciﬁcally, we use MPPI [34] to solve for an optimal control sequence distribution at the current startstate x t in iteration t .
The main result of MPPI suggests that the optimal control sequence distribution 167 Q minimizes the “free energy” of the dynamical system and this free energy can be calculated fromthe cost of the state trajectory under Q .
Mathematically, the probability density p Q ( V ⇤ ) can beexpressed as a function of the state cost S ( V, x t ; ✓ ) , with respect to a Gaussian “base” distribution 170 B ( U
B , ⌃ ) that depends on the control cost:
[Equation]
where Z is the partition function.
For the quadratic control cost in (2), we have U B = (0 , 0 , . . . )
Intuitively, the expression in (4) says that the probability density of V ⇤ is the product of two factors,one penalizing high control cost and one penalizing high state cost.
So controls with large values orresulting in high-cost states occur with probability exponentially small.
Equation (4) provides the optimal control sequence distribution under the current cost.
Supposethat the control sequences for expert demonstrations D t follow a distribution E .
We deﬁne the loss
L ( ✓ ; D t , x t ) as the KL-divergence between the two distributions:
[Equation]
which RHIRL seeks to minimize in each iteration.
While the loss L is deﬁned in terms of control 179 sequence distributions, the expert demonstrations
D provide state information and not control 180 information.
However, each control sequence V induces a corresponding state sequence ⌧ for agiven start state x 0 , and ⌧ determines the cost of V according to (2).
We show in the next subsectionthat @
L / @
✓ can be computed efﬁciently using only state information from D .
This makes RHIRL 183 appealing for learning tasks in which control labels are difﬁcult or impossible to acquire.
3.4 Gradient Optimization
To simplify notations, we remove the explicit dependency on x t and D t and assume that in iteration 186 t of RHIRL, all control sequences are applied with x t as the start state and expert demonstrationscome from D t .
We have
[Equation]
The ﬁrst line follows directly from the deﬁnition, and the derivation for the second line is available in
Appendix A.
We estimate the two integrals in (6) through sampling.
For the ﬁrst integral, we can use the expertdemonstrations as samples.
For the second integral, we cannot sample p Q directly, as the optimalcontrol distribution Q is unknown in advance.
Instead, we sample from a known Gaussian distributionwith density p ( V | U, ⌃ ) and apply importance sampling so that
[Equation]
The importance weight w ( V ) is given by
[Equation]
To simplify the ﬁrst ratio in (8), we substitute in the expression for p Q from (4):
[Equation]
We then simplify the second ratio, as both are Gaussian distributions with the same covariancematrix ⌃ :
[Equation]
where u k and v k , k = 0 , 1 , . .
, K  1 are the mean controls and the sampled controls from p ( V | U, ⌃ ) , 199 respectively.
Similarly, u B k , k = 0 , 1 , . . .
, K  1 are the mean controls for the base distribution, and 200 they are all 0 in our case.
Substituting (9) and (10) into (8), we have
[Equation]
For each sampled control sequence V , we evaluate the expression in (11) and normalize over allsamples to obtain w ( V ) .
To summarize, we estimate @
L / @
✓ through sampling:
[Equation]
where V i , i = 1 , . . .
, N are the control sequences for the expert demonstrations in D t and V j , j = 205 1 , 2 , . . .
, M are the sampled control sequences.
Equation (12) connects @
L / @✓ with @ S/ @✓ .
The 206 state cost function S is represented as a shallow neural network, and its derivative can be obtainedeasily through back-propagation.
To evaluate @@✓ S ( V i , x t ; ✓ ) , we do not actually use the expert control 208 sequences, as they are unknown.
We use the corresponding state trajectories in
D t directly, as the 209 state cost depends only on the visited states.
See equation (3).
Finally, we approximate the optimal mean control sequence according to (7):
[Equation]
The ﬁrst element in the control sequence U is the chosen control for the current time step t .
Weremove the ﬁrst element from U and append zero at the end.
The new control sequence is then usedas the mean for the sampling distribution in the next iteration.
3.5 Analysis
Since RHIRL performs local optimization sequentially over many steps, one main concern is erroraccumulation over time.
For example, standard behavior cloning has one-step error ✏ and cumulativeerror O ( T 2 ✏ ) over T steps, because of covariate shift [14].
The DAgger algorithm reduces the errorto O ( T ✏ ) by querying the expert repeatedly during online learning [14].
We prove a similar result for
RHIRL, which uses ofﬂine expert demonstrations only.
In iteration t of RHIRL, let p E ( V t | x t ) be the 220 K -step expert demonstration distribution and p Q ( V t | U B , ⌃ , x t ; ✓ ) be the computed K -step optimal 221 control distribution for some ﬁxed control parameters ✓ .
RHIRL minimizes the KL-divergencebetween these two distributions in each iteration.
Let p E ( x ) be the state marginal distribution of expertdemonstrations and p RHC ( x ; ✓ ) be the state marginal distribution of the computed RHC policy under ✓over the entire task duration T .
Intuitively, we want the control policy under the learned cost to visitstates similar to those of expert demonstrations.
In other words, p RHC ( x ; ✓ ) and p E ( x ) are close.
Theorem 3.1.
If D KL
[Equation]
< ✏ for all t = 0 , 1 , ..., T  1 , then
D
[Equation]
✏ / 2 .
The theorem says that RHIRL’s cumulative error, measured in total variation distance D TV between 229 p
E ( x ) and p RHC ( x ; ✓ ) , grows linearly with T .
The proof consists of three steps.
First, in each iteration 230 t , if the KL-divergence between two control sequence distributions are bounded by ✏ , so is the
KL-divergence between the two corresponding state distributions induced by control sequences.
Next,we show that the KL-divergence between the state distributions over two successive time steps arebounded by the same ✏ .
Finally, we switch from KL-divergence to total variation distance and applythe triangle inequality to obtain the ﬁnal result.
Note that our local optimization’s objective is deﬁned
Pendulum-v
LunarLander
Hopper-v
Walker2d-v
Ant-v
CarRacing-v |S| : 3-D 8-D 11-D 17-D 111-D 96 ⇥ 96 (image) |A| : 1-D 2-D 3-D 6-D 8-D 3-D T :      in KL divergence, while the ﬁnal error bound is in TV distance.
We switch the distance measuresto get the best of both: minimizing the KL divergence leads to strong local optimization result but
KL itself is not a proper metric.
Therefore, we further bound the KL divergence by TV distanceto obtain a proper metric bound for the ﬁnal result.
The full proof is given in Appendix B. Since
RHC performs local optimization in each iteration, we cannot guarantee global optimality.
However,the theorem indicates that unlike standard behavior cloning, the cumulative error of RHIRL growslinearly and not quadratically in T .
This shows one advantage of IRL over behavior cloning from thetheoretical angle.
Given a control policy V with the resulting state marginal distribution p V ( x ) , we can calculate theexpected total cost of V by integrating the one-step cost over p V .
Now suppose that the one-step costis bounded.
Theorem 3.1 then implies that the regret in total cost, compared with the expert policy,also grows linearly in T .
3.6 Extension to Stochastic Dynamics
Suppose that the system dynamics is stochastic: x t +1 = f
( x t , v t , !
t ) ,
where !
t is a random variable 249 that models the independent system noise.
RHIRL still applies, with modiﬁcations.
We redeﬁnethe total cost functions
˜ J ( V, x 0 ; ✓ ) and ˜ S ( V, x 0 ; ✓ ) by taking expectation over the system noisedistribution.
When calculating the importance weight ˜ w ( V ) in (11), we sample over the noisedistribution to estimate the expected total state cost.
Finally, we may need more samples whenestimating the gradient in (12), because of the increased variance due to the system noise.
The fullproof is given in Appendix C. We acknowledge that the experiments in this work are of deterministicdynamics, we leave the empirical results on RHIRL’s extension to stochastic dynamics to futurework.
Experiments
We investigate two main questions.
Does RHIRL scale up to high-dimensional continuous controltasks?
Does RHIRL learn a robust cost function under noise?
4.1 Setup
We compare RHIRL with two leading IRL algorithms, namely AIRL [9] and f -IRL [20], and oneimitation learning algorithm, GAIL [17].
In particular, f -IRL is a recent algorithm that achievesleading performance on high-dimensional control tasks.
We use the implementation of AIRL,
GAIL, and f -IRL from the f -IRL’s ofﬁcial repository along with the reported hyperparameters [20],whenever possible.
We also perform hyperparameter search on a grid to optimize the performance ofevery method on every task.
The speciﬁc hyperparameter settings used are reported in Appendix D.2.
Our benchmark set consists of six continuous control tasks (Figure 2) from OpenAI Gym [36], withincreasing sizes of state and action spaces.
For the most complex task, CarRacing, the input consistsof 96 ⇥ 96 raw images, resulting in an enormous state space that poses great challenges [37].
To our 270 knowledge, RHIRL is the ﬁrst few IRL algorithms to attempt such a high-dimensional space.
For faircomparison, we customize all tasks to the ﬁxed task horizon settings (Figure 2.
See Appendix D.2 fordetails on task parameter settings.
We use WorldModel [37] to generate expert demonstration data for CarRacing and use SAC [38]for the other tasks.
We add Gaussian noise to the input controls and collect expert demonstrationsat different control noise levels.
The covariance of the control noise is unknown to all methods,including, in particular, RHIRL.
To measure the performance of the learned cost function and policy, we score its induced optimalpolicy using the ground-truth cost function.
For ease of comparison with the literature, we use negatedcost values, i.e., rewards, in all reported results.
Higher values indicate better performance.
Eachexperiment is repeated 10 times to estimate the performance variance.
4.2 Scalability
The performance is reported as the ratio of the learned policy’s average return and the expert’s average return.
The absolute average returns and the standard deviations are reported in Appendix D.4.
Negative ratios are clipped to 0 .
The two numbers under the name of each environment indicate the dimensions of the state space and the action space, respectively.
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We compare RHIRL with othermethods, ﬁrst in noise-free environments (Figure 3) and thenwith increasing noise levels (Table 1).
Figure 3 shows the learningcurve of each methods in noisefree environments.
Overall, 290 RHIRL converges faster andachieves higher return, especiallyfor tasks with higher state spacedimensions.
This improved performance suggests that the beneﬁt of local optimization adoptedby RHIRL outweighs its potential limitations.
Table 1 shows the ﬁnal performance of all methods at variousnoise levels.
RHIRL clearly outperforms AIRL and GAIL in allexperiments.
So we focus ourdiscussion on comparison with fIRL.
In noise-free environments,
RHIRL and f -IRL perform comparably on most tasks.
On CarRacing, the most challengingtask, RHIRL performs much better.
RHIRL manages to learnthe critical driving behaviors illustrated in Figure 4, despite thehigh-dimensional image input.
However, RHIRL does not manage to learn to drive fast enough.
That is the main reason whyit under-performs the expert.
In comparison, f -IRL only learns to follow a straight lane after a largenumber of environment steps, and still fails to make a sharp turn after 3 . 0 ⇥ 10 7 environment steps.
In the noisy environments, the advantage of RHIRL over f -IRL is more pronounced even on someof the low-dimensional tasks, because RHIRL accounts for the control noise explicitly in the costfunction.
[Figure]
Next we evaluate RHIRL and other methods for robustness under noise.
A robust cost functionencodes the expert’s true intent.
It is expected to yield consistent performance over different noiselevels, regardless of noise in expert demonstrations.
For each task, a cost function is learned in a noise-free environment and is then used to re-optimizea policy in the corresponding noisy environments.
Speciﬁcally for GAIL, since it learns a policyand does not recover the associated cost function, we directly apply the learned policy in noisyenvironments.
4.3 Robustness
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Table 2 shows that noise causes performance degradation in all methods.
However, RHIRL is more robust in comparison.
For simple tasks, Pendulum and Lunarlander, RHIRL and f -IRL perform consistently well across different noise levels,while GAIL and AIRL fail to maintaintheir good performance, when the noiselevel increases.
For the more challengingtasks, Hopper and Walker, RHIRL’s performance degrades mildly, and f -IRL suffers more signiﬁcant performance degradation.
It is worth noting that the expertdemonstrations used to train the transferredcost function are from the perfect system.
Therefore, some expert actions and statesmay no longer be optimal or feasible in ahighly noisy environment.
Moreover, thecost function trained in the perfect systemcannot reason about the long-term consequences of an action in a high noise environment.
Therefore, it is challenging forthe learned cost function to be robust to ahighly noisy environment, as capturing thetrue intention of the expert is difﬁcult.
4.4 Effect of Receding Horizon K
The receding horizon K allows RHIRLto trade off optimality and efﬁciency.
Tostudy its effect, we use Pendulum with 356 K 2 { 5 , 20 , 30 , 75 } .
When K is small, 357 RHIRL learns fast with fewer environmentsteps, but may be unstable.
For K = 5 , theperformance jumps after the ﬁrst few steps, but ﬂuctuates drastically, as K is much smaller thanthe number of look-ahead steps for the pendulum to swing up.
The learned cost function tends tooverﬁt to local behaviors and forget the useful reward signals learned previously.
With larger Kvalues ( 20 or 30 ), the performance of the cost function learned is more stable, without much lossin efﬁciency.
However, when K is too large, e.g., K = 75 , RHIRL becomes less efﬁcient and takes longer to converge, because that excessive K values enlarge the variance of sampled controlsequences.
Conclusion
RHIRL is a scalable and robust IRL algorithm for highdimensional, noisy, continuous systems.
Our experimentsshow that RHIRL outperforms several leading IRL algorithms on multiple benchmark tasks, especially whenexpert demonstrations are noisy.
RHIRL’s choice of local rather than global optimizationis an important issue that deserves further investigation.
Overall, we view this as an interesting trade-off betweenscalability and optimality.
While this trade-off is wellknown in reinforcement learning, optimal control, andgeneral optimization problems, it is mostly unexplored in
IRL.
Further, local optimization may tie the learned costwith the optimizer.
It would be interesting to examinewhether the learned cost transfers easily to other domains with different optimizers.
We are keen toinvestigate these important issues and their implications to IRL as our next step.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
a) Do in
the
contributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] See Section 5(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a)
[Table]
the
[Table]
to
[Table]
and the total
[Table]
spent on participant compensation?
[N/A [Table]