A Ranking Game for Imitation Learning
Abstract
We propose a new framework for imitation learning—treating imitation as a twoplayer ranking-based game between a policy and a reward.
In this game, the reward agent learns to satisfy pairwise performance rankings between behaviors, while the policy agent learns to maximize this reward.
In imitation learning, near-optimal expert data can be difficult to obtain, and even in the limit of infinite data cannot imply a total ordering over trajectories as preferences can.
On the other hand, learning from preferences alone is challenging as a large number of preferences are required to infer a high-dimensional reward function, though preference data is typically much easier to collect than expert demonstrations.
The classical inverse reinforcement learning (IRL) formulation learns from expert demonstrations but provides no mechanism to incorporate learning from offline preferences and vice versa.
We instantiate the proposed ranking-game framework with a novel ranking loss giving an algorithm that can simultaneously learn from expert demonstrations and preferences, gaining the advantages of both modalities.
Our experiments show that the proposed method achieves state-of-the-art sample efficiency and can solve previously unsolvable tasks in the Learning from Observation (LfO) setting.
Project code and details can be found at https://hari-sikchi.github.io/rank-game
Introduction
Reinforcement learning relies on environmental reward feedback to learn meaningful behaviors.
Reward specification is a hard problem [1], thus motivating imitation learning (IL) as a technique to bypass reward specification and learn from expert data, often via Inverse Reinforcement Learning (IRL) techniques.
Learning from expert information (imitation learning) alone can require efficient exploration when the expert actions are unavailable as in LfO [2].
Incorporating preferences over potentially suboptimal trajectories for reward learning can help reduce the exploration burden by regularizing the reward function and providing effective guidance for policy optimization.
Previous literature in learning from preferences either assumes no environment interaction [3, 4] or assumes an active query framework with a restricted reward class [5].
The classical IRL formulation suffers from two issues: (1) Learning from expert demonstrations and learning from preferences/rankings provide complementary advantages for increasing learning efficiency [5, 6]; however, existing IRL methods that learn from expert demonstrations provide no mechanisms to incorporate offline preferences and vice versa.
(2) Optimization is difficult, making learning sample inefficient [7, 8] due to the adversarial min-max game.
Our primary contribution is an algorithmic framework casting imitation learning as a ranking game that addresses both of the above issues in IRL.
This framework treats imitation as a ranking game between two agents: a reward agent and a policy agent—the reward agent learns to satisfy pairwise performance rankings between different behaviors represented as state-action or state visitations, while the policy agent maximizes its performance under the learned reward function.
The ranking game is detailed in Figure[-
-]
IL Method
Offline
Expert
Ranking
Reward
Active Human Preferences
Data
Loss
Function
Query
MaxEntIRL, AdRIL,GAN-GCL, ✗
LfD supremum non-linear ✗ GAIL, f -MAX, AIRL
BCO,GAIfO, DACfO, ✗
LfO supremum non-linear ✗ OPOLO, f -IRL
TREX, DREX ✓ ✗
Bradley-Terry non-linear ✗
BREX ✓ ✗
Bradley-Terry linear ✗
DemPref ✓
LfO/LfD
Bradley-Terry linear ✓
Ibarz et al[6] ✓
LfD
Bradley-Terry non-linear ✓ rank-game ✓
LfO/LfD
L k non-linear ✗ [Figure] the reward function by interacting with the environment. The Reward agent satisfies a set of behavior rankings obtained from various sources: generated by the policy agent (vanilla), automatically generated (auto), or offline annotated rankings obtained from a human or offline dataset (
, or and is specified by three components: (1) The dataset of pairwise behavior rankings, (2) A ranking loss function, and (3)
An optimization strategy.
This game encompasses a large subset of both inverse reinforcement learning (IRL) methods and methods that learn from suboptimal offline preferences.
Popular IRL methods such as GAIL, AIRL, f -MAX [8, 9, 10] are instantiations of this ranking game in which rankings are given only between the learning agent and the expert, and a gradient descent ascent (GDA) optimization strategy is used with a ranking loss that maximizes the performance gap between the behavior rankings.
The ranking loss used by the prior IRL approaches is specific to the comparison of optimal (expert) vs. suboptimal (agent) data and precludes the incorporation of comparisons among suboptimal behaviors.
In this work, we instantiate the ranking game by proposing a new ranking loss ( L k ) that facilitates incorporation of rankings over suboptimal trajectories for reward learning.
Our theoretical analysis reveals that the proposed ranking loss results in a bounded performance gap with the expert that depends on a controllable hyperparameter.
Our ranking loss can also ease policy optimization by supporting data augmentation to make the reward landscape smooth and allowing control over the learned reward scale.
Finally, viewing our ranking game in the Stackelberg game framework (see Section 3)—an efficient setup for solving general-sum games—we obtain two algorithms with complementary benefits in non-stationary environments depending on which agent is set to be the leader.
In summary, this paper formulates a new framework rank-game for imitation learning that allows us to view learning from preferences and demonstrations under a unified perspective.
We instantiate the framework with a principled ranking loss that can naturally incorporate rankings provided by diverse sources.
Finally, by incorporating additional rankings—auto-generated or offline—our method: (a) outperforms state-of-the-art methods for imitation learning in several MuJoCo simulated domains by a significant margin and (b) solves complex tasks like imitating to reorient a pen with dextrous manipulation using only a few observation trajectories that none of the previous LfO baselines can solve.
Related Work
Imitation learning methods are broadly divided into two categories:
Behavioral cloning [11, 12] and
Inverse Reinforcement Learning (IRL) [8, 9, 13, 14, 15, 16, 17].
Our work focuses on developing a new framework in the setting of IRL through the lens of ranking.
Table 1 shows a comparison of the proposed rank-game method to prior works. Classical Imitation Game for IRL : The classical imitation game for IRL aims to solve the adversarial min-max problem of finding a policy that minimizes the worst-case performance gap between the agent and the expert. A number of previous works [9, 10, 18] have focused on analyzing the properties of this min-max game and its relation to divergence minimization.
Under some additional regularization, this min-max objective can be understood as minimizing a certain f -divergence [8, 9, 10] between the agent and expert state-action visitation. More recently, [18] showed that all forms of imitation learning (BC and IRL) can be understood as performing moment matching under differing assumptions. In this work, we present a new perspective on imitation in which the reward function is learned using a dataset of behavior comparisons, generalizing previous IRL methods that learn from expert demonstrations and additionally giving the flexibility to incorporate rankings over suboptimal behaviors. Learning from Preferences and Suboptimal Data : Learning from preferences and suboptimal data is important when expert data is limited or hard to obtain.
Preferences [5, 19, 20, 21, 22] have the advantage of providing guidance in situations expert might not get into, and in the limit provides full ordering over trajectories which expert data cannot.
A previous line of work [3, 4, 23, 24] has studied this setting and demonstrated that offline rankings over suboptimal behaviors can be effectively leveraged to learn a reward function.
[5, 6, 22] studied the question of learning from preferences in the setting when a human is available to provide online preferences 1 (active queries), while [5] additionally assumed the reward to be linear in known features.
Our work makes no such assumptions and allows for integrating offline preferences and expert demonstrations under a common framework.
Learning from Observation (LfO): LfO is the problem setting of learning from expert observations.
This is typically more challenging than the traditional learning from demonstration setting (LfD), because actions taken by the expert are unavailable.
LfO is broadly formulated using two objectives: state-next state marginal matching [ 25, 26, 27] and direct state marginal matching [28, 29].
Some prior works [30, 31, 32] approach LfO by inferring expert actions through a learned inverse dynamics model.
These methods assume injective dynamics and suffer from compounding errors when the policy is deployed.
A recently proposed method OPOLO [26] derives an upper bound for the LfO objective which enables it to utilize off-policy data and increase sample efficiency.
Our method outperforms baselines including OPOLO, by a significant margin.
Background
We consider a learning agent in a Markov Decision Process (MDP) [33, 34] which can be defined as a tuple: M = ( S , A , P, R, γ, ρ 0 ) , where S and A are the state and action spaces; P is the state transition probability function, with P ( s ′ | s, a ) indicating the probability of transitioning from s to s ′ when taking action a ; R : S ×
A → R is the reward function bounded in [0 , R max ] ; We consider MDPs with infinite horizon, with the discount factor γ ∈ [0 , 1] , though our results extend to finite horizons as well; p 0 is the initial state distribution.
We use Π and R to denote the space of policies and reward functions respectively.
A reinforcement learning agent aims to find a policy π : S → A that maximizes its expected return, J ( R ; π ) =− γ E
( s,a ) ∼ ρ π ( s,a ) [ R ( s, a )] , where ρ π ( s, a ) is the stationary state-action distribution induced by π .
In imitation learning, we are provided with samples from the state-action visitation of the expert ρ π E ( s, a ) but the reward function of the expert is unknown.
We will use ρ E ( s, a ) as a shorthand for ρ π E ( s, a ) .
Classical Imitation Learning :
The goal of imitation learning is to close the imitation gap J ( R ; π E ) −
J ( R ; π ) defined with respect to the unknown expert reward function R .
Several prior works [8, 18, 28, 35] tackle this problem by minimizing the imitation gap on all possible reward hypotheses.
This leads to a zero-sum (min-max) game formulation of imitation learning in which a policy is optimized with respect to the reward function that induces the largest imitation gap:
[Equation]
Here, the imitation gap is upper bounded as follows ( ∀ π ):
[Equation]
Note that, when the performance gap is maximized between the expert π E and the agent π , we can observe that the worst-case reward function f π induces a ranking between policy behaviors based on their performance: ρ E ⪰ ρ π :=
E ρ E
( s,a )
[ f π ( s, a )] ≥ E ρ π ( s,a )
[ f π ( s, a )] , ∀ π .
Therefore, we can regard the above loss function that maximizes the performance gap (Eq. 2) as an instantiation of the ranking-loss.
We will refer to the implicit ranking between the agent and the expert ρ E ⪰ ρ π as vanilla rankings and this variant of the ranking-loss function as the supremum-loss .
Stackelberg Games : A Stackelberg game is a general-sum game between two agents where one agent is set to be the leader and the other a follower.
The leader in this game optimizes its objective under the assumption that the follower will choose the best response for its own optimization objective.
More concretely, assume there are two players A and B with parameters θ A , θ B and corresponding losses
L A ( θ A , θ B ) and L B ( θ A , θ B ) .
A Stackelberg game solves the following bi-level optimization when A is the leader and B is the follower: min θ
[Equation]
[36] showed that casting model-based RL as an approximate Stackelberg game [37] leads to performance benefits and reduces training instability in comparison to the commonly used GDA [38] and Best Reponse (BR) [39] methods.
[40, 41] prove convergence of Stackelberg games under smooth player cost functions and show that they reduce the cycling behavior to find an equilibrium and allow for better convergence.
A Ranking Game for Imitation Learning
In this section, we first formalize the notion of the proposed two-player general-sum ranking game for imitation learning.
We then propose a practical instantiation of the ranking game through a novel ranking-loss ( L k ).
The proposed ranking game gives us the flexibility to incorporate additional rankings—both auto-generated (a form of data augmentation mentioned as ‘auto’ in Fig. 1) and offline (‘pref’ in Fig.
1)—which improves learning efficiency.
Finally, we discuss the Stackelberg formulation for the two-player ranking game and discuss two algorithms that naturally arise depending on which player is designated as the leader.
4.1 The Two-Player Ranking Game Formulation
We present a new framework, rank-game , for imitation learning which casts it as a general-sum ranking game between two players — a reward and a policy.
[Equation]
In this formulation, the policy agent maximizes the reward by interacting with the environment, and the reward agent attempts to find a reward function that satisfies a set of pairwise behavior rankings in the given dataset D p ; a reward function satisfies these rankings if E ρ πi
[ R ( s, a )] ≤ E ρ πj
[ R ( s, a )] , ∀
ρ π i ⪯
ρ π j ∈ D p , where ρ π i , ρ π j can be state-action or state vistitations.
The dataset of pairwise behavior rankings D p can be comprised of the implicit ‘vanilla’ rankings between the learning agent and the expert’s policy behaviors ( ρ π ⪯ ρ E ), giving us the classical
IRL methods when a specific ranking loss function – supremum-loss is used [8, 9, 10].
If rankings are provided between trajectories, they can be reduced to the equivalent ranking between the corresponding state-action/state visitations.
In the case when D p comprises purely of offline trajectory performance rankings then, under a specific ranking loss function ( Luce-shepard ), the ranking game reduces to prior reward inference methods like T-REX [3, 4, 23, 24]. Thus, the ranking game affords us a broader perspective of imitation learning, going beyond only using expert demonstrations. 4.2 Ranking Loss L k for the Reward Agent
We use a ranking-loss to train the reward function—an objective that minimizes the distortion [42] between the ground truth ranking for a pair of entities { x, y } and rankings induced by a parameterized function R : X → R for a pair of scalars { R ( x ) , R ( y ) } .
One type of such a ranking-loss is the supremum-loss in the classical imitation learning setup.
1: Initialize policy π 0 θ , reward funtion R ϕ , empty dataset D π . empirical expert data ˆ ρ E 2: for t = 1 ..T iterations do 3:
Collect empirical visitation data ˆ ρ π tθ with π tθ in the environment.
Set D π
= { (ˆ ρ π ⪯ ˆ ρ E ) } 4:
Train reward R ϕ to satisfy rankings in D π using ranking loss L k in equation 3.
Optimize policy under the reward function: π t +1 θ ← argmax π ′ J ( R ϕ ; π ′ ) 6: end for
We propose a class of ranking-loss functions L k that attempt to induce a performance gap of k for all behavior preferences in the dataset.
Formally, this can be implemented with the regression loss:
[Equation]
where D p contains behavior pairs ( ρ π i , ρ π j ) s.t ρ π i ⪯ ρ π j .
The proposed ranking loss allows for learning bounded rewards with user-defined scale k in the agent and the expert visitations as opposed to prior works in Adversarial Imitation Learning [8, 9, 17].
Reward scaling has been known to improve learning efficiency in deep RL; a large reward scale can make the optimization landscape less smooth [43, 44] and a small scale might make the action-gap small and increase susceptibility to extrapolation errors [45].
In contrast to the supremum loss, L k can also naturally incorporate rankings provided by additional sources by learning a reward function satisfying all specified pairwise preferences.
The following theorem characterizes the equilibrium of the rank-game for imitation learning when L k is used as the ranking-loss.
Theorem 4.1.
(Performance of the rank-game equilibrium pair) Consider an equilibrium of the imitation rank-game ( ˆ π, ˆ R ), such that the ranking loss L k generalization error is bounded by 2 R 2 max ϵ r and the policy is near-optimal with J ( ˆ R ;
ˆ π ) ≥ J ( ˆ R ; π ) − ϵ π ∀ π , then at this equilibrium pair under the expert’s unknown reward function R gt bounded in [0 , R Emax ] :
[Equation]
If reward is a state-only function and only expert observations are available, the same bound applies to the LfO setting.
Proof.
We defer the proof to Appendix A.
Theoretical properties: We now discuss some theoretical properties of L k .
Theorem 1 shows that rank-game has an equilibrium with bounded performance gap with the expert.
An optimization step by the policy player, under a reward function optimized by the reward player, is equivalent to minimizing an f -divergence with the expert.
Equivalently, at iteration t in Algorithm 1:
[Equation]
We elaborate on the regret of this idealized algorithm in Appendix A. Theorem 1 suggests that large values of k can guarantee the agent’s performance is close to the expert.
In practice, we observe intermediate values of k also preserve imitation equilibrium optimality with a benefit of promoting sample efficient learning (as an effect of reward scaling described earlier).
We discuss this observation further in Appendix D.9.
rank-game naturally extends to the LfO regime under a state-only reward function where Theorem 4.1 results in a divergence bound between state-visitations of the expert and the agent.
A state-only reward function is also a sufficient and necessary condition to ensure that we learn a dynamics-disentangled reward function [17].
L k can incorporate additional preferences that can help learn a regularized/shaped reward function that provides better guidance for policy optimization, reducing the exploration burden and increasing sample efficiency for IRL.
A better-guided policy optimization is also expected to incur a lower ϵ π .
However, augmenting the ranking dataset can lead to a decrease in the intended performance gap ( k eff < k ) between the agent and the expert (Appendix A).
This can loosen the bound in Eq 4 and lead to non-optimal imitation learning.
We hypothesize that given informative preferences, decreased ϵ π can compensate for potentially decreased intended performance gap k eff to ensure near optimal imitation.
In our experiments, we observe this hypothesis holds true; we enjoy sample efficiency benefits without losing any asymptotic performance.
To leverage these benefits, we present two methods for augmenting the ranking dataset below and defer the implementation details to Appendix B.
4.2.1 Augmenting the Ranking Dataset
Reward loss w/ automatically generated rankings (auto) :
In this method, we assume access to the behavior-generating trajectories in the ranking dataset.
For each pairwise comparison ρ i ⪯ ρ j present in the dataset, L k sets the regression targets for states in ρ i to be 0 and for states visited by ρ j to be k .
Equivalently, we can rewrite minimizing L k as regressing an input of trajectory τ i to vector 0 , and τ j to vector k 1 where τ i , τ j are trajectories that generate the behavior ρ i , ρ j respectively.
We use the comparison ρ i ⪯ ρ j to generate additional behavior rankings ρ i ⪯ ρ λ 1 ,ij ⪯ ρ λ 2 ,ij .. ⪯ ρ λ P ,ij ⪯ ρ j where 0 < λ 1 < λ 2 < ... <
λ P < 1 .
The behavior ρ λ p ,ij is obtained by independently sampling the trajectories that generate the behaviors
ρ i , ρ j and taking convex combinations i.e τ λ p ,ij = λ
p τ i + (1 − λ p )
τ j and their corresponding reward regressions targets are given by λ p
+ (1 − λ p )
k 1 .
This form of data augmentation can be interpreted as mixup [46] regularization in the trajectory space.
Mixup has been shown to improve generalization and adversarial robustness [46, 47] by regularizing the first and second-order gradients of the parameterized function.
Following the general principle of using a smoothed objective with respect to inputs to obtain effective gradient signals, explicit smoothing in the trajectory space can also help reduce the policy optimization error ϵ π .
A didactic example showing rewards learned using this method is shown in Figure 2.
In a special case when the expert’s unknown reward function is linear in observations, these rankings reflect the true underlying rankings of behaviors.
Reward loss w/ offline annotated rankings (pref) :
Another way of increasing learning efficiency is augmenting the ranking dataset containing the vanilla ranking ( ρ π ⪯ ρ E ) with offline annotated rankings.
These rankings may be provided by a human observer or obtained using an offline dataset of behaviors with annotated reward information, similar to the datasets used in offline RL [48, 49].
We combine offline rankings by using a weighted loss between L k for satisfying vanilla rankings ( ρ π ⪯ ρ E ) and offline rankings, grounded by an expert.
Providing offline rankings alone that are sufficient to explain the reward function of the expert [3] is often a difficult task and the number of offline preferences required depends on the complexity of the environment.
In the LfO setting, learning from an expert’s state visitation alone can be a hard problem due to exploration requirements [2].
This ranking-loss combines the benefits of using preferences to shape the reward function and guide policy improvement while using the expert to guarantee near-optimal performance.
4.3 Optimizing the Two-Player General-Sum Ranking Game as a Stackelberg Game
Solving the ranking-game in the Stackelberg setup allows us to propose two different algorithms depending on which agent is set to be the leader and utilize the learning stability and efficiency afforded by the formulation as studied in [36, 40, 41].
Policy as leader (PAL) :
Choosing policy as the leader implies the following optimization:
[Equation]
Reward as leader (RAL): Choosing reward as the leader implies the following optimization:
[Equation]
We follow the first order gradient approximation for leader’s update from previous work [36] to develop practical algorithms.
This strategy has been proven to be effective and avoids the computational complexity of calculating the implicit Jacobian term ( dθ ∗ B /dθ A ).
PAL updates the reward to near convergence on dataset D π
( D π contains rankings generated using the current policy agent only π ⪯ π E ) and takes a few policy steps.
Note that even after the first-order approximation, this optimization strategy differs from GDA as often only a few iterations are used for training the reward even in hyperparameter studies like [50].
RAL updates the reward conservatively.
This is achieved through aggregating the dataset of implicit rankings from all previous policies obtained during training.
PAL’s strategy of using on-policy data D π for reward training resembles that of methods including GAIL [8, 51], f -MAX [9], and f -IRL [28].
RAL uses the entire history of agent visitation to update the reward function and resembles methods such as apprenticeship learning and DAC [14, 52].
PAL and RAL bring together two seemingly different algorithm classes under a unified Stackelberg game viewpoint.
Experimental Results
We compare rank-game against state-of-the-art LfO and LfD approaches on MuJoCo benchmarks having continuous state and action spaces.
The LfO setting is more challenging since no actions are available, and is a crucial imitation learning problem that can be used in cases where action modalities differ between the expert and the agent, such as in robot learning.
We focus on the LfO setting in this section and defer the LfD experiments to Appendix D.2.
We denote the imitation learning algorithms that use the proposed ranking-loss L k from Section 4.2 as RANK-{PAL, RAL}.
We refer to the rank-game variants which use automatically generated rankings and offline preferences as (auto) and (pref) respectively following Section 4.2.
In all our methods, we rely on an off-policy model-free algorithm, Soft Actor-Critic (SAC) [53], for updating the policy agent.
We design experiments to answer the following questions:
Asymptotic Performance and Sample Efficiency : Is our method able to achieve near-expert performance given a limited number (1) of expert observations?
Can our method learn using fewer environment interactions than prior state-of-the-art imitation learning (LfO) methods? 2.
Utility of preferences for imitation learning : Current LfO methods struggle to solve a number of complex manipulation tasks with sparse success signals.
Can we leverage offline annotated preferences through rank-game in such environments to achieve near-expert performance?
Choosing between PAL and RAL methods : Can we characterize the benefits and pitfalls of each method, and determine when one method is preferable over the other?
Ablations for the method components : Can we establish the importance of hyperparameters and design decisions in our experiments?
Baselines: We compare RANK-PAL and RANK-RAL against representative LfO approaches that covers a spectrum of on-policy and off-policy model-free methods from prior work: GAIfO [8, 51],
DACfO [52], BCO [30], f -IRL [28] and recently proposed OPOLO [26] and IQLearn [54].
We do not assume access to expert actions in this setting.
Our LfD experiments compare to the IQLearn [54],
DAC [52] and BC baselines.
Detailed description for baselines can be found in Appendix D.2.
5.1 Asymptotic Performance and Sample Efficiency
In this section [Table] we compare RANK-PAL(auto) and RANK-RAL(auto) to baselines on a set of MuJoCo locomotion tasks of varying complexities: Swimmer-v2, Hopper-v2, HalfCheetah-v2,
Walker2d-v2, Ant-v2 and Humanoid-v2 .
In this experiment [Table] we provide one expert trajectory for all methods and do not assume access to any offline annotated rankings [Table]
[Equation]
[Table]
Asymptotic Performance :
Table 2 shows that both rank-game methods are able to reach nearexpert asymptotic performance with a single expert trajectory.
BCO shows poor performance which can be attributed to the compounding error problem arising from its behavior cloning strategy.
GAIfO and DACfO use GDA for optimization with a supremum loss and show high variance in their asymptotic performance whereas rank-game methods are more stable and low-variance.
[Figure]
A v e r age
R e w a r d
Hopper-v     1e Walker-v      51e
Humanoid-v
[Figure]
Sample Efficiency : Figure 3 shows that RANK-RAL and RANK-PAL are among the most sample efficient methods for the LfO setting
[Figure] outperforming the recent state-of-the-art method OPOLO [26] by a significant margin.
We notice that IQLearn fails to learn in the LfO setting.
This experiment demonstrates the benefit of the combined improvements of the proposed ranking-loss with automatically generated rankings.
Our method is also simpler to implement than OPOLO [Figure] as we require fewer lines of code changes on top of SAC and need to maintain fewer parameterized networks compared to OPOLO which requires an additional inverse action model to regularize learning
[Figure]
5.2 Utility of Preferences in Imitation
[Figure] [Figure] Our experiments on complex manipulation environments—door opening with a parallel-jaw gripper [55] and pen manipulation with a dexterous adroit hand [56] – reveal that none of the prior LfO methods are able to imitate the expert even under increasing amounts of expert data.
This failure of LfO methods can be potentially attributed to the exploration requirements of LfO compared to LfD [2], coupled with the sparse successes encountered in these tasks, leading to poorly guided policy gradients.
In these experiments, we show that rank-game can incorporate additional information in the form of offline annotated rankings to guide the agent in solving such tasks.
These offline rankings are obtained by uniformly sampling a small set of trajectories (10) from the replay buffer of SAC [53] labeled with a ground truth reward function.
We use a weighted ranking loss (pref) from Section 4.2.
Figure 4 shows that RANK-PAL/RAL(pref) method leveraging offline ranking is the only method that can solve these tasks, whereas prior LfO methods and RANK-PAL/RAL(auto) with automatically generated rankings struggle even after a large amount of training.
We also point out that T-REX, a method that learns using the preferences alone is unable to achieve near-expert performance, thereby highlighting the benefits of learning from expert demonstrations alongside a set of offline preferences.
5.3 Comparing PAL and RAL
PAL uses the agent’s current visitation for reward learning, whereas RAL learns a reward consistent with all rankings arising from the history of the agent’s visitation.
These properties can present certain benefits depending on the task setting.
To test the potential benefits of PAL and RAL, we consider two non-stationary imitation learning problems, similar to [56] – one in which the expert changes it’s intent and the other where dynamics of the environment change during training in the Hopper-v2 locomotion task.
For changing intent, we present a new set of demonstrations where the hopper agent hops backward rather than forward.
For changing envi- ronment dynamics, we increase the mass of the hopper agent by a factor of 1.2.
Changes are introduced at 1e5 time steps during training at which point we notice a sudden performance drop. {+2.0 Timesteps 1e 
 A v e r age R+}
e w a r d
Intent-Adaptation 0.
2.0 Timesteps 1e5   4000 Dynamics-Adaptation RANK-PAL RANK-RAL In Figure 5 (left), we notice that PAL adapts faster to intent changes, whereas RAL needs to unlearn the rankings obtained from the agent’s history and takes longer to adapt.
Figure(right) shows that RAL adapts faster to the changing dynamics of the system, as it has already learned a good global notion of the dynamics-disentangled reward function in the LfO setting, whereas PAL only has a local understanding of reward as a result of using ranking obtained only from the agent’s current visitation.
Ablation of Method Components: Appendix D contains eight additional experiments to study the importance of hyperparameters and design decisions.
Our ablations validate the importance of using automatically generated rankings, the benefit of ranking loss over supremum loss, and sensitivity to hyperparameters like the intended performance gap k , policy iterations, and the reward regularizer.
Conclusion
In this work, we present a new framework for imitation learning that treats imitation as a two-player ranking-game between a policy and a reward function.
Unlike prior works in imitation learning, the ranking game allows incorporation of rankings over suboptimal behaviors to aid policy learning.
We instantiate the ranking game by proposing a novel ranking loss which guarantees agent’s performance to be close to expert for imitation learning.
Our experiments on simulated MuJoCo tasks reveal that utilizing additional ranking through our proposed ranking loss leads to improved sample efficiency for imitation learning, outperforming prior methods by a significant margin and solving some tasks which were unsolvable by previous LfO methods.
Limitations and Negative Societal Impacts: Preferences obtained in the real world are usually noisy [57, 58, 59] and one limitation of rank-game is that it does not suggest a way to handle noisy preferences.
Second, rank-game proposes modifications to learn a reward function amenable to policy optimization but these hyperparameters are set manually.
Future work can explore methods to automate learning such reward functions.
Third, despite learning effective policies we observed that we do not learn reusable robust reward functions [28].
Negative Societal Impact: Imitation learning can cause harm if given demonstrations of harmful behaviors, either accidentally or purposefully.
Furthermore, even when given high-quality demonstrations of desirable behaviors, our algorithm does not provide guarantees of performance and thus could cause harm if used in high-stakes domains without sufficient safety checks on learned behaviors.  Acknowledgements
We thank Jordan Schneider, Wenxuan Zhou, Caleb Chuck, and Amy Zhang for valuable feedback on the paper.
This work has taken place in the Personal Autonomous Robotics Lab (PeARL) at The
University of Texas at Austin.
PeARL research is supported in part by the NSF (IIS-1724157, IIS1749204, IIS-1925082), AFOSR (FA9550-20-1-0077), and ARO (78372-CS).
This research was also sponsored by the Army Research Office under Cooperative Agreement Number W911NF-19-2-0333.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research
Office or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.