A Ranking Game for Imitation Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose a new framework for imitation learning—treating imitation as a twoplayer ranking-based game between a policy and a reward.
In this game, the rewardagent learns to satisfy pairwise performance rankings between behaviors, while thepolicy agent learns to maximize this reward.
In imitation learning, near-optimalexpert data can be difficult to obtain, and even in the limit of infinite data cannotimply a total ordering over trajectories as preferences can.
On the other hand,learning from preferences alone is challenging as a large number of preferencesare required to infer a high-dimensional reward function, though preference data istypically much easier to collect than expert demonstrations.
The classical inversereinforcement learning (IRL) formulation learns from expert demonstrations butprovides no mechanism to incorporate learning from offline preferences and viceversa.
We instantiate the proposed ranking-game framework with a novel rankingloss giving an algorithm that can simultaneously learn from expert demonstrationsand preferences, gaining the advantages of both modalities.
Our experiments showthat the proposed method achieves state-of-the-art sample efficiency and can solvepreviously unsolvable tasks in the Learning from Observation (LfO) setting.
Introduction
Reinforcement learning relies on environmental reward feedback to learn meaningful behaviors.
Reward specification is a hard problem [39], thus motivating imitation learning (IL) as a techniqueto bypass reward specification and learn from expert data, often via Inverse Reinforcement Learning(IRL) techniques.
Learning from expert observations (imitation learning) alone can require efficientexploration when the expert actions are unavailable as in LfO [36].
Incorporating preferences overpotentially suboptimal trajectories for reward learning can help reduce the exploration burden byregularizing the reward function and providing effective guidance for policy optimization.
Previousliterature in learning from preferences either assumes no environment interaction [10, 9] or assumesan active query framework with a restricted reward class [47].
The classical IRL formulation suffersfrom two issues: (1) Learning from expert demonstrations and learning from preferences/rankingsprovide complementary advantages for increasing learning efficiency [30, 47]; however, existing IRL methods that learn from expert demonstrations provide no mechanisms to incorporate offlinepreferences and vice versa.
(2) Optimization is difficult, making learning sample inefficient [5, 28]due to the adversarial min-max game.
Our primary contribution is an algorithmic framework casting imitation learning as a ranking game which addresses both of the above issues in IRL.
This framework treats imitation as a ranking game between two agents: a reward agent and a policy agent—thereward agent learns to satisfy pairwise performance rankings between different behaviors represented as state-action or state visitations, while the policy agent maximizes its performance under the learned reward function.
The ranking game is detailed in Figure {+
+}
[Figure]
agent
Reward Agent
Dataset 𝐷 𝜌 !"
#$% ≼ 𝜌 & 𝜌 ’’ ≼ 𝜌 ’( 𝜌 (’ ≼ 𝜌 (( .. 𝜌 $)’’
≼ 𝜌 $( 𝜌 ’*$% ≼ 𝜌
(*$% 𝜌 (*$% ≼ 𝜌 +*$% .
𝜌 $)’
*$% ≼ 𝜌 $*$% 𝜌 $*$% ≼ 𝜌
[Figure]
and is specified by three components: (1) Thedataset of pairwise behavior rankings, (2) Aranking loss function, and (3)
An
strategy.
This game encompasses a large subsetof both inverse reinforcement learning (IRL)methods and methods which learn from suboptimal offline preferences.
Popular IRL methodssuch as GAIL, AIRL, f -MAX [28, 22, 34] areinstantiations of this ranking game in whichrankings are given only between the learningagent and the expert, and a gradient descentascent (GDA) optimization strategy is used witha ranking loss that maximizes the performancegap between the behavior rankings.
The ranking loss used by the prior IRLapproaches is specific to the comparison ofoptimal (expert) vs. suboptimal (agent) data,and precludes incorporation of comparisonsamong suboptimal behaviors.
In this work, weinstantiate the ranking game by proposing a newranking loss ( L k ) that facilitates incorporationof rankings over suboptimal trajectories forreward learning.
Our theoretical analysis reveals that the proposed ranking loss results in a boundedperformance gap with the expert that depends on a controllable hyperparameter.
Our ranking losscan also ease policy optimization by supporting data augmentation to make the reward landscapesmooth and allowing control over the learned reward scale.
Finally, viewing our ranking game in the
Stackelberg game framework (see Section 3)—an efficient setup for solving general-sum games—weobtain two algorithms with complementary benefits in non-stationary environments depending onwhich agent is set to be the leader.
In summary, this paper formulates a new framework rank-game for imitation learning that allowsus to view learning from preferences and demonstrations under a unified perspective.
We instantiatethe framework with a principled ranking loss that can naturally incorporate rankings provided by diverse sources.
Finally, by incorporating additional rankings—auto-generated or offline—our method:(a) outperforms state-of-the-art methods for imitation learning in several MuJoCo simulated domainsby a significant margin and (b) solves complex tasks like imitating to reorient a pen with dextrous manipulation using only a few observation trajectories that none of the previous LfO baselines can solve.
Related Work
Imitation learning methods are broadly divided into two categories:
Behavioral cloning [48, 54] and
Inverse Reinforcement Learning (IRL) [44, 1, 72, 18, 20, 28, 22].
Our work focuses on developing anew framework in the setting of IRL through the lens of ranking.
Table 1 shows a comparison of theproposed rank-game method to prior works.
Classical Imitation Game for IRL :
The classical imitation game for IRL aims to solve theadversarial min-max problem of finding a policy that minimizes the worst-case performance gapbetween the agent and the expert.
A number of previous works [22, 60, 34] have focused onanalyzing the properties of this min-max game and its relation to divergence minimization.
Undersome additional regularization, this min-max objective can be understood as minimizing a certainf -divergence [28, 22, 34] between the agent and expert state-action visitation.
More recently, [60]showed that all forms of imitation learning (BC and IRL) can be understood as performing momentmatching under differing assumptions. In this work, we present a new perspective on imitation IL Method Offline
Expert
Ranking Reward Active Human Preferences Data
Loss
Function
Query
MaxEntIRL, AdRIL,GAN-GCL, ✗
LfD supremum non-linear ✗ GAIL, f -MAX, AIRL
BCO,GAIfO, DACfO, ✗
LfO supremum non-linear ✗ OPOLO, f -IRL
TREX, DREX ✓ ✗
Bradley-Terry non-linear ✗
BREX ✓ ✗
Bradley-Terry linear ✗
DemPref ✓
LfO/LfD
Bradley-Terry linear ✓
Ibarz et al[30] ✓
LfD
Bradley-Terry non-linear ✓ rank-game ✓
LfO/LfD
L k non-linear ✗ in which the reward function is learned using a dataset of behavior comparisons, generalizingprevious IRL methods that learn from expert demonstrations and additionally giving the flexibilityto incorporate rankings over suboptimal behaviors.
is important when expert data is limited or hard to obtain.
Preferences [3, 65, 55, 14, 47] have theadvantage of providing guidance in situations expert might not get into, and in the limit providesfull ordering over trajectories which expert data cannot.
A previous line of work [10, 11, 9, 13] hasstudied this setting and demonstrated that offline rankings over suboptimal behaviors can be effectivelyleveraged to learn a reward function.
[14, 47, 30] studied the question of learning from preferencesin the setting when a human is available to provide online preferences 1 (active queries), while [47]additionally assumed the reward to be linear in known features.
Our work makes no such assumptionsand allows for integrating offline preferences and expert demonstrations under a common framework.
Learning from Observation (LfO): LfO is the problem setting of learning from expert observations.
This is typically more challenging than the traditional learning from demonstration setting (LfD),because actions taken by the expert are unavailable.
LfO is broadly formulated using two objectives:state-next state marginal matching [63, 71, 58] and direct state marginal matching [45, 43].
Someprior works [61, 67, 16] approach LfO by inferring expert actions through a learned inverse dynamicsmodel.
These methods assume injective dynamics and suffer from compounding errors when thepolicy is deployed.
A recently proposed method OPOLO [71] derives an upper bound for the LfOobjective which enables it to utilize off-policy data and increase sample efficiency.
Our methodoutperforms baselines including OPOLO, by a significant margin.
Background
We consider a learning agent in a Markov Decision Process (MDP) [49, 59] which can be definedas a tuple: M = ( S , A , P, R, γ, ρ 0 ) , where S and A are the state and action spaces; P is the statetransition probability function, with P ( s ′ | s, a ) indicating the probability of transitioning from s to s ′when taking action a ; R : S ×
A → R is the reward function bounded in [0 , R max ] ; We consider
MDPs with infinite horizon, with the discount factor γ ∈ [0 , 1] , though our results extend to finitehorizons as well; p 0 is the initial state distribution.
We use Π and R to denote the space of policiesand reward functions respectively.
A reinforcement learning agent aims to find a policy π : S → Athat maximizes its expected return, J ( R ; π ) =
11 − γ E ( s,a ) ∼ ρ π ( s,a ) [ R ( s, a )] , where ρ π ( s, a ) is thestationary state-action distribution induced by π .
In imitation learning, we are provided with samplesfrom the state-action visitation of the expert ρ π E ( s, a ) but the reward function of the expert isunknown.
We will use ρ E ( s, a ) as a shorthand for ρ π E ( s, a ) .
Classical Imitation Learning :
The goal of imitation learning is to close the imitation gap J ( R ; π E ) −
J ( R ; π ) defined with respect to the unknown expert reward function R .
Several prior works [28, 60,38, 45] tackle this problem by minimizing the imitation gap on all possible reward hypotheses.
This leads to a zero-sum (min-max) game formulation of imitation learning in which a policy is optimizedwith respect to the reward function that induces the largest imitation gap:
[Equation]
Here, the imitation gap is upper bounded as follows ( ∀ π ):
[Equation]
Note that, when the performance gap is maximized between the expert π E and the agent π , we canobserve that the worst-case reward function f π induces a ranking between policy behaviors basedon their performance: ρ E ⪰ ρ π :=
E ρ E
( s,a )
[ f π ( s, a )] ≥ E ρ π ( s,a )
[ f π ( s, a )] , ∀ π .
Therefore, we canregard the above loss function that maximizes the performance gap (Eq. 2) as an instantiation of theranking-loss.
We will refer to the implicit ranking between agent and the expert ρ E ⪰ ρ π as vanillarankings and this variant of the ranking-loss function as the supremum-loss .
Stackelberg Games : A Stackelberg game is a general-sum game between two agents where one agentis set to be the leader and the other a follower.
The leader in this game optimizes its objective under theassumption that the follower will choose the best response for its own optimization objective.
Moreconcretely, assume there are two players A and B with parameters θ A , θ B and corresponding losses
L A ( θ A , θ B ) and L B ( θ A , θ B ) .
A Stackelberg game solves the following bi-level optimization when A is the leader and B is the follower: min θ A L A ( θ A , θ ∗ B ( θ A ))
s.t θ ∗ B ( θ A ) =
arg min θ L B ( θ A , θ ) .[51] showed that casting model-based RL as an approximate Stackelberg game [6] leads toperformance benefits and reduces training instability in comparison to the commonly used GDA [56]and Best Reponse (BR) [12] methods.
[17, 69] prove convergence of Stackelberg games undersmooth player cost functions and show that they reduce the cycling behavior to find an equilibriumand allow for better convergence.
A Ranking Game for Imitation Learning
In this section, we first formalize the notion of the proposed two-player general-sum ranking gamefor imitation learning.
We then propose a practical instantiation of the ranking game through anovel ranking-loss ( L k ).
The proposed ranking game gives us the flexibility to incorporate additionalrankings—both auto-generated (a form of data augmentation mentioned as ‘auto’ in Fig. 1) andoffline (‘pref’ in Fig.
1)—which improves learning efficiency.
Finally, we discuss the Stackelbergformulation for the two-player ranking game and discuss two algorithms that naturally arise dependingon which player is designated as the leader.
4.1 The Two-Player Ranking Game Formulation
We present a new framework, rank-game , for imitation learning which casts it as a general-sumranking game between two players — a reward and a policy.
[Equation]
In this formulation, the policy agent maximizes the reward by interacting with the environment, andthe reward agent attempts to find a reward function that satisfies a set of pairwise behavior rankingsin the given dataset D p ; a reward function satisfies these rankings if E ρ πi [ R ( s, a )] ≤ E ρ [-π 
-]πj
[ R ( s, a )] ,∀ ρ π i ⪯ ρ π j ∈ D p , where ρ π i , ρ π j can be state-action or state vistitations.
The dataset of pairwise behavior rankings D p can be comprised of the implicit ‘vanilla’ rankingsbetween the learning agent and the expert’s policy behaviors ( ρ π ⪯ ρ E ), giving us the classical
IRL methods when a specific ranking loss function – supremum-loss is used [28, 22, 34].
Ifrankings are provided between trajectories, they can be reduced to the equivalent ranking between thecorresponding state-action/state visitations.
In the case when D p comprises purely of offline trajectoryperformance rankings then, under a specific ranking loss function ( Luce-shepard ), the ranking game
1: Initialize policy π 0 θ , reward funtion R ϕ , empty dataset D π . empirical expert data ˆ ρ E 2: for t = 1 ..T iterations do 3:
Collect empirical visitation data ˆ ρ π tθ with π tθ in the environment.
Set D π
= { (ˆ ρ π ⪯ ˆ ρ E ) } 4:
Train reward R ϕ to satisfy rankings in D π using ranking loss L k in equation 3.
Optimize policy under the reward function: π t +1 θ ← argmax π ′ J ( R ϕ ; π ′ ) 6: end for reduces to prior reward inference methods like T-REX [10, 11, 9, 13].
Thus, the ranking game affordsus a broader perspective of imitation learning, going beyond only using expert demonstrations.
4.2 Ranking Loss L k for the Reward Agent
We use a ranking-loss to train the reward function—an objective that minimizes the distortion [31]between the ground truth ranking for a pair of entities { x, y } and rankings induced by a parameterizedfunction R : X → R for a pair of scalars { R ( x ) , R ( y ) } .
One type of such a ranking-loss is thesupremum-loss in the classical imitation learning setup.
We propose a class of ranking-loss functions L k that attempt to induce a performance gap of k for allbehavior preferences in the dataset.
Formally, this can be implemented with the regression loss:
[Equation]
where D p contains behavior pairs ( ρ π i , ρ π j ) s.t ρ π i ⪯ ρ π j .
The proposed ranking loss allows for learning bounded rewards with user-defined scale k in the agentand the expert visitations as opposed to prior works in Adversarial Imitation Learning [28, 20, 22].
Reward scaling has been known to improve learning efficiency in deep RL; a large reward scale canmake the optimization landscape less smooth [27, 24] and a small scale might make the action-gapsmall and increase susceptibility to extrapolation errors [7].
In contrast to the supremum loss, L kcan also naturally incorporate rankings provided by additional sources by learning a reward functionsatisfying all specified pairwise preferences.
The following theorem characterizes the equilibrium ofthe rank-game for imitation learning when L k is used as the ranking-loss.
Theorem 4.1.
(Performance of the rank-game equilibrium pair) Consider an equilibrium of theimitation rank-game ( ˆ π, ˆ R ), such that the ranking loss L k generalization error is bounded by2 R 2 max ϵ r and the policy is near-optimal with J ( ˆ R ;
ˆ π ) ≥ J ( ˆ R ; π ) − ϵ π ∀ π , then at this equilibriumpair under the expert’s unknown reward function R gt bounded in [0 , R Emax ] :
[Equation]
If reward is a state-only function and only expert observations are available, the same bound appliesto the LfO setting.
Proof.
We defer the proof to Appendix A.
Theoretical properties: We now discuss some theoreticalproperties of L k .
Theorem 1 shows that rank-game hasan equilibrium with bounded performance gap with theexpert.
An optimization step by the policy player, under areward function optimized by the reward player, is equivalent to minimizing an f -divergence with the expert.
Equivalently, at iteration t in Algorithm 1: max π t E ρ πt
[ R ∗ t ] − E ρ πE
[ R ∗ t ] =
min π t D f ( ρ π t ∥ ρ π E ) .
We elaborate on theregret of this idealized algorithm in Appendix A. Theorem1 suggests that large values of k can guarantee the agent’sperformance is close to the expert.
In practice, we observeintermediate values of k also preserve imitation equilibrium optimality with a benefit of promoting sample efficient learning (as an effect of reward scaling described earlier).
We discuss this observation further in Appendix D.9.
rank-game naturally extends to the LfO regime under a state-only reward function where Theorem 4.1 results in a divergencebound between state-visitations of the expert and the agent.
A state-only reward function is also a sufficient and necessary condition to ensure that we learn a dynamics-disentangled reward function [20].
L k can incorporate additional preferences that can help learn a regularized/shaped reward functionthat provides better guidance for policy optimization, reducing the exploration burden and increasingsample efficiency for IRL.
A better-guided policy optimization is also expected to incur a lowerϵ π .
However, augmenting the ranking dataset can lead to decrease in the intended performance gap( k eff < k ) between the agent and the expert (Appendix A).
This can loosen the bound in Eq 4 and leadto non-optimal imitation learning.
We hypothesize that given informative preferences, decreased ϵ πcan compensate potentially decreased intended performance gap k eff to ensure near optimal imitation.
In our experiments, we observe this hypothesis holds true; we enjoy sample efficiency benefitswithout losing any asymptotic performance.
To leverage these benefits, we present two methods foraugmenting the ranking dataset below and defer the implementation details to Appendix B.
4.2.1 Generating the Ranking Dataset
Reward loss w/ automatically generated rankings (auto) :
In this method, we assume access to thebehavior generating trajectories in the ranking dataset.
For each pairwise comparison ρ i ⪯ ρ j presentin the dataset, L k sets the regression targets for states in ρ i to be 0 and for states visited by ρ j to bek .
Equivalently, we can rewrite minimizing L k as regressing an input of trajectory τ i to vector 0 , andτ j to vector k 1 where τ i , τ j are trajectories that generate the behavior ρ i , ρ j respectively.
We use thecomparison ρ i ⪯ ρ j to generate additional behavior rankings ρ i ⪯ [Equation] where 0 < λ 1 < λ 2 < ... <
λ P < 1 .
The behavior ρ λ p ,ij is obtained by independently samplingthe trajectories that generate the behaviors
ρ i , ρ j and taking convex combinations i.e τ λ p ,ij =λ
p τ i + (1 − λ p )
τ j and their corresponding reward regressions targets are given by λ p
+ (1 − λ p )
k 1 .
This form of data augmentation can be interpreted as mixup [68] regularization in the trajectoryspace.
Mixup has been shown to improve generalization and adversarial robustness [25, 68] byregularizing the first and second order gradients of the parameterized function.
Following the generalprinciple of using a smoothed objective with respect to inputs to obtain effective gradient signals,explicit smoothing in the trajectory-space can also help reduce the policy optimization error ϵ π .
Adidactic example showing rewards learned using this method is shown in Figure 2.
In a special casewhen the expert’s unknown reward function is linear in observations, these rankings reflect the trueunderlying rankings of behaviors.
Reward loss w/ offline annotated rankings (pref) :
Another way of increasing learning efficiencyis augmenting the ranking dataset containing the vanilla ranking ( ρ π ⪯ ρ E ) with offline annotatedrankings.
These rankings may be provided by a human observer or obtained using an offline datasetof behaviors with annotated reward information, similar to the datasets used in offline RL [19, 41].
We combine offline rankings by using a weighted loss between L k for satisfying vanilla rankings( ρ π ⪯ ρ E ) and offline rankings, grounded by an expert.
Providing offline rankings alone thatare sufficient to explain the reward function of the expert [10] is often a difficult task and thenumber of offline preferences required depends on the complexity of the environment.
In the LfOsetting, learning from an expert’s state visitation alone can be a hard problem due to explorationrequirements [36].
This ranking-loss combines the benefits of using preferences to shape the rewardfunction and guide policy improvement while using the expert to guarantee near-optimal performance.
4.3 Optimizing the Two-Player General-Sum Ranking Game as a Stackelberg Game
Solving the ranking-game in the Stackelberg setup allows us to propose two different algorithmsdepending on which agent is set to be the leader and utilize the learning stability and efficiencyafforded by the formulation as studied in [51, 69, 17].
Policy as leader (PAL) :
Choosing policy as the leader implies the following optimization:
[Equation]
Reward as leader (RAL): Choosing reward as the leader implies the following optimization:
[Equation]
We follow the first order gradient approximation for leader’s update from previous work [51] to develop practical algorithms.
This strategy has been proven to be effective and avoids the computationalcomplexity of calculating the implicit Jacobian term ( dθ ∗ B /dθ A ).
PAL updates the reward to near convergence on dataset D π
( D π contains rankings generated using the current policy agent only π ⪯ π E )and takes a few policy steps.
Note that even after the first-order approximation, this optimizationstrategy differs from GDA as often only a few iterations are used for training the reward even in hyperparameter studies like [46].
RAL updates the reward conservatively.
This is achieved through aggregating the dataset of implicit rankings from all previous policies obtained during training.
PAL’s strategyof using on-policy data D π for reward training resembles that of methods including GAIL [28, 62],f -MAX [22], and f -IRL [45].
RAL uses the entire history of agent visitation to update the rewardfunction and resembles methods such as apprenticeship learning and DAC [1, 37].
PAL and RALbring together two seemingly different algorithm classes under a unified Stackelberg game viewpoint.
Experimental Results
We compare rank-game against state-of-the-art LfO and LfD approaches on MuJoCo benchmarkshaving continuous state and action spaces.
The LfO setting is more challenging since no actions areavailable, and is a crucial imitation learning problem that can be used in cases where action modalitiesdiffer between the expert and the agent, such as in robot learning.
We focus on the LfO setting in thissection and defer the LfD experiments to Appendix D.2.
We denote the imitation learning algorithmsthat use the proposed ranking-loss L k from Section 4.2 as RANK-{PAL, RAL}.
We refer to therank-game variants which use automatically generated rankings and offline preferences as (auto)and (pref) respectively following Section 4.2.
In all our methods, we rely on an off-policy model-freealgorithm, Soft Actor-Critic (SAC) [26], for updating the policy agent.
We design experiments to answer the following questions:
Asymptotic Performance and Sample Efficiency : Is our method able to achieve near-expertperformance given a limited number (1) of expert observations?
Can our method learn using fewerenvironment interactions than prior state-of-the-art imitation learning (LfO) methods?2.
Utility of preferences for imitation learning : Current LfO methods struggle to solve a numberof complex manipulation tasks with sparse success signals.
Can we leverage offline annotatedpreferences through rank-game in such environments to achieve near-expert performance?3.
Choosing between PAL and RAL methods : Can we characterize the benefits and pitfalls of eachmethod, and determine when one method is preferable over the other?4.
Ablations for the method components : Can we establish the importance of hyperparameters anddesign decisions in our experiments?
Baselines: We compare RANK-PAL and RANK-RAL against 6 representative LfO approaches thatcovers a spectrum of on-policy and off-policy model-free methods from prior work: GAIfO [62, 28],
DACfO [37], BCO [61], f -IRL [45] and recently proposed OPOLO [71] and IQLearn [21].
We donot assume access to expert actions in this setting.
Our LfD experiments compare to the IQLearn [21],
DAC [37] and BC baselines.
Detailed description for baselines can be found in Appendix D.2.
5.1 Asymptotic Performance and Sample Efficiency
In this section, we compare RANK-PAL(auto) and RANK-RAL(auto) to baselines on a set of MuJoColocomotion tasks of varying complexities: Swimmer-v2, Hopper-v2, HalfCheetah-v2,
Walker2d-v2, Ant-v2 and Humanoid-v2 .
In this experiment, we provide one expert trajectory for all methods and do not assume access to any offline annotated rankings.
Asymptotic Performance :
Table 2 shows that both rank-game methods are able to reach nearexpert asymptotic performance with a single expert trajectory.
BCO shows poor performance which
[Table]
can be attributed to the compounding error problem arising from its behavior cloning strategy.
GAIfOand DACfO use GDA for optimization with a supremum loss and show high variance in theirasymptotic performance whereas rank-game methods are more stable and low-variance. [Figure] Walker [Figure]
v
[Figure]
Humanoid-v
[Figure]
Sample Efficiency : Figure 3 shows that RANK-RAL and RANK-PAL are among the most sampleefficient methods for the LfO setting, outperforming the recent state-of-the-art method
OPOLO [71]by a significant margin.
We notice that IQLearn fails to learn in the LfO setting.
This experimentdemonstrates the benefit of the combined improvements of the proposed ranking-loss with automatically generated rankings.
Our method is also simpler to implement than OPOLO, as we requirefewer lines of code changes on top of SAC and need to maintain fewer parameterized networkscompared to OPOLO which requires an additional inverse action model to regularize learning.
[Figure]
5.2 Utility of Preferences in Imitation
Our experiments on complex manipulation environments—door opening with a parallel- jaw gripper [70] and pen manipulation with adexterous adroit hand [50] – reveal that noneof the prior LfO methods are able to imitatethe expert even under increasing amounts ofexpert data.
This failure of LfO methods canbe potentially attributed to the explorationrequirements of LfO compared to LfD [36],coupled with the sparse successes encounteredin these tasks, leading to poorly guided policy gradients.
In these experiments, we show thatrank-game can incorporate additional information in the form of offline annotated rankings toguide the agent in solving such tasks.
These offline rankings are obtained by uniformly samplinga small set of trajectories (10) from the replay buffer of SAC [26] labeled with a ground truth rewardfunction.
We use a weighted ranking loss (pref) from Section 4.2.
Figure 4 shows that RANK-PAL/RAL(pref) method leveraging offline ranking is the only method thatcan solve these tasks, whereas prior LfO methods and RANK-PAL/RAL(auto) with automatically generated rankings struggle even after a large amount of training.
We also point out that T-REX, amethod that learns using the preferences alone is unable to achieve near-expert performance, therebyhighlighting the benefits of learning from expert demonstrations alongside a set of offline
preferences.5.3 Comparing PAL and RAL
[Figure]
PAL uses the agent’s current visitation for reward learning, whereas RAL learns a rewardconsistent with all rankings arising from the history of the agent’s visitation.
These propertiescan present certain benefits depending on thetask setting.
To test the potential benefits of PALand RAL, we consider two non-stationary imitation learning problems, similar to [50] – one inwhich the expert changes it’s intent and the otherwhere dynamics of the environment change during training in the Hopper-v2 locomotion task.
For changing intent, we present a new set ofdemonstrations where the hopper agent hopsbackwards rather than forward.
For changingenvironment dynamics, we increase the mass ofthe hopper agent by a factor of 1.2.
Changes areintroduced at 1e5 time steps during training at which point we notice a sudden performance drop.
In Figure 5 (left), we notice that PAL adapts faster to intent changes, whereas RAL needs to unlearnthe rankings obtained from the agent’s history and takes longer to adapt.
Figure 5 (right) shows that
RAL adapts faster to the changing dynamics of the system, as it has already learned a good globalnotion of the dynamics-disentangled reward function in the LfO setting, whereas PAL only has a localunderstanding of reward as a result of using ranking obtained only from the agent’s current visitation.
Ablation of Method Components: Appendix D contains eight additional experiments to study theimportance of hyperparameters and design decisions.
Our ablations validate the importance of usingautomatically generated rankings, the benefit of ranking loss over supremum loss, and sensitivity tohyperparameters like the intended performance gap k , policy iterations, and the reward regularizer.
Conclusion
In this work, we present a new framework for imitation learning that treats imitation as a two-playerranking-game between a policy and a reward function.
Unlike prior works in imitation learning, theranking game allows incorporation of rankings over suboptimal behaviors to aid policy learning.
Weinstantiate the ranking game by proposing a novel ranking loss which guarantees agent’s performanceto be close to expert for imitation learning.
Our experiments on simulated MuJoCo tasks reveal thatutilizing additional ranking through our proposed ranking loss leads to improved sample efficiencyfor imitation learning, outperforming prior methods by a significant margin and solving some taskswhich were unsolvable by previous LfO methods.
and Negative Societal Impacts:
Preferences obtained in real world are usuallynoisy [40, 32, 8] and one limitation of rank-game is that it does not suggest a way to handle noisypreferences.
Second, rank-game proposes modifications to learn a reward function amenable topolicy optimization but these hyperparameters are set manually.
Future work can explore methods toautomate learning such reward functions.
Third, despite learning effective policies we observed thatwe do not learn reusable robust reward functions [45].
Negative Societal Impact: Imitation learningcan cause harm if given demonstrations of harmful behaviors, either accidentally or purposefully.
Furthermore, even when given high-quality demonstrations of desirable behaviors, our algorithm doesnot provide guarantees of performance, and thus could cause harm if used in high-stakes domainswithout sufficient safety checks on learned behaviors.