Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
Anonymous Author(s)
AfﬁliationAddress email
Abstract: A promising approach to solving challenging long-horizon tasks hasbeen to extract behavior priors (skills) by ﬁtting generative models to large ofﬂinedatasets of demonstrations.
However, such generative models inherit the biasesof the underlying data and result in poor and unusable skills when trained on imperfect demonstration data.
To better align skill extraction with human intent wepresent Skill Preferences (SkiP), an algorithm that learns a model over humanpreferences and uses it to extract human-aligned skills from ofﬂine data.
After extracting human-preferred skills, SkiP also utilizes human feedback to solve downstream tasks with RL.
We show that SkiP enables a simulated kitchen robot tosolve complex multi-step manipulation tasks and substantially outperforms priorleading RL algorithms with human preferences as well as leading skill extractionalgorithms without human preferences.
Introduction
Deep reinforcement learning (RL) is a framework for solving temporally extended tasks that hasresulted in a number of breakthroughs in autonomous control including mastery of the game of
Go [1, 2], learning to play video games [3, 4, 5], and learning basic robotic control [6, 7].
However,today’s RL systems require substantial manual human effort to engineer rewards for each task whichcomes with two fundamental drawbacks.
The human effort required to design rewards is impractical to scale across numerous and diverse task categories and the engineered rewards can often beexploited by the RL agent to produce unintended and potentially unsafe control policies [8, 9, 10].
Moreover, it becomes increasingly difﬁcult to design reward functions for the kinds of complex taskswith compositional structure often encountered real-world settings.
In this work, we are interestedin the following research question - how can we learn robotic control policies that are aligned withhuman intent and capable of solving complex real-world tasks?
Human-in-the-loop RL [11, 12, 13] has emerged as a promising approach to better align RL withhuman intent that proposes an alternate approach to traditional RL algorithm design.
Rather thanmanually engineering a reward function and then training the RL agent, human-in-the-loop RLproposes for humans to provide feedback interactively to the agent as it is training.
This paradigmshift sidesteps reward exploitation by providing the RL algorithm immediate feedback to align itbest with human intent and, if efﬁcient in terms of human labels required, has the potential to scale
RL training across a diverse variety of tasks more reliably than reward engineering.
So far human-in-the-loop RL systems have been used to play Atari games [12], solve simulatedlocomotion and manipulation tasks [11, 13], and better align the output of language models [14].
While these initial results have been promising, human-in-the-loop methods are still out of reach forthe kinds of long-horizon compositional tasks that are desired for real-world robotics.
The primaryreason is that current methods do not scale efﬁciently with respect to human labels for more challenging tasks.
As task complexity increases, the number of human feedback interactions required toattain a suitable policy becomes impractical.
To address the ability of RL algorithms to scale to more complex long-horizon tasks, a number ofrecent works [15, 16, 17] have proposed data-driven extraction of behavioral priors, which we referto as skills .
In these methods, a behavioral prior is ﬁt to an ofﬂine dataset of demonstrations andis then used to guide the RL policy to solve downstream tasks by regularizing it to stay near thebehavioral distribution.
Such methods have been shown to successfully solve tasks such as diverseobject manipulation [16] and operating a kitchen with a robotic arm [15].
However, they still requireengineered rewards for the downstream tasks and, more importantly, assume access to a clean ofﬂinedataset of expert demonstrations that are speciﬁcally relevant to the downstream tasks.
In real-worldscenarios, such clean datasets are highly unlikely to exist.
We desire skill extraction methods thatare robust to noisy datasets, collected by a range of policies, with highly multi-modal structure.
In this work, we introduce Skill Preferences (SkiP), an algorithm that integrates human-in-the-loop
RL with data-driven skill extraction.
Our main insight is that human feedback can be incorporatednot only for downstream RL, as is done in prior work, but also for extracting human-aligned skills.
SkiP learns a human preference function and uses it to weigh the likelihood of trajectories in theofﬂine dataset based on their degree of alignment with human intent.
By incorporating human feedback during skill extraction, SkiP is able to extract structured human-preferred skills from noisyofﬂine data and addresses the core limitation of prior skill extraction approaches - the dependenceon curated expert datasets.
SkiP is both capable of efﬁciently extracting skills and solving differentdownstream tasks with respect to human labels.
Similar to how prior work in human-in-the-loop
RL suggested replacing manually engineered reward functions with human feedback, our work suggests to replace the manual effort needed to curate clean ofﬂine datasets with human feedback.
Wesummarize our main contributions below:
We introduce Skill Preferences (SkiP), an algorithm that incorporates human feedback toextract skills from ofﬂine data and utilize those skills to solve downstream tasks.2.
We show that, unlike prior leading methods for data-driven skill extraction, SkiP is able toextract structured skills from noisy ofﬂine datasets.3.
We show that SkiP is able to solve complex multi-step manipulation tasks in robotic kitchen
environment substantially more efﬁciently than prior leading human-in-the-loop and skillextraction baselines.
Background
Reinforcement Learning: As is common with RL methods, we assume that the control process isa Markov Decision Process (MDP) with discounted returns.
Such MDPs are deﬁned by the tuple
M = ( S , A , R, ρ 0 , γ ) consisting of states s ∈ S , actions a ∈ A , rewards R =
R ( s, a ) , an initialstate distribution s 0 ∼ ρ 0 ( · ) , and a discount factor γ ∈ [0 , 1) .
A control policy maps states toactions within the MDP and usually takes the form of a probability distribution – a ∼ π ( ·| s ) .
The value function V π ( s ) and action-value function Q π ( s, a ) describe the value with respect to futureexpected returns with respect to an initial state or state-action pair.
[Equation]
where the ﬁrst expectation E M ,π denotes actions are sampled according to π and future states aresampled according to the MDP dynamics.
The goal in RL is the learn the optimal policy:
[Equation]
In addition to the standard MDP setting, our method will also learn skills z ∈ Z which consistof an encoder that maps state-action sequences to a skill q ( e ) ( z | s t , a t , . . .
, s t
+ H − 1 , a t + H − 1 ) and adecoder that maps state-skill pairs to atomic actions q ( d ) ( a 1 , a 2 , ..., a H | s, z ) .
Method
The two primary contributions of SkiP are (i) introducing human feedback during the skill extractionprocess to learn structured skills from noisy data and (ii) utilizing human preferences over skillsfor downstream RL training.
Our approach shown schematically in Fig.
1 and detailed in full in
Algo.
Due to utilizing human feedback to learn the behavioral prior, unlike prior approaches ofskill extraction from ofﬂine data [17, 16, 15], our method is robust to suboptimal or noisy data.
The SkiP Algorithm:
We ﬁrst summarize the algorithm and then proceed with its derivation.
Shownin Algo.
1, SkiP consists of two phases - (i) skill extraction and (ii) skill execution .
A human teacherprovides feedback during both phases.
During skill extraction, a human teacher labels whether atrajectory is preferred or not (for details see Sec. 4) to train a preference classiﬁer.
A behavioral prioris then ﬁt to the ofﬂine data with a weighted human preference function.
During skill execution, thelearned skills are rolled out by an RL agent - a Soft Actor-Critic (SAC) [18] - that is trained with taskspeciﬁc human preferences.
As such, human feedback is used during both phases of the algorithm.
We proceed to deﬁne notation and provide a derivation.
Preliminaries and Notation: Our method is composed of two phases - (i) the skill extractionphase and (ii) the skill execution phase.
During the skill extraction phase, we are given an ofﬂine dataset D which consists of task-agnostic, multi-modal, and potentially noisy demonstrations.
We denote trajectory sequences as τ t =
( s t , a t , . . .
, s t
+ H − 1 , a t + H − 1 ) , action sequencesas a t = ( a t , . . .
, a t + H − 1 ) , and skills which decode into action sequences as z ∈ Z .
Learning Behavioral Priors with Human Feedback (Skill Extraction): Our main insight is to usehuman preferences in order to ﬁt a weighted behavioral prior over an ofﬂine dataset of (potentiallynoisy) demonstrations.
Our method builds on prior work for behavioral extraction from ofﬂine datavia expected maximum likelihood latent variable models [17, 16, 15].
Speciﬁcally, prior work [17, 16, 15] considers a parameterized generative model p α
( a t | s t ) overaction sequences where a t =
( a t , . .
. , a t + H − 1 ) that represents a behavioral prior and is trained toreplicate the transition statistics in the ofﬂine dataset:
[Equation]
In our approach, we consider an adaptive behavioral prior that is biased towards trajectories thatachieve higher rewards according to the human preference function.
This can be particularly usefulin diverse datasets collected with suboptimal or noisy policies or multiple policies of varying expertise.
For example, one could imagine multiple humans collecting demonstrations or multiple
robotsexploring their environment.
Similar to Siegel et al.
[19], we seek a behavioral prior that is biasedtowards the high reward trajectories in the dataset while also staying close to the average statisticsin the dataset.
However, unlike prior work on weighted behavioral priors [19, 20, 21] the weight isdetermined through the human preference function and we aim to maximize action-sequence likelihood as opposed to single-timestep actions.
=
===
Skill Extraction Phase =
==
=
INPUT: ofﬂine dataset ˜ B
Initialize prior p , skill encoder q φ 2 and skill decoder p φ 1 .
Initialize learned preference classiﬁer P ψ
A human provides labels ( y 1 , y 2 , ... ) for 10% of the trajectories in ˜ B and stores them in a new buffer ˜ D for each iteration do Update ψ by maximizing E ( y,τ ) ∼ ˜ D
[ y · logP ψ ( τ ) + (1 − y ) · log (1 − P ψ ( τ ))] for each iteration do Update p , q φ 2 , p φ 1 by optimizing L prior (3) { Update preference weighted behavioral prior }
=
===
Skill Execution Phase =
===
Initialize parameters of actor π θ 1 , critics Q θ 2 and Q ¯ θ 2 and reward model (cid:98) R η
Initialize a dataset of preference D ← ∅ and a dataset of transitions B ← ∅ for Each iteration do for Each environment step do z t ∼ π ( z t | s t ) , s t + H ∼ p ( s t + H | s t , z t ) , B ← B ∪ ( s t , z t , (cid:98) R η ( s t , z t ) , s t + H ) if iteration % K =
=
then for step t = 1...
M do ( τ ( z ) 0 , τ ( z ) 1 ) ∼ B , query human for label y , D ← D ∪ ( τ ( z ) 0 , τ ( z ) 1 , y ) { Get preference labels } for each gradient step of (cid:98) R η do
Sample ( τ ( z ) 0 , τ ( z ) 1 , y ) ∼ D , update (cid:98) R η with min L reward (7) { Update preferences }
Relabel entire replay buffer B with (cid:98) R η for each gradient step of agent do
Sample ( s, a, s (cid:48) , R ) ∼ B , update π θ 1 by optimizing L SACactor (Appendix B.1) { Update agent }
Update Q θ 2 and Q ¯ θ 2 by optimizing L SACcritic (Appendix B.1)
We formulate this as:
[Equation]
 such that E τ ∼D
[ D KL [Equation] p α
[Equation]
p
[Equation]
[Equation]
δ [Equation] [Equation] where ¯ p denotes the empirical behavioral policy and ω ( s t , a t ) is the weighting function.
The nonparametric solution to the above optimization is given by:
[Equation]
where we have used ∝ to avoid speciﬁcation of the normalization factor, and η represents a temperature parameter that is related to the constraint level δ .
The above non-parametric policy can beprojected into the space of parametric neural network policies as [20, 19]:
[Equation]
For the choice of the weighting function, we use the learned preference classiﬁer P ψ ( y | τ ) whichinputs a trajectory and outputs the likelihood of this trajectory being human-preferred with y ∈ [0 , 1] .
P ψ ( y | τ ) is learned by sampling a small subset of the ofﬂine dataset and soliciting human feedbackto label preferred versus not preferred trajectory: ω ( τ t ) :
= log P ψ ( τ t ) .
In this process, we treat the temperature η as the hyper-parameter choice.
This implicitly deﬁnes theconstraint threshold δ , and makes the problem speciﬁcation and optimization more straightforward.
For our practical implementation, we ﬁt a variational autoencoder similar to [17, 15] but softlyweighted to maximize the likelihood of human-preferred transitions.
We introduce a latent variablez with a Guassian prior such that the ELBO loss is given by:
[Equation]
This is the standard β -VAE loss applied to action sequence modeling where β is a scalar controllingthe regularization strength and φ 1 , φ 2 are neural network parameters that are optimized during training.
Note that q φ 2 encodes trajectories into a latent vector and p φ 1 decodes latent vectors and the starting state back into action sequences.
Our training objective weighs this loss with the preferencefunction.
Thus, our overall skill extraction objective is to maximize:
[Equation]
Reward learning and human preferences over skills (Skill Execution): Unlike traditional RLwhere the hand-engineered rewards are available, we consider the preference-based RL framework [11, 12, 13, 22]: a (human) teacher provides preferences between the agent’s behaviors and theagent uses this feedback to perform the task.
In order to incorporate human preferences into deep
RL, Christiano et al.
[11] proposed a framework that learns a reward function (cid:98) R η from preferences.
In this work, we modify the preference framework to operate not over atomic state-action transitionsbut rather state-skill transitions that have substantially longer time spans.
Formally, we assume access to an ofﬂine dataset (the agent’s replay buffer)
B of state-action transitions and sample state-skill sequence pairs τ ( z ) 1 , τ ( z )for which a human provides a binary labely ∈ { 0 , 1 } , where τ ( z ) =
( s t , z t , s t
+ H , z t + H , . . .
, s ( t + M ) H , z ( t + M ) H ) where H is the length ofactions the skill decodes to and M is the total number of state-skill transitions.
Note how suchtrajectories are H times longer than if we were to sample state-action trajectories of length M .
The reward function (cid:98) R therefore ﬁts a Bernoulli distribution across sequences.
In this work, welearn a parameterized reward function (cid:98) R η as in [13] utilizing a Bradley-Terry model [23] in thefollowing manner:
[Equation]
Here, the operator A (cid:31) B means that A is preferred to B .
(cid:98) R η can therefore be interpreted as abinary preference classiﬁer where labels are provided through human feedback.
The parameters ηof the neural network are updated by optimizing a binary cross-entropy loss:
[Equation]
Experimental Setup
Environments: For our experiments, we use the robot kitchen environment and ofﬂine dataset fromthe D4RL suite [24].
This environment requires a 7-DOF (6-DOF arm and 1-DOF gripper) roboticarm to solve complex multi-step tasks in a kitchen.
Due to the 7-DOF control and compositionallong-horizon nature of the tasks, this environment cannot be solved by standard methods such as
SAC or behavior cloning [15].
Ofﬂine dataset: We desire our method to work on suboptimal ofﬂine data and, unlike prior skillextraction approaches [15, 16, 17] do not assume that the ofﬂine dataset consists solely of expertdemonstrations.
We simulate a noisy ofﬂine dataset by combining 601 expert trajectories and 601noisy trajectories generated by random policy.
The expert trajectories involve various structuredkitchen interactions such as opening the microwave and operating the stove.
We solicit humanfeedback on 10% of the total trajectories or equivalently 120 human labels .
Downstream tasks: We use 6 different downstream tasks shown in Fig. 2 that vary in difﬁculty toevaluate our approach.
The task suite consists of tasks that require one, two, or three subtasks tobe completed in a row in order to achieve the overall goal.
We note that even the tasks with onesubtask is challenging for RL methods that operate over atomic actions and do not leverage skills,as is shown in the experimental results.
Simulated human: Similar to prior work [11, 13], we obtain feedback from simulated humanteachers instead of real humans.
During skill extraction, human provides labels whether a trajectory is noisy or structured.
During skill execution, the simulated human assigns positive labels totrajectory segments that have made more progress toward completing the desired task.
Progress iscalculated by computing || s M · H − ¯ s ||
2 − || s 1 − ¯ s || 2 , where ¯ s is the state when the target task iscompleted.
[Figure] [Figure]
with a noisy ofﬂine dataset which consists of both expert and random actions our method ﬁts a behavioral prior to the ofﬂine data using human feedback to identify human-preferred motions which results in a set of diverse skills that can then be ﬁnetuned to downstream tasks.
Baselines: In addition to our method, we compare to Atomic Preferences which we based on PEBBLE [13]: a state-of-the-art human preference RL method.
it pretrains the SAC agent with behaviorcloning over the optimal ofﬂine dataset and trains the online SAC agent with human preferences overatomic transitions instead of high-level skill transitions.
We also compare to Flat Prior which learnsa single-step action prior on the atomic action space over the optimal dataset and trains an online
SAC agent regularized with the action prior over ground-truth reward.
The Oracle we compare to is
SPiRL , a leading skill extraction with access to the ground truth (expert demonstrations and groundtruth reward) in Fig.
Experimental Results
For the experimental evaluation of our approach, we investigate the following questions: (a)
Can
SkiP solve challenging long-horizon tasks and how does our method compare to prior leading approaches?
(b) How do SkiP compare to an oracle baseline that extracts skills from perfect expertdemonstrations and has access to the ground truth reward?
(c) Is it necessary to provide human feedback during skill extraction or is it sufﬁcient to ﬁt an unweighted behavioral prior over the ofﬂinedata?
(d) How should we incorporate human feedback during the skill execution phase?
Main Results: We evaluate SkiP and related baselines on the 6 tasks shown in Fig.
2 and displaythe learning curves in Fig.
We observe that SkiP is the only method (except for the Oracle) that iscapable of solving the majority of tasks in the robot kitchen task suite and outperforms the baselineson all environments.
On 5 out of 6 tasks, SkiP is able to match the oracle baseline asymptoticallywhich means that it arrives at the optimal solution.
SkiP is also human-label efﬁcient.
During skill extraction, only 120 labels are required to trainthe preference classiﬁer.
During skill execution, 300-1K labels are required to solve most tasksdepending on the task’s complexity.
We hypothesize that human label efﬁciency is better during theskill extraction phase because classifying structured and noisy skills from a static ofﬂine dataset is easier than classifying task-speciﬁc preferences from an evolving replay buffer.
Further human labelefﬁciency improvements pose interesting research directions for future work.
Ablation Studies: To further understand the properties of the SkiP algorithm, we investigatewhether human feedback is necessary during skill extraction as well as how the human preferencereward function compares to alternate approaches to human feedback during skill execution.
Is it necessary to provide human feedback during skill extraction or is it sufﬁcient to ﬁt an unweighted behavioral prior over the ofﬂine data?
The ofﬂine dataset used throughout this paperconsists of suboptimal data that is a mixture of expert and random actions.
We compare ﬁtting ahuman-feedback weighted behavioral prior as opposed to an unweighted behavioral prior that maximizes the likelihood of all action sequences equally.
For the skill execution phase, both methodshave access to the same human preference reward function.
The results shown in Fig.
5 indicatethat the method, which extracts skills without human feedback, is unable to solve any of the tasks suggesting that human feedback is essential for skill extraction from suboptimal ofﬂine data.
How should we incorporate human feedback during the skill execution phase?
Instead of preferences, a simpler approach to learning from human feedback is to provide binary feedback if a task(or subtask) has been solved and learning a reward classiﬁer to guide the RL agent.
We implement this by providing a positive reward of 1 for a high-level transition ( s t , z, s t + H ) when a subtask hasbeen completed and 0 otherwise.
Using the same number of human queries for both approaches, wecompare learning with preferences as opposed to learning from sparse rewards.
For both approaches,we use human feedback for skill extraction.
As shown in Fig.
6, RL with a reward classiﬁer for subtask completion is able to solve some tasks but generally performs much worse than RL with humanpreferences.
Related Work
Human-in-the-loop Reinforcement Learning:
Several works have successfully utilized feedbackfrom real humans to train RL agents [25, 11, 12, 26, 13, 27, 28].
One of major directions is directlyutilizing the human feedback as a learning signal [29, 27, 25] but assumed unlimited access tohuman labels which limited their practicality for more challenging tasks.
To address this limitation,a number of works proposed learning reward model from human feedback [26, 28, 30, 31, 32, 33].
Recently, several works have successfully combined human preferences with deep RL algorithmsto learn basic locomotion skills as well as playing video games from pixels using human [11, 12,34, 13].
However, these methods are limited to short-horizon or cyclic tasks and do not scale tomore challenging compositional multi-step tasks.
In this work, we investigate how to scale humanpreferences to such challenging tasks by specifying preferences over skills.
Data-driven Extraction of Behavioral Priors: Behavioral prior or skill extraction refers to ﬁttinga distribution over an ofﬂine dataset of demonstrations and biasing the agent’s policy towards themost likely actions from that distribution.
Commonly used for ofﬂine RL [21, 19, 20], behavioralpriors learned through maximum likelihood latent variable models can also been used as skills forstructured exploration in RL [16], to solve complex long-horizon tasks from sparse rewards [15, 17],and regularize ofﬂine RL policies [21, 20, 35].
A limitation of these skill extraction methods is thatthe quality of the behavioral prior is highly dependent on the demonstrations in the ofﬂine dataset.
Since a behavioral prior models maximum likelihood transitions in the ofﬂine dataset, suboptimal,noisy, or irrelevant transitions can degrade downstream policy learning.
In this work, we introducehuman feedback into the skill extractions phase to learn a human preferred behavioral prior whichenables skill extraction methods to be robust to suboptimal ofﬂine data.
Conclusion
We presented Skill Preferences (SkiP) an algorithm that uses human feedback for both skill extraction as well as execution, and showed that SkiP enables robotic agents to solve long-horizon compositional manipulation tasks.
We hope that this work excites other researchers about the potentialof learning with skills and human feedback.
Acknowledgements
We would like to thank Berkeley DeepDrive, Tencent, ONR Pecase N and NSF NRI2024675 for supporting this research.