D EMONSTRATION -G UIDED R EINFORCEMENT L EARNING WITH L EARNED S KILLS
A BSTRACT
Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback and a set of target task demonstrations.
Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator’s exact muscle movements.
Naturally, such learning will be slow, but often new behaviors are not completely unseen: they share subtasks with behaviors we have previously learned.
In this work, we aim to exploit this shared subtask structure to increase the efﬁciency of demonstrationguided RL.
We ﬁrst learn a set of reusable skills from large ofﬂine datasets of prior experience collected across many tasks.
We then propose an algorithm for demonstration-guided RL that efﬁciently leverages the provided demonstrations by following the demonstrated skills instead of the primitive actions, resulting in substantial performance improvements over prior demonstration-guided RL approaches.
We validate the effectiveness of our approach on long-horizon maze navigation and complex robot manipulation tasks.
I NTRODUCTION
[Figure]
Humans are remarkably efﬁcient at acquiring new skills from demonstrations: often a single demonstration of the desired behavior and a few trials of the task are sufﬁcient to master it (Bekkering et al., 2000;
Al-Abood et al., 2001;
Hodges et al., 2007).
To allow for such efﬁcient learning, we can leverage a large number of previously learned behaviors (Al-Abood et al., 2001; Hodges et al., 2007).
Instead of imitating precisely each of the demonstrated muscle movements, humans can extract the performed skills and leverage the rich repertoire of already acquired skills to efﬁciently reproduce the desired behavior.
Demonstrations are also commonly used in reinforcement learning (RL) to guide exploration and improve sample efﬁciency (Vecerik et al., 2017; Hester et al., 2018; Rajeswaran et al., 2018; Nair et al., 2018; Zhu et al., 2018).
However, such demonstration-guided RL approaches attempt to learn tasks from scratch : analogous to a human trying to imitate a completely unseen behavior by following every demonstrated muscle movement, they try to imitate the primitive actions performed in the provided demonstrations.
As with humans, policies learned with such step-by-step imitation are brittle (Ross et al., 2011), and thus these approaches require a large number of demonstrations and environment interactions to learn a new task.
We propose to improve the efﬁciency of demonstration-guided RL by leveraging prior experience in the form of an ofﬂine “task-agnostic“ experience dataset, collected not on one but across many tasks (see Figure 1).
Given such a dataset, our approach extracts reusable skills: robust short-horizon behaviors that can be recombined to learn new tasks.
Like a human imitating complex behaviors through the chaining of known skills, we can leverage this rich repertoire of skills for efﬁcient demonstration-guided RL on a new task by guiding policy learning using the demonstrated skills instead of the primitive actions.
Concretely, we propose a demonstration-guided RL algorithm that builds on recent works in skillbased RL (Pertsch et al., 2020a; Ajay et al., 2020) in order to learn low-dimensional representations of short-horizon skills from ofﬂine datasets, and then learns new tasks efﬁciently by leveraging these skills to follow a given set of demonstrations.
Across challenging navigation and robotic manipulation tasks we ﬁnd that our approach signiﬁcantly improves the learning efﬁciency over prior demonstration-guided RL approaches.
In summary, the contributions of our work are threefold: (1) we introduce the problem of leveraging task-agnostic ofﬂine datasets for accelerating demonstration-guided RL on unseen tasks, (2) we propose a skill-based algorithm for efﬁcient demonstration-guided RL and (3) we show the effectiveness of our approach on a maze navigation and two complex robotic manipulation tasks.
R ELATED W ORK
Imitation learning.
Learning from Demonstration, also known as imitation learning (Argall et al., 2009), is a common approach for learning complex behaviors by leveraging a set of demonstrations.
Most prior approaches for imitation learning are either based on behavioral cloning (BC, Pomerleau (1989)), which uses supervised learning to mimic the demonstrated actions, or inverse reinforcement learning (IRL, Abbeel & Ng (2004);
Ho & Ermon (2016)), which infers a reward from the demonstrations and then trains a policy to optimize it.
However, BC commonly suffers from distribution shift and struggles to learn robust policies (Ross et al., 2011), while IRL’s joint optimization of reward and policy can result in unstable training.
More generally, the performance of most imitation learning methods is upper bounded by the performance of the demonstrator.
Demonstration-guided RL.
A number of prior works aim to mitigate these problems by combining reinforcement learning with imitation learning.
This allows the agent to leverage demonstrations for overcoming exploration challenges in RL while using RL to increase robustness and performance of the imitation learning policies.
Prior work on demonstration-guided RL can be categorized into three groups: (1) approaches that use BC to initialize and regularize policies during RL training (Rajeswaran et al., 2018; Nair et al., 2018), (2) approaches that place the demonstrations in the replay buffer of an off-policy RL algorithm (Vecerik et al., 2017; Hester et al., 2018), and (3) approaches that augment the environment rewards with rewards extracted from the demonstrations (Zhu et al., 2018;
Peng et al., 2018; Merel et al., 2017).
While these approaches can leverage demonstrations to improve the efﬁciency of RL, they all treat each new task as an independent learning problem, i.e., attempt to learn policies from scratch without taking any prior experience into account.
As a result, they require many demonstrations to learn effectively, which is especially expensive since a new set of task-speciﬁc demonstrations needs to be collected for every new task.
Online RL with ofﬂine datasets.
As an alternative to expensive task-speciﬁc demonstrations, multiple recent works have proposed to accelerate reinforcement learning by leveraging task-agnostic experience in the form of large datasets collected across many tasks (Pertsch et al., 2020a; Siegel et al., 2020; Nair et al., 2020; Ajay et al., 2020; Singh et al., 2021; 2020).
In contrast to demonstrations, such task-agnostic datasets can be collected cheaply from a variety of sources, e.g., using previously trained agents across a diverse set of tasks (Fu et al., 2020; Gulcehre et al., 2020), through agents autonomously exploring their environment (Hausman et al., 2018; Sharma et al., 2020) or via human teleoperation (Schaal et al., 2005; Gupta et al., 2019; Mandlekar et al., 2018; Lynch et al., 2020).
Once collected, the dataset can be reused for learning many downstream tasks.
Though being cheaper to collect, the task-agnostic data will usually not contain demonstrations for the downstream task, making learning less efﬁcient than demonstration-guided approaches.
ssssssaaaa 4 aq ω ( z | s , a ) π ϕ ( a t | s t , z ) s t π θ ( z | s t ) π ϕ ( a t | s t , z )
Environment z a t
Pre-Trained q ζ ( z | s 0 ) q ω ( z | s , a )
Regularization
Skill [Figure]
Posterior Training
Downstream RLs 2 ssssss d 0: T
Skill-based RL.
One class of approaches for leveraging such ofﬂine datasets that is particularly suited for learning long-horizon behaviors is skill-based RL (Hausman et al., 2018; Merel et al., 2019;
Kipf et al., 2019; Merel et al., 2020; Shankar et al., 2019; Whitney et al., 2020; Gupta et al., 2019;
Lee et al., 2020; Lynch et al., 2020;
Pertsch et al., 2020b;a).
These methods extract a set of reusable skills from task-agnostic datasets and learn new tasks by recombining them.
Yet, such approaches perform reinforcement learning over the set of extracted skills to learn the downstream task.
Although being more efﬁcient than RL over primitive actions, they still require a large number of environment interactions to learn long-horizon tasks.
In our work we combine the best of both worlds: by using large, task-agnostic datasets and a small number of task-speciﬁc demonstrations, we accelerate the learning of long-horizon tasks while reducing the number of required demonstrations.
A PPROACH
Our goal is to use skills extracted from task-agnostic prior experience data to improve the efﬁciency of demonstration-guided RL on a new task.
We aim to leverage a set of provided demonstrations by following the performed skills as opposed to the primitive actions.
Therefore, we need a model that can (1) leverage prior data to learn a rich set of skills and (2) identify the skills performed in the demonstrations in order to follow them.
In the following, we will formally deﬁne our problem setting, brieﬂy summarize relevant prior work on RL with learned skills and then describe our approach for demonstration-guided RL.
3. P RELIMINARIES
Problem Formulation We assume access to two types of datasets: a large task-agnostic ofﬂine dataset and a small task-speciﬁc demonstration dataset.
The task-agnostic dataset D = { s t , a t , ... } consists of trajectories of meaningful agent behaviors, but includes no demonstrations of the target task.
We only assume that its trajectories contain short-horizon behaviors that can be reused to solve the target task.
Such data can be collected without a particular task in mind using a mix of sources, e.g., via human teleoperation, autonomous exploration, or through policies trained for other tasks.
Since it can be used to accelerate many downstream task that utilize similar short-term behaviors we call it task-agnostic .
In contrast, the task-speciﬁc data is a much smaller set of demonstration trajectories D demo =
{ s dt , a dt , ... } that are speciﬁc to a single target task.
The downstream learning problem is formulated as a Markov decision process (MDP) deﬁned by a tuple ( S , A , T , R, ρ, γ ) of states, actions, transition probabilities, rewards, initial state distribution, and discount factor.
Our goal is to learn a policy π θ ( a | s ) with parameters θ that maximizes the discounted sum of rewards
[Equation]
, where T is the episode horizon.
Skill Prior RL Our goal is to extract skills from task-agnostic experience data and reuse them for demonstration-guided RL.
Prior work has investigated the reuse of learned skills for accelerating
RL (Pertsch et al., 2020a).
In this section, we will brieﬂy summarize their proposed approach Skill
Prior RL (SPiRL) and then describe how our approach improves upon it in the demonstration-guided
RL setting.
SPiRL deﬁnes a skill as a sequence of H consecutive actions a = { a t , . .
. , a t + H − 1 } , where the skill horizon H is a hyperparameter.
SPiRL uses the task-agnostic data to jointly learn (1) a generative model of skills p ( a | z ) , that decodes latent skill embeddings z into executable action sequences, and (2) a state-conditioned prior distribution p ( z | s ) over skill embeddings.
For learning a new downstream task, SPiRL trains a high-level skill policy π ( z | s ) whose outputs get decoded into executable actions using the pre-trained skill decoder.
Crucially, the learned skill prior is used to guide the policy during downstream RL by maximizing the following divergence-regularized RL objective:
[Equation]
Here, the KL-divergence term ensures that the policy remains close to the learned skill prior, guiding exploration during RL.
By combining this guided exploration with temporal abstraction via the learned skills, SPiRL substantially improves the efﬁciency of RL on long-horizon tasks.
3. S KILL R EPRESENTATION L EARNING
To effectively reuse skills extracted from task-agnostic data for demonstration-guided RL, we need an expressive skill representation.
The open-loop skill decoder used in SPiRL (see Section 3.1) is not conditioned on states and therefore limits the expressiveness of the learned skills.
Instead, we follow prior work on skill-based RL (Lynch et al., 2020; Ajay et al., 2020) and extend SPiRL’s low-level decoder to a full closed-loop skill policy π ( a | s, z ) that is conditioned on the current environment state.
In our experiments we found this closed-loop decoder to improve performance (see Section C for an empirical comparison).
Figure 2 (left) summarizes our skill learning model.
It consists of two parts: the skill inference network q ω ( z | s 0: H − 1 , a 0: H − 2 ) and the closed-loop skill policy π φ ( a t | s t , z ) .
Note that in contrast to SPiRL the skill inference network is state conditioned to account for the state-conditioned low-level policy.
During training we randomly sample an H -step state-action trajectory from the task-agnostic dataset.
We then pass it to the skill inference network, which predicts the low-dimensional skill embedding z , that encodes the subtask information.
This skill embedding is then input into the low-level policy π φ ( a t
| s t , z t ) for every input state.
The policy is trained to imitate the given action sequence, thereby learning to reproduce the behaviors encoded by the skill embedding z .
We optimize the latent skill representation using variational inference, which leads to our full skill learning objective:
[Equation]
We use a unit Gaussian prior p ( z ) and weight the embedding regularization term with a factor β (Higgins et al., 2017).
3. D EMONSTRATION -G UIDED RL WITH L EARNED S KILLS
To leverage the learned skills for accelerating demonstration-guided RL on a new task, we propose to use a hierarchical policy learning scheme (see Figure 2, right): we learn a high-level policy π θ
( z | s ) which outputs latent skill embeddings z and transfer the pre-trained low-level policy.
[Table]
/hu3aQ7a+mDg8d4MM/P8WHCNjvNtFVZW19Y3ipulre2d3b3y/kFLy0QxaDIppOr4VIPgETSRo4BOrICGvoC2P76a+e17UJrL6A4nMfRCOox4wBlFI/XLVU8EQkrlhRRHjIr0etr3EB4wHUAop57K3H654tScDPYycXNSITka/fKXN5AsCSFCJqjWXdeJsZdShZwJmJa8RENM2ZgOoWtoREPQvTR7Z2pXjTKwA6lMRWhn6u+JlIZaT0LfdM6u1oveTPzP6yYYXPRSHsUJQsTmi4JE2CjtWTb2gCtgKCaGUKa4udVmI6ooQ5NgyYTgLr68TFqnNdepubdnlfplHkeRHJFjckJcck7q5IY0SJMw8kieySt5s56sF+vd+pi3Fqx85pD8gfX5A06ZnYY=</latexit> Start
Goal
Our approach is guided by the task-speciﬁc skill posterior q ζ ( z | s ) within the support of the demonstrations ( green ) and by the task-agnostic skill prior p a ( z | s ) otherwise ( red ).
The agent also receives a reward bonus for reaching states within the support of the demonstrations.
Our goal is to leverage the task-speciﬁc demonstrations to guide learning of the high-level policy on the new task.
In Section 3.1, we showed how SPiRL (Pertsch et al., 2020a)
leverages a learned skill prior p a ( z | s ) to guide exploration.
However, this prior is task-agnostic, i.e., it encourages exploration of all skills that are meaningful to be explored, independent of which task the agent is trying to solve.
Even though SPiRL’s objective makes learning with a large number of skills more efﬁcient, it still encourages the policy to explore many skills that are not relevant to the downstream task.
In this work, we instead propose to leverage target task demonstrations to learn a task-speciﬁc skill distribution, which we call skill posterior q ζ ( z | s ) (in contrast to the skill prior it is conditioned on the target task, hence “posterior”).
We train this skill posterior by using the pre-trained skill inference model q ω ( z | s 0: H − 1 , a 0: H − 2 ) to extract the embeddings for the skills performed in the demonstration sequences (see Figure 2, middle):
[Equation]
where D KL denotes the Kullback-Leibler divergence.
A naive approach for leveraging the skill posterior is to simply use it to replace the skill prior in
Equation 1, i.e., to regularize the policy to stay close to the skill posterior in every state.
However, the trained skill posterior is only accurate within the support of the demonstration dataset (cid:98)D demo (cid:99) .
Since |D demo | (cid:28) |D| , this support will necessarily only be a small subset of all states (see Figure 3) and thus the skill posterior will often provide incorrect guidance in states outside the demonstrations’ support.
Instead, we propose to use a three-part objective to guide the policy during downstream learning.
Our goal in formulating this objective is to (1) follow the skill posterior within the support of the demonstrations, (2) follow the skill prior outside the demonstration support, and (3) encourage the policy to reach states within the demonstration support.
Crucial for all three components is to determine whether a given state is within the support of the demonstration data.
We propose to use a learned discriminator D ( s ) to answer this question.
It is trained to distinguish demonstration and non-demonstration states using samples from the demonstration and task-agnostic datasets, respectively.
Once trained, we use its output to weight terms in our objective that regularize the policy towards the skill prior or posterior.
Additionally, we provide a reward bonus for reaching states which the discriminator classiﬁes as being within the demonstration support.
This results in the following
J t for our full RL objective:
[Equation]
θ
[Equation] z t [Equation] s t ) , q
[Equation]
z t | s t ))
[Equation]
s
[Equation]
regularization , with ˜ r ( s t , z t ) =
(1 − κ ) · r ( s t , z t ) [Equation] κ · [Equation]
discriminator [Table] The weighting factor κ is a hyperparameter; α and α q are either constant or tuned automatically via dual gradient descent (Haarnoja et al., 2018b).
For policy optimization, we use a modiﬁed version of the SPiRL algorithm (Pertsch et al., 2020a), which itself is based on Soft Actor-Critic (Haarnoja et al., 2018a) (for the full algorithm see appendix, Section A).
The discriminator reward follows common reward formulations used in adversarial imitation learning (Finn et al., 2016; Fu et al., 2018; Zhu et al., 2018; Kostrikov et al., 2019).
1 More generally, our formulation combines IRL-like and BC-like objectives: it uses learned rewards and it directly trains the policy to match the demonstration’s skill distribution.
E XPERIMENTS 4. E XPERIMENTAL S ETUP AND C OMPARISONS
Ours
SPiRL
BC+RL
GAIL+RL
SAC Replay       leverage the provided demonstrations demonstration-guided RL approaches across all tasks. The comparison to
SPiRL shows that demonstrations can improve the learning efﬁciency even if the agent has access to large amounts of prior experience.
To evaluate whether our method can efﬁciently use the task-agnostic data, we compare it to prior demonstrationguided RL approaches on three complex, long-horizon tasks: a 2D maze navigation task, a robotic kitchen manipulation task and a robotic ofﬁce cleaning task (see Figure 4, left).
Maze Navigation.
For this task we adapt the maze navigation task from Pertsch et al.
(2020a): we further increase task complexity by adding randomness to the agent’s initial position.
To solve the task, the agent needs to navigate a maze from the randomized start position to a ﬁxed goal position using planar velocity commands.
It only receives a binary reward upon reaching the goal.
This environment is challenging for prior demonstration-guided RL approaches since demonstrations of the task span hundreds of time steps, making step-by-step imitation of primitive actions inefﬁcient.
We collect a task-agnostic ofﬂine experience dataset with 3000 sequences by using a motion planner to ﬁnd paths between randomly sampled start-goal pairs.
For the target task we sample an unseen start-goal pair and then use the same motion planner to collect a much smaller set of 2 demonstrations for reaching the ﬁxed goal position.
Although the task-agnostic dataset does not provide demonstrations for reaching the target task goal, it can still be used to extract relevant short-horizon skills like navigating hallways or passing through narrow doors.
Robot Kitchen Environment.
We use the environment of Gupta et al.
(2019) in which a 7DOF robot agent needs to perform a sequence of four subtasks, such as opening the microwave or switching on the light, in the correct order.
It receives a binary reward upon completion of each consecutive subtask.
In addition to the long task horizon, this environment requires precise control of a high-DOF manipulator, testing the scalability of demonstration-guided RL approaches.
We useteleoperated sequences performing various subtask combinations (from Gupta et al.
as our task-agnostic experience dataset D and separate a set of 20 demonstrations for one particular sequence of subtasks, which we deﬁne as our target task (see Figure 4, middle).
We further mitigate biases in the task-agnostic dataset by resampling trajectories to balance the subtask occurrence probabilities (see Section 4.4 for a detailed discussion of biases in the ofﬂine dataset).
Robot Ofﬁce Environment.
In this task, a 5 DOF robot agent needs to clean an ofﬁce environment by performing subtasks like picking and placing objects or opening and closing a drawer.
It receives binary rewards for the completion of each subtask in the correct order.
In addition to the challenges of the kitchen environment, this task tests an algorithm’s ability to learn long-horizon behaviors with freely manipulatable objects.
It also requires the algorithm to handle a greater diversity in the task-agnostic data: we collect 5300 training trajectories by placing the objects at random positions in the environment and performing random subtasks using scripted policies.
We also collect a set ofdemonstrations for the unseen target task with new object locations and an unseen subtask sequence using the same scripted policies.
We compare our approach to multiple prior demonstration-guided RL approaches that represent the different classes of existing algorithms introduced in Section 2.
In contrast to our method, these approaches are not able to leverage task-agnostic prior experience:
• BC + RL : initializes a policy with behavioral cloning of the demonstrations, then continues to apply BC loss while ﬁnetuning the policy with Soft Actor-Critic (SAC, Haarnoja et al.
(2018a)), representative of Rajeswaran et al.
; Nair et al.
• GAIL + RL
(Zhu et al., 2018): combines rewards from the environment and adversarial imitation learning (GAIL, Ho & Ermon (2016)), optimizes the policy using PPO (Schulman et al., 2017).
• Demo Replay : initializes the replay buffer of an SAC agent with the demonstration data and applies prioritized replay of the demonstration transitions during updates, representative of Vecerik et al.
We further compare our approach to RL-only methods to show the beneﬁt of using demonstration data:
• SAC (Haarnoja et al., 2018a): is the state-of-the-art model-free RL algorithm, it neither uses ofﬂine experience nor demonstrations.
• SPiRL (Pertsch et al., 2020a): extracts skills from task-agnostic experience and performs prior-guided RL on the target task (see Section 3.1).
We substitute the open-loop skill representation in Pertsch et al.
(2020a)
with our closed-loop representation (see Section 3.2), since this allows for fairer comparison and we found the performance to be generally superior (for an empirical comparison, see Section C).
For further details on the environments, data collection, and implementation details, see appendix
Section B
4. D EMONSTRATION -G UIDED RL WITH L EARNED S KILLS
Maze Navigation.
We compare the downstream task performance of the tested methods on the maze navigation task in Figure 4 (right).
Prior approaches for demonstration-guided RL struggle to learn the task since task-rewards are sparse and only two demonstrations are provided.
With such small coverage, behavioral cloning of the demonstrations’ primitive actions leads to brittle policies which are hard to ﬁnetune (for an analysis of the inﬂuence of the number of demonstrations, see Section 4.3).
The Replay agent improves over SAC without demonstrations and partly succeeds at the task, but learning is slow.
The IRL-based approach is able to follow part of the demonstrated behavior, but fails to reach the ﬁnal goal and as a result does not receive the sparse environment reward (see
Figure 8 for qualitative results).
SPiRL, in contrast, is able to leverage ofﬂine, task-agnostic experience to learn to solve the task, but requires a substantial amount of environment interactions: the task-agnostic skill prior encourages the policy to try many skills before converging to the ones that solve the downstream task.
In contrast, our approach leverages the task-speciﬁc skill posterior to quickly explore the relevant skills, leading to signiﬁcant efﬁciency gains.
We further analyze our approach in Figure 5: we visualize the output of the learned discriminator
D ( s ) , as well as divergences between the policy and skill prior and posterior.
The discriminator accurately estimates the demonstration support, providing a good weighting for prior and posterior regularization, as well as a dense reward bonus.
The policy successfully minimizes divergence to the task-speciﬁc skill posterior within the demonstration support and follows the skill prior otherwise.
Robotic Manipulation.
We show the performance comparison on the robotic manipulation tasks in Figure 4 (right) 2 .
Both tasks are more challenging since they require precise control of a high-DOF manipulator.
We ﬁnd that approaches for demonstration-guided RL that do not leverage task-agnostic experience struggle to learn either of the tasks since following the demonstrations step-by-step is inefﬁcient and prone to accumulating errors.
SPiRL, in contrast, is able to learn meaningful skills from the ofﬂine datasets, but struggles to explore the task-relevant skills and therefore learns slowly.
Our approach uses the demonstrations to learn to chain the extracted skills and solve the tasks.
4. A BLATION S TUDIES
Number of Demonstrations.
In Figure(left) we investigate the robustness of demonstration-guided RL approaches to a variety of demonstration set sizes on the maze navigation task.
We compare our approach to BC+RL, since we found it to achieve the most consistent results across different numbers of demonstrations.
While both approaches beneﬁt from larger demonstration sets, we can observe that the advantage from leveraging prior experience is particularly large in cases with few provided demonstrations.
These results are intuitive, since BC+RL needs to use the demonstrations as guidance for learning each low-level action while our approach merely needs to learn to recombine skills it has already mastered using the ofﬂine data.
Thus, it requires less dense supervision and can cope with fewer demonstrations.
Our approach’s ability to use task-agnostic data to reduce the number of required demonstrations and environment interactions improves two of the most costly aspects of learning tasks in the real world.
Thus, the ability to leverage task-agnostic data is an important step for scaling demonstration-guided RL to diverse real-world tasks.
Objective Ablations.
We ablate different parts of our downstream RL objective on the kitchen task in Figure 6 (right).
We ﬁnd that removing the discriminator reward bonus (" no-GAIL ") leads to slower convergence since the agent lacks a dense reward signal.
Naively replacing the skill prior in Equation 1 with the learned skill posterior (" post-only ") fails to learn the task.
Without the support-based weighting the policy is constrained to follow the skill posterior in parts of the state space which the posterior was not trained on.
Finally, removing the learned skill posterior and optimizing a discriminator bonus augmented reward using SPiRL (" no-post ") fails to learn the task.
We also ablate the inﬂuence of environment rewards to test the efﬁcacy of skill-based learning in the pure imitation setting in appendix, Section E.
4. D ATA A LIGNMENT A NALYSIS
We have shown that our approach is able to combine task-agnostic prior experience and task-speciﬁc demonstrations to accelerate RL.
Since the task-agnostic data can be reused to learn many tasks, the marginal cost of learning a new task is dominated by the cost of collecting the task-speciﬁc demonstrations.
A key question is therefore: in what scenarios does the combination of task-agnostic experience and demonstrations improve efﬁciency and when can we save costs and solely rely on the task-agnostic dataset?
We hypothesize that the distribution of observed behaviors in the task-agnostic dataset is key to answering this question: if it is not well aligned with the target behaviors, learning is inefﬁcient.
Thus, this alignment determines the efﬁciency gain our method can achieve through the addition of demonstrations.
In our previous robotic kitchen experiments we balanced the distribution of behaviors in the task-agnostic dataset to mitigate such biases in the selection of the downstream task.
To empirically analyze the effects of good and bad data-task alignment, we now conduct additional experiments with the original, biased data distribution of Gupta et al.
For a detailed analysis of the data biases and the chosen downstream tasks, see Section F.
We compare the performance of our method to SPiRL, which solely relies on task-agnostic data.
OursSPiRL OursSPiRL
The beneﬁt of using demonstrations in addition to prior experience diminishes if the prior experience is closely aligned to the target task ( left ).
Efﬁciency gains are high when the learned prior is mis-aligned with the target task, leading to inefﬁcient exploration ( right ).
In the well-aligned case (Figure 7, left), we ﬁnd that both SPiRL and our approach are able to learn the task efﬁciently.
Since the skill prior encourages effective exploration on the downstream task, the beneﬁt of the additional demonstrations leveraged in our method is marginal.
In contrast, if task-agnostic data and downstream task are mis-aligned (Figure 7, right), we ﬁnd that SPiRL struggles to learn the task due to a conﬂict between the two objectives in Equation 1: maximizing task reward and minimizing divergence from the skill prior.
Our approach is able to learn the task more reliably, since it encourages the policy to reach demonstration-like states and then follow the skill posterior, which by design is well-aligned with the downstream task.
In summary, our analysis ﬁnds that approaches which leverage both task-agnostic data and demonstrations, improve over methods that use either of the data sources alone across all tested tasks.
We ﬁnd that combining the data sources is particularly beneﬁcial in two cases:
• Diverse Task-Agnostic Data.
Demonstrations can focus exploration on the task-relevant skills if the task-agnostic skill prior encourages exploration of a large, potentially irrelevant set of skills (see Section 4.2).
• Mis-Aligned Task-Agnostic Data.
Demonstrations can compensate mis-alignment between task-agnostic data and target task by guiding exploration with the skill posterior instead of the mis-aligned prior.
Both scenarios are important since it is challenging to control the quality of the prior experience dataset, particularly if collected at scale from multiple sources.
Our experiments show that the combination of prior experience and demonstrations can make the learner more robust to ﬂuctuations in the characteristics of the prior experience dataset.
C ONCLUSION
We have proposed a novel approach for demonstration-guided RL that is able to leverage task-agnostic experience datasets for accelerated learning of unseen tasks.
Our agent extracts a set of reusable skills from the task-agnostic dataset.
It then identiﬁes the skills performed in the task-speciﬁc demonstrations and uses them to guide downstream RL.
In three challenging environments we ﬁnd that our approach learns unseen tasks more efﬁciently than both, prior demonstration-guided RL approaches that are not able to leverage task-agnostic experience, as well as skill-based RL methods that cannot effectively incorporate demonstrations.
A F ULL A LGORITHM
We detail our full algorithm for demonstration-guided RL with learned skills in Algorithm 1.
It is based on the SPiRL algorithm for RL with learned skills Pertsch et al.
(2020a) which in turn builds on
Soft-Actor Critic Haarnoja et al.
(2018a), an off-policy model-free RL algorithm.
We mark changes of our algorithm with respect to SPiRL and SAC in red in Algorithm 1.
The hyperparameters α and α q can either be constant, or they can be automatically tuned using dual gradient descent Haarnoja et al.
(2018b)
; Pertsch et al. (2020a).
In the latter case, we need to deﬁne a set of target divergences δ, δ q .
The parameters α and α q are then optimized to ensure that the expected divergence between policy and skill prior and posterior distributions is equal to the chosen target divergence (see Algorithm 1).
B I MPLEMENTATION AND E XPERIMENTAL D ETAILS B. I MPLEMENTATION D ETAILS : P RE -T RAINING
We introduce our objective for learning the skill inference network q ω ( z | s, a ) and low-level skill policy π φ ( a t
| s t , z ) in Section 3.2.
In practice, we instantiate all model components with deep neural networks Q ω , Π φ respectively, and optimize the full model using back-propagation.
We also jointly train our skill prior network P a .
We follow the common assumption of Gaussian, unit-variance output distributions for low-level policy actions, leading to the following network loss:
[Equation]
Here (cid:98)·(cid:99) indicates that we stop gradient ﬂow from the prior training objective into the skill inference network for improved training stability.
After training the skill inference network with above objective, we train the skill posterior network Q ζ by minimizing KL divergence to the skill inference network’s output on trajectories sampled from the demonstration data.
We minimize the following objective:
[Equation]
[Table]
We use a 1-layer LSTM with 128 hidden units for the inference network and 3-layer MLPs withhidden units in each layer for the low-level policy.
We encode skills of horizon 10 into 10 -dimensional skill representations z .
Skill prior and posterior networks are implemented as 5-layer MLPs with 128 hidden units per layer.
They both parametrize mean and standard deviation of Gaussian output distributions.
All networks use batch normalization after every layer and leaky ReLU activation functions.
We tune the regularization weight β to be 1e − 2 for the maze and 5e − 4 for kitchen and ofﬁce environment.
For the demonstration discriminator D ( s ) we use a 3-layer MLP with only 32 hidden units per layer to avoid overﬁtting.
It uses a sigmoid activation function on the ﬁnal layer and leaky ReLU activations otherwise.
We train the discriminator with binary cross-entropy loss on samples from task-agnostic and demonstration datasets:
[Equation]
We optimize all networks using the RAdam optimizer Liu et al.
(2020) with parameters β 1 = 0 .
9 and β 2 = 0 .
999 , batch size 128 and learning rate 1e − 3 .
On a single NVIDIA Titan X GPU we can train the skill representation and skill prior in approximately 5 hours, the skill posterior in approximatelyhours and the discriminator in approximately 3 hours.
B. I MPLEMENTATION D ETAILS : D OWNSTREAM RL
The architecture of the policy mirrors the one of the skill prior and posterior networks.
The critic is a simple 2-layer MLP with 256 hidden units per layer.
The policy outputs the parameters of a Gaussian action distribution while the critic outputs a single Q -value estimate.
We initialize the policy with the weights of the skill posterior network.
We use the hyperparameters of the standard SAC implementation Haarnoja et al.
(2018a) with batch size 256, replay buffer capacity of 1e6 and discount factor γ = 0 .
We collect 5000 warmup rollout steps to initialize the replay buffer before training.
We use the Adam optimizer Kingma & Ba (2015) with β 1 = 0 .
9 , β 2 = 0 .
999 and learning rate 3e − 4 for updating policy, critic and temperatures α and α q .
Analogous to SAC, we train two separate critic networks and compute the Q -value as the minimum over both estimates to stabilize training.
The corresponding target networks get updated at a rate of τ = 5e − 3 .
The policy’s actions are limited in the range [ − 2 , 2] by a tanh "squashing function" (see Haarnoja et al.
(2018a), appendix C).
We use automatic tuning of α and α q in the maze and the ofﬁce environment and set both target divergences to 1 and 7 respectively.
In the kitchen environment we obtained best results by using constant values of α = α q = 5e − 2 .
In all experiments we set κ = 0 .
For all RL results we average the results of three independently seeded runs and display mean and standard deviation across seeds.
B. I MPLEMENTATION D ETAILS : C OMPARISONS
BC+RL.
This comparison is representative of demonstration-guided RL approaches that use BC objectives to initialize and regularize the policy during RL Rajeswaran et al.
; Nair et al.
We pre-train a BC policy on the demonstration dataset and use it to initialize the RL policy.
We use
SAC to train the policy on the target task.
Similar to Nair et al.
we augment the policy update with a regularization term that minimizes the L2 loss between the predicted mean of the policy’s output distribution and the output of the BC pre-trained policy 3 .
Demo Replay.
This comparison is representative of approaches that initialize the replay buffer of an off-policy RL agent with demonstration transitions Vecerik et al.
; Hester et al.
In practice we use SAC and initialize a second replay buffer with the demonstration transitions.
Since the demonstrations do not come with reward, we heuristically set the reward of each demonstration trajectory to be a high for the maze 4 for the environments) on the and zero everywhere
, we sample half of training mini-batch from the normal buffer demonstration buffer.
of SAC remain unchanged.
B. E NVIRONMENT D ETAILS
Maze Navigation.
We adapt the maze navigation task from Pertsch et al.
(2020a) which extends the maze navigation tasks from the D4RL benchmark Fu et al.
The starting position is sampled uniformly from a start region and the agent receives a one-time sparse reward of 100 when reaching the ﬁxed goal position, which also ends the episode.
The 4D observation space contains 2D position and velocity of the agent.
The agent is controlled via 2D velocity commands.
[Figure]
Robot Kitchen Environment.
We use the kitchen environment from Gupta et al.
For solving the target task, the agent needs to execute a ﬁxed sequence of four subtasks by controlling an Emika Franka Panda 7-DOF robot via joint velocity and continuous gripper actuation commands.
The 30-dimensional state space contains the robot’s joint angles as well as objectspeciﬁc features that characterize the position of each of the manipulatable objects.
We mitigate biases in the distribution over subtasks transitions by re-sampling sequences of the provided dataset (see Section F for details).
Further, we collect a version of the demonstration dataset with increased support by initializing the environment state to states randomly sampled from the demonstrations and rolling out a random policy for 10 steps.
We use this data augmentation technique for increasing the support while pretraining the demonstration discriminator D ( s ) 4 .
Robot Ofﬁce Environment.
We create a novel ofﬁce cleanup task in which a 5-DOF WidowX robot needs to place a number of objects into designated containers, requiring the execution of a sequence of pick, place and drawer open and close subtasks (see Figure 9).
The agent controls position and orientation of the end-effector and a continuous gripper actuation, resulting in a 7 -dimensional action space.
For simulating the environment we build on the Roboverse framework Singh et al.
During collection of the task-agnostic data we randomly sample a subset of three of the four objects as well as a random order of target containers and use scripted policies to execute the task (see Figure 14).
We only save successful executions.
For the target task we ﬁx object positions and require the agent to place three objects in ﬁxed target containers.
The 76dimensional state space contains the agent’s end-effector position and orientation as well as position and orientation of all objects and containers.
C S KILL R EPRESENTATION C OMPARISON
Ours Pertsch et al., In Section 3.2 we described our skill representation based on a closed-loop low-level policy as a more powerful alternative to the action trajectory-based representation of Pertsch et al.
(2020a).
To compare the performance of the two representations we perform rollouts with the learned skill prior: we sample a skill from the prior and rollout the low-level policy for H steps.
We repeat this until the episode terminates and visualize the results for multiple episodes in maze and kitchen environment in Figure 10.
In Figure 10 (top) we see that both representations lead to effective exploration in the maze environment.
Since the 2D maze navigation task does not require control in high-dimensional action spaces, both skill representations are sufﬁcient to accurately reproduce behaviors observed in the task-agnostic training data.
In contrast, the results on the kitchen environment (Figure 10, bottom) show that the closedloop skill representation is able to more accurately control the high-DOF robotic manipulator and reliably solve multiple subtasks per rollout episode.
We hypothesize that the closed-loop skill policy is able to learn more robust skills from the task-agnostic training data, particularly in high-dimensional control problems.
D D EMONSTRATION -G UIDED RL C OMPARISONS WITH T ASK -A GNOSTIC E XPERIENCE
In Section 4.2 we compared our approach to prior demonstration-guided RL approaches which are not designed to leverage task-agnostic datasets.
We applied these prior works in the setting they were designed for: using only task-speciﬁc demonstrations of the target task.
Here, we conduct experiments in which we run these prior works using the combined task-agnostic and task-speciﬁc datasets to give them access to the same data that our approach used.
From the results in Figure 11 we can see that none of the prior works is able to effectively leverage the additional task-agnostic data.
In many cases the performance of the approaches is worse than when only using task-speciﬁc data (see Figure 4).
Since prior approaches are not designed to leverage task-agnostic data, applying them in the combined-data setting can hurt learning on the target task.
In contrast, our approach can effectively leverage the task-agnostic data for accelerating demonstration-guided RL.
E S KILL -B ASED I MITATION L EARNING
We ablate the inﬂuence of the environment reward feedback on the performance of our approach by setting the reward weight γ = 0 , thus relying solely on the learned discriminator reward.
Our goal is to test whether our approach is able to leverage task-agnostic experience to improve the performance of pure imitation learning , i.e., learning to follow demonstrations without environment reward feedback.
We compare our method to common approaches for imitation learning: behavioral cloning (BC,
Pomerleau (1989)) and generative adversarial imitation learning (GAIL, Ho & Ermon (2016)).
We also experiment with a version of our skillbased imitation learning approach that performs online ﬁnetuning of the pre-trained discriminator D ( s ) using data collected during training of the imitation policy.
We summarize the results of the imitation learning experiments in Figure 12.
Learning purely by imitating the demonstrations, without additional reward feedback, is generally slower than demonstration-guided RL.
Yet, we ﬁnd that our approach is able to leverage task-agnostic data to effectively imitate complex, long-horizon behaviors while prior imitation learning approaches struggle.
Further, online ﬁnetuning of the learned discriminator slightly improves imitation learning performance.
F K ITCHEN D ATA A NALYSIS
[Figure]
imitation learning methods approach lever age prior experience to enable the imitation of complex long horizon behaviors the pre-trained discriminator D ( s ) further improves performance.
For the kitchen manipulation experiments we use the dataset provided by Gupta et al.
as task-agnostic pre-training data.
It consists of 603 teleoperated sequences, each of which shows the completion of four consecutive subtasks.
In total there are seven possible subtasks: opening the microwave, moving the kettle, turning on top and bottom burner, ﬂipping the light switch and opening a slide and a hinge cabinet.
In Figure 13 we analyze the transition probabilities between subtasks in the task-agnostic dataset.
We can see that these transition probabilities are not uniformly distributed, but instead certain transitions are more likely than others, e.g., it is much more likely to sample a training trajectory in which the agent ﬁrst opens the microwave than one in which it starts by turning on the bottom burner.
In Section 4.4 we test the effect this bias in transition probabilities has on the learning of target tasks.
Concretely, we investigate two cases: good alignment between task-agnostic data and target task and
[Figure] [Figure]
Demonstration Data
mis-alignment between the two.
In the former case we choose the target task Microwave - KettleLight Switch - Hinge Cabinet , since the required subtask transitions are likely under the training data distribution.
For the mis-aligned case we choose Microwave - Light Switch - Slide Cabinet - Hinge
Cabinet as target task, since particularly the transition from opening the microwave to ﬂipping the light switch is very unlikely to be observed in the training data.
In our experiments in Section 4.2 we minimize bias introduced through the target task choice by balancing the subtask transition probabilities in the task-agnostic dataset.
We achieve this by resampling training trajectories such that it is equally likely to sample any of the ﬁrst two subtask transitions, i.e., after resampling it is equally likely to sample a training trajectory that starts with
Bottom Burner - Top Burner as it is to sample a trajectory starting with Microwave - Kettle .
As a result, the task-agnostic data has equal alignment to a large set of possible downstream tasks.