Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics
Abstract:
Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning.
Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent.
However, this skill space is expansive, and not all skills are relevant for a given robot state, making exploration difﬁcult.
Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space.
We ﬁrstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only sampling skills relevant to a given state based on prior experience.
Next, we propose a low-level residual policy for ﬁne-grained skill adaptation enabling downstream RL agents to adapt to unseen task variations.
Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while signiﬁcantly accelerating exploration, outperforming prior works.
Code and videos are available on our project website: https://krishanrana.github.io/reskill .
Introduction
Humans are remarkably efﬁcient at learning new behaviours.
We can attribute this to their ability to constantly draw on relevant prior experiences and adapt this knowledge to facilitate learning.
Building on this observation, recent works have proposed various methods to incorporate the use of prior experience in deep reinforcement learning (RL) to accelerate robot learning.
One approach that has gained attention recently is the use of skills which are short sequences of single-step actions representing useful and task-agnostic behaviours extracted from datasets of expert demonstrations [1, 2, 3].
In the manipulation domain, for example, such skills could include move-left , grasp-object and lift-object .
These skills are typically embedded into a latent space, forming the action space for a high-level RL policy.
While yielding signiﬁcant improvements over learning from scratch , there are still outstanding challenges for training RL agents using this latent skill space .
Firstly, na¨ıve exploration by sampling randomly over all skills can be extremely inefﬁcient [2, 3, 4].
Typical skill spaces encode an expansive set of generic skills, with only a small fraction of skills relevant for execution in a given robot state.
Furthermore, the set of skills relevant to a given state do not typically cluster in the same neighbourhood in the skill space, i.e. the conditional density is multi-modal , making targeted sampling challenging.
We address this issue of inefﬁcient exploration by learning a state-conditioned skill prior over the skill space, which directly biases the agent’s exploration towards sampling only those skills that are deemed relevant in its current state.
Unlike previous works [3], our skill-prior captures the complex multi-modality of the skill space allowing us to sample directly from this density during exploration.
[Figure]
Secondly current approaches to skill-based RL assume that the skills are optimal and that downstream tasks are drawn from the same distribution used to create the skill space.
Consequently, skill-based learning methods have limited generality and adaptability to task variations.
For example if the available skills extracted solely
We summarise the main contributions of our approach denoted as Residual Skill Policies (ReSkill) as follows: (1) we propose a novel state-conditioned skill prior that enables direct sampling of relevant skills for guided exploration, (2) we introduce a low-level residual policy that can adapt the skills to variations in the downstream task, and (3) we demonstrate how our two contributions both accelerate skill-based learning and enable the agent to attain higher ﬁnal performance in novel environments across four manipulation tasks, outperforming all baseline methods.
Related Work
Skill-Based RL This has emerged as a promising approach to effectively leverage prior knowledge within the RL framework by embedding reusable skills into a continuous skill space via stochastic latent variable models [1, 2, 3, 5, 6].
Skill-based learning provides temporal abstraction that allows for effective exploration of the search space, with the ability to draw on previously used behaviours to accelerate learning.
However as described previously, the skill space can be difﬁcult to explore.
Pertsch et al.
[3] proposed the use of a prior over skills to help better guide exploration.
They learn a unimodal Gaussian prior that approximates the density of relevant skills for a given state and regularise the downstream RL agent towards the prior during training to indirectly bias exploration.
Singh et al.
[4] proposed a more expressive strategy for learning these priors via the use of stateconditioned generative models.
While this was conducted at the single-step action level, they show that they can directly bias exploration by sampling relevant actions from this density.
In this work, we extend this idea to skill-based action spaces.
Combining RL with Classical Control Our work is closely related to methods that seek to combine the strengths of classical control and RL for accelerated learning.
In contrast to expert datasets or human demonstrators, classical control systems are cheap and readily available for most robotics tasks.
Johannink et al.
[7] proposed the residual reinforcement learning framework in which an RL agent learns an additive residual policy that modiﬁes the behaviours of a predeﬁned handcrafted controller to simplify exploration.
Other works in this area have shown that combining the two modes of control allows for sample efﬁcient initialisation [8, 9, 10], temporal abstraction for accelerated learning [11], and safe exploration [12].
While strongly motivating the re-use of existing handcrafted controllers, these methods are restricted to solving only one particular task for which the controller was designed, requiring a new controller for each new task.
This limits the generality of these approaches to novel tasks.
In this work, we decompose these controllers into task-agnostic skills and explore how they can be repurposed for learning a wide range of tasks.
Hierarchical RL (HRL) These methods enable autonomous decomposition of challenging longhorizon decision-making tasks for efﬁcient learning.
Typical HRL approaches utilise a high-level policy that selects from a set of low-level temporally extended actions or options [13] that provide broader coverage of the exploration space when executed [14].
Prior work in hierarchical reinforcement learning can be divided into two main categories: learning low-level and high-level policies through active interaction with an environment [15, 16, 17, 18, 19], and learning options from demonstrations before re-composing them to perform long-horizon tasks through RL [20, 21, 22].
Our work shares similarities with both strategies.
We follow the latter approach by pre-training a low-level skill module from demonstration data on which a high-level policy operates, while additionally learning a low-level residual policy online for ﬁne-grained skill adaptation.
Problem Formulation
We focus on skill-based RL which leverages skills as the actions for a high-level policy π HL , where a skill a is deﬁned as a sequence of atomic actions { a t , ..., a t + H − 1 } over a ﬁxed horizon H .
Skills are typically embedded into a continuous latent space Z from which π HL can sample during exploration [1, 2, 3].
Latent skill spaces expressively capture vast amounts of task-agnostic prior experience for re-use in RL.
However, they pose several challenges that can negatively impact downstream learning.
Firstly, current methods rely on expert and exhaustive demonstration datasets that capture skills for a wide range of potential task variations.
Such datasets can be arduous and expensive to obtain.
Secondly, as the skill space abstracts away the RL agent’s atomic action space, the generality of skill-based RL is greatly limited to tasks that closely match the distribution of those captured by the dataset.
This poses a trade-off between generality and sample efﬁciency in downstream learning [23].
Furthermore, latent skill spaces are expansive and can be arbitrarily structured, making random exploration difﬁcult.
Learning structure within this space can better inform exploration to accelerate downstream learning.
The downstream task is formulated as a Markov Decision Process (MDP) deﬁned by a tuple ( S , A , T , R , γ ) consisting of states, actions, transition probabilities, rewards and a discount factor.
We learn the high-level policy π HL ( z | s ) to select the appropriate latent skills z in a given state s to maximise the discounted sum of H -step rewards J = E π HL [ {+(cid:80) T − 1 t =
+}
γ t ˜ r ] , where T is the episode horizon, ˜ r is the H -step reward deﬁned as ˜ r
= (cid:80)
Ht =1 r t and r t is the single-step reward.
Approach
We aim to re-purpose existing classical controllers as skills to facilitate RL in solving new tasks.
We decompose demonstration trajectories produced by these controllers into task-agnostic skills and embed them into a continuous space
Z using stochastic latent variable models.
To facilitate exploration for downstream RL in this skill space, we propose a state-conditioned skill prior p ( z | s 0 ) over skills that directly biases the agent towards only sampling skills relevant to a given state based on prior experience.
Finally, to enable truly general-purpose learning with the skill space, we propose a low-level residual policy π δ ( δa |· ) that can adapt the embedded skills to task variations not captured by the demonstration data.
This gives the RL agent access to its single-step action space and relaxes the need for both expert and exhaustive datasets.
[Figure]
S S
S
S H ….
S… n n’ n’’
SkillPrior f(z|n,s 1 )
[Figure]
SkillPrior
[Figure] Fixed [Figure] z a a a a H a a a a H z z’ z’’ z’’’ =
g
Our approach can be decomposed into three sub-problems: (1) skill extraction from existing controllers, (2) learning a skill embedding and skill prior, and (3) training a hierarchical RL policy within this skill space with a low-level, residual skill adaptation policy.
We discuss each of these steps in more detail below.
4.1 Data Collection
We re-purpose existing handcrafted controllers as sources of demonstration data for basic manipulation tasks involving pushing and grasping an object on an empty table.
While these controllers are relatively simple, the trajectories produced possess a vast range of skills that can be recomposed to solve more complex tasks.
We note here that these trajectories do not need to be optimal given our adaptable downstream RL formulation which we will describe in more detail in the following sections.
The trajectories consist of state-action pairs and we perform unsupervised skill extraction by randomly slicing a H length segment from them.
We utilise both the extracted sequence of actions a as well as the corresponding states s to learn the state-conditioned skill space described in the next section.
The state vector comprises of joint angles, joint velocities, gripper position and object positions in the scene, while the actions are continuous 4D vectors representing end-effector and gripper velocities in Cartesian space.
More details about data collection and the speciﬁc handcrafted controllers used can be found in Appendix ?? .
4.2 Learning a State-Conditioned Skill Space for RL
Learning a state-conditioned skill space consists of two key steps: (1) embedding the extracted skills into a latent space; (2) learning a state-conditioned prior over the skills which we can sample from during exploration.
Figure 2 provides a summary of our approach and we describe each component in detail below.
4.2.1 Embedding the skills
We follow the approach proposed in prior works [1, 24], where skills a are embedded into a latent space using a variational autoencoder (VAE) [25].
The VAE comprises of a probabilistic encoder q φ and decoder p θ with network weights φ and θ respectively, which maps a skill action sequence to a latent embedding space and vice versa.
Our encoder q φ ( z | s , a ) jointly processes the full stateaction sequence, while our decoder p θ ( a t | z, s t ) reconstructs individual atomic actions conditioned on current state s t and skill embedding z .
The loss function for our VAE for a single training example is given by
[Equation]
where p ( z ) ∼ N (0 , I ) and β ∈ [0 , 1] is the weighting for the regularisation term [25].
The closedloop nature of the decoder considers the current robot state s t as well as sampled skill z when recovering actions, which was found to signiﬁcantly improve performance in downstream learning [26], particularly in dynamic environments.
4.2.2 Learning the state-conditioned skill prior
[Figure]
The skill prior samples from regions containing embedded and relevant skills, reducing the chances of sampling irrelevant regions (
) not containing any meaningful skills.
The nature of the VAE reconstruction loss described in Equation (1) allows us to sample skills from the latent space by using the prior p ( z ) ∼ N (0 , I ) .
However, sampling directly from the prior can be inefﬁcient, since the samples will range across the full, expansive set of available skills as well as regions that may not constitute any embeddings as shown in Figure 3.
In reality, for a particular robot state, only a small subset of these skills are relevant for exploration.
To address this inefﬁciency, we propose to learn a state-conditioned skill prior over the latent embedding space, which allows us to more frequently sample the relevant skills likely to be useful in a given state, thus accelerating exploration.
Mathematically, we are looking to learn a conditional probability density over the latent skill space p
( z | s 0 ) that we can draw samples from when training π HL .
This conditional density is typically highly multi-modal, i.e. for a given robot state, many skills can be relevant that may be far apart in the topology of the latent skill space.
Prior works [3] propose a simple unimodal density (e.g. Gaussian) which does not sufﬁciently capture this multi-modality and hence is unsuitable to directly sample skills from.
One of our contributions is around gracefully handling this multi-modality by using the real-valued non-volume preserving transformations (real NVP) method by Dinh et al.
[27] based on normalising ﬂows [28].
Conditional real NVP [29] learns a mapping f : Z×S → G which we can use to generate samples from p ( z | s ) .
Spaces Z and G are identical, i.e. Z ∼ =
G ∼ = R d and furthermore f is bijective between Z and G for a ﬁxed state s .
Sampling is achieved as follows:
ﬁrst sample g ∼ p G ( g ) from a simple distribution (e.g. p G ( g ) ∼ N (0 , I ) ) and then transform g into skill space
Z using f − 1 , i.e. f − 1 ( g, s ) ∼ p ( z | s ) .
Since f fully deﬁnes p ( z | s ) , we henceforth refer to f as the skill prior.
We refer the reader to Dinh et al.
[27] for more information about real NVP.
We train our stateconditioned skill prior jointly with our VAE embedding module.
Figure 3 illustrates the multimodality present in the latent skill space and how our skill prior captures this.
Implementation details are provided in Appendix ?? .
4.3 Reinforcement Learning in a State-Conditioned Skill Space
Once trained, the decoder and skill prior weights are frozen and incorporated within the RL framework.
Our high-level RL policy denoted π HL ( g | s ) is a neural network that maps a state to a vector g ∈ G in the domain of the skill prior transformation f − 1 deﬁned in Section 4.2.2.
The skill prior f − 1 is then used to transform g to latent skill embedding z .
Our use of normalising ﬂows for our skills prior is motivated by [4], who hypothesised that the bijective nature of f allows the RL agent
Given: high-level policy π HL , residual policy π δ , skill decoder p , skill prior f
Inputs: state s t 1: g ∼ π HL ( g | s t ) (cid:46) sample a vector g from the high-level policy 2: z ← f − 1 ( g, s t ) (cid:46) map g to a state-relevant skill using the skill prior 3: for skill horizon H do 4: a (cid:48)
← p ( a (cid:48) | s t , z ) (cid:46) decode the skill to a single-step action using the skill decoder 5: δa ∼ π δ ( δa | s t , z, a (cid:48) )
(cid:46) sample a corrective action from the residual policy 6: a t = a (cid:48)
+ δa (cid:46) sum the decoded and residual action 7: return action a t (cid:46) execute the combined action in the environment and continue to retain full control over the broader skill space Z : for any given z , there exists a g that generates z = f − 1 ( g, s ) in the original MDP.
This means that all skills from Z can be sampled by the skill prior with non-zero probability.
Once a latent skill z is attained by the high-level agent, the closed-loop decoder can reconstruct each action a (cid:48) sequentially conditioned on the current state for the skill horizon H .
To increase the diversity of downstream tasks that can be learned using the ﬁxed skill space, we introduce a lowlevel residual policy π δ that can adapt the decoded skills at the executable action level, providing the hierarchical agent with full control over the MDP.
We condition this residual policy on the current state s , selected skill z , and the action proposal generated by the skill decoder a (cid:48) .
The produced action δa ∼ π δ ( δa | s, z, a (cid:48) ) is added to the decoded action a (cid:48) before being executed on the robot.
This low-level system is operated at a frequency of H Hz based on the skill horizon, while the outer loop operates at 1 Hz.
Figure 1 summarises our Residual Skill Policy (ReSkill) architecture and pseudo-code for action selection is provided in Algorithm 1.
Given the non-stationarity imposed by the low-level residual policy being trained jointly with the high-level policy, we utilise an on-policy
RL algorithm for training the hierarchical agent.
Implementation details are provided in Appendix ?? .
Experiments
We evaluate our approach in a robotic manipulation domain simulated in MuJoCO [30], involving a 7-DoF Fetch robotic arm interacting with objects placed in front of it as shown in Figure 4.
Evaluation is performed over four downstream tasks, each one not solvable purely with the prior controllers used for data collection as described in Appendix ?? .
Each task is adapted from the residual reinforcement learning set of environments proposed by Silver et al.
We describe each of the downstream tasks in detail in Appendix ?? .
We compare our approach against several state-of-the-art RL methods; these methods either operate directly on the single-step action space or train agents using a skills-based approach.
The speciﬁc methods we compared against are as follows:
Scripted Controller: The average performance of the handcrafted controllers used for data collection across the 4 environments.
Behavioural Cloning (BC) + Fine-Tuning:
Trains an agent by ﬁrst using supervised learning on state-action pairs from the training set, followed by RL ﬁne-tuning.
SAC and PPO:
Trains an agent using Soft Actor-Critic [32] and Proximal Policy Optimisation [33] respectively with random weight initialisation.
HAC:
Trains an agent using Hierarchical Actor-Critic [19] using a hierarchy of 2 for all tasks.
PaRRot: Leverages ﬂow-based density estimation to map the agent’s action space to single-step actions that are likely to be useful in a given state [4].
SPiRL: A hierarchical skill-based approach utilising KL regularisation of the RL policy towards a Gaussian skill prior [3] 7.
ReSkill (No Skill Prior): Ablation study where the high-level policy samples skills directly using π HL ( z | s
[Figure] [Figure]
Results
The results across each task are summarised in Figure 5.
Our approach, ReSkill, outperforms all the baselines, attaining the highest performing policy given its ability to adapt to task variations while additionally converging with superior sample efﬁciency given the guidance of the state-conditioned skill prior.
Note, the baseline scripted controllers are sub-optimal across all four tasks as indicated by the dashed black line.
SAC, PPO and HAC fail to learn altogether, given the inability of random exploration alone to yield any meaningful behaviours in sparse reward settings.
Results are incrementally better using BC initialisation and ﬁne-tuning; however, the agent still attains suboptimal performance with high variance across the trials.
A critical difference between PaRRot and
ReSkill is the use of skills in ReSkill instead of single-step actions.
The temporal abstraction provided by skills allows ReSkill to signiﬁcantly outperform PaRRot in sample efﬁciency across all tasks.
Further, we note that PaRRot struggles to learn a high-performing policy in the Slippery-Push and Complex-Hook domain which we could attribute to its inability to deal with the variations in the downstream task given the constrained action space.
SPiRL accelerates learning, yielding better sample efﬁciency than all atomic-action-based methods on Table-Cleanup and Complex-Hook .
However, it still fails to attain high ﬁnal policy performance as it only composes high-level skills without the ability to adapt them to task variations, as shown by ReSkill.
We additionally provide an ablation study in Figure 6 to better understand the impact of the skill prior and residual agent on the overall skill-based learning architecture.
ReSkill (No Skill Prior) demonstrates the importance of our sampling-based prior in accelerating learning and enabling our agent to learn altogether in the more complex stacking task.
The ReSkill (No Residual) plots demonstrate the importance of the residual in enabling the agent to attain high ﬁnal performance by re-introducing access to the single-step action space for ﬁne-grained control.
This allows the agent to appropriately balance the trade-off between generality and sample efﬁciency when leveraging skills in RL.
[Figure]
Furthermore, comparing ReSkill (No Residual) to SPiRL, we note that our approach attains faster convergence to the ﬁnal policy.
We can attribute this to the nature of our skill prior which enables the agent to directly sample and execute state-relevant skills from the early stages of training.
In contrast, SPiRL relies on an indirect bias to the agent’s exploratory actions via KL-regularisation towards a unimodal skill prior.
This requires an initial ”burn-in” phase before the policy is appropriately regularised towards the skill prior in order for the agent to start sampling relevant skills.
We provide a more detailed ablation study of the impact of our proposed skill-prior in Appendix ?? .
6.1 Limitations
While we demonstrate signiﬁcantly faster learning and better asymptotic performance than all the baseline methods, ReSkill still requires thousands of environment samples before converging to a high success rate.
This limited our ability to demonstrate our method on a physical robot.
An interesting future direction to explore would be to integrate our approach with an off-policy RL algorithm such as SAC in order to make better use of exploratory experience for improved sample efﬁciency.
This, however, would require careful consideration of the non-stationarity imposed by the low-level residual component.
Secondly, while we note that the skill prior used in this work is bijective and should theoretically allow the agent to access the entire range of skills in the larger skill space, the likelihood of selecting useful skills not seen in the training dataset becomes very low.
This would particularly be an issue when there is a signiﬁcant mismatch between the training dataset and the downstream task.
We note here that the residual policy could play a major in addressing this, however, we leave a thorough evaluation of this to future work.
Finally, learning the state-conditioned skill space requires two modules, speciﬁcally the VAE embedding module and ﬂows-based skill prior module.
Future work will explore using a single generative model such as a conditional VAE [34] to jointly handle the tasks of both modules.
Conclusion
This work proposes Residual Skill Policies (ReSkill), a general-purpose skill-based RL framework that enables effective skill re-use for adaptable and efﬁcient learning.
We learn a high-level skill agent within a state-conditioned skill space for accelerated skill composition, as well as a lowlevel residual agent that allows for ﬁne-grained skill adaptation to variations in the task and training environment.
Additionally, we show that our framework allows us to re-purpose a handful of scripted controllers to solve a wide range of unseen tasks.
We evaluate ReSkill across a range of manipulation tasks and demonstrate its ability to learn faster and achieve substantially higher ﬁnal returns across all task variations than the baseline methods.
We see this as a promising step towards the effective reuse of prior knowledge for efﬁcient learning without limiting the generality of the agent in learning a wide range of tasks.
Acknowledgments
K.R., M.X., M.M. and N.S. acknowledge the continued support from the Queensland University of
Technology (QUT) through the Centre for Robotics.
This research was additionally supported by the
QUT Postgraduate Research Award.
The authors would like to thank Robert Lee, Jad Abou-Chakra,
Fangyi Zhang, Vibhavari Dasagi, Tom Coppin and Jordan Erskine for their valuable discussions towards this contribution.
We would also like to thank the reviewers for their insightful and constructive comments towards the signiﬁcant improvement of this paper.