R ESET -F REE L IFELONG L EARNING WITH S KILL -S PACE P LANNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The objective of lifelong reinforcement learning (RL) is to optimize agents which can continuously adapt and interact in changing environments.
However, current
RL approaches fail drastically when environments are non-stationary and interactions are non-episodic.
We propose Lifelong Skill Planning (LiSP), an algorithmic framework for lifelong RL based on planning in an abstract space of higher-order skills.
We learn the skills in an unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model.
Moreover, our framework permits skill discovery even from ofﬂine data, thereby reducing the need for excessive real-world interactions.
We demonstrate empirically that LiSP successfully enables long-horizon planning and learns agents that can avoid catastrophic failures even in challenging non-stationary and non-episodic environments derived from gridworld and MuJoCo benchmarks.
I NTRODUCTION
Intelligent agents, such as humans, continuously interact with the real world and make decisions to maximize their utility over the course of their lifetime.
This is broadly the goal of lifelong reinforcement learning (RL), which seeks to automatically learn artiﬁcial agents that can mimic the continuous learning capabilities of real-world agents.
This goal is challenging for current RL algorithms as real-world environments can be non-stationary, requiring the agents to continuously adapt to changing goals and dynamics in robust fashions.
In contrast to much of prior work in lifelong RL, our focus is on developing RL algorithms that can operate in non-episodic or “reset-free” settings and learn from both online and ofﬂine interactions.
This setup approximates real-world learning where we might have plentiful logs of ofﬂine data but resets to a ﬁxed start distribution are not viable and our goals and environment change.
Performing well in this setting is key for developing autonomous agents that can learn without laborious human supervision in non-stationary, high-stakes scenarios.
[Figure]
However, the performance of standard RL algorithms drops signiﬁcantly in non-episodic settings.
To illustrate this issue, we ﬁrst pre-train agents to convergence in the episodic Hopper environment (Brockman et al., 2016) with state-of-the-art model-free and model-based RL algorithms: Soft Actor Critic (SAC) (Haarnoja et al., 2018) and Model-Based Policy Optimization (MBPO)
(Janner et al., 2019), respectively.
These agents are then trained further in a reset-free setting, representing a real-world scenario where agents seek to improve generalization via continuing to adapt at a test time where resets are more expensive.
The learning curves are shown in Figure 1.
In spite of a near-perfect initialization, all the agents proceed to fail catastrophically, suggesting that current gradient-based RL methods are inherently unstable in non-episodic settings.
This illustrative experiment complements prior work highlighting other failures of RL algorithms in non-stationary and nonepisodic environments: Co-Reyes et al.
ﬁnd current RL algorithms fail to learn in a simple gridworld environment without resets and Lu et al.
(2019) ﬁnd RL algorithms struggle to learn and adapt to nonstationarity even with access to the ground truth dynamics model.
We can attribute these failures to RL algorithms succumbing to sink states .
Intuitively, these are states from which agents struggle to escape, have low rewards, and suggest a catastrophic halting of learning progress (Lyapunov, 1992).
For example, an upright walking agent may fall over and fail to return to a standing position, possibly because of underactuated joints.
A lifelong agent must seek to avoid such disabling sink states, especially in the absence of resets.
We introduce Lifelong Skill Planning (LiSP), an algorithmic framework for reset-free, lifelong RL that uses long-horizon, decision-time planning in an abstract space of skills to overcome the above challenges.
LiSP employs a synergistic combination of model-free policy networks and model-based planning, wherein we use a policy to execute certain skills, planning directly in the skill space.
This combination offers two beneﬁts: (1) skills constrain the search space to aid the planner in ﬁnding solutions to long-horizon problems and (2) skills mitigate errors in the dynamics model by constraining the distribution of behaviors.
We demonstrate that agents learned via LiSP can effectively plan for longer horizons than prior work, enabling better long-term reasoning and adaptation.
Another key component of the LiSP framework is the ﬂexibility to learn skills from both online and ofﬂine interactions.
For online learning, we extend Dynamics-Aware Discovery of Skills (DADS), a model-based algorithm for unsupervised skills discovery (Sharma et al., 2019), with a skill-practice proposal distribution.
We demonstrate that the use of this proposal distribution signiﬁcantly ampliﬁes the signal for learning skills in reset-free settings.
For ofﬂine learning from logged interactions, we employ a similar approach as above but with a modiﬁcation of the reward function to correspond to the extent of disagreement amongst the models in a probabilistic ensemble (Kidambi et al., 2020).
Our key contributions can be summarized as follows:
• We identify skills as a key ingredient for overcoming the challenges to achieve effective lifelong RL in reset-free environments.
• We propose Lifelong Skill Planning (LiSP), an algorithmic framework for reset-free lifelong RL with two novel components: (a) a skill learning module that can learn from both online and ofﬂine interactions, and (b) a long-horizon, skill-space planning algorithm.
• We propose new challenging benchmarks for reset-free, lifelong RL by extending gridworld and MuJoCo OpenAI Gym benchmarks (Brockman et al., 2016).
We demonstrate the effectiveness of LiSP over prior approaches on these benchmarks in a variety of nonstationary, multi-task settings, involving both online and ofﬂine interactions.
B ACKGROUND
Problem Setup.
We represent the lifelong environment as a sequence of Markov decision processes (MDPs).
The lifelong MDP M is the concatenation of several MDPs ( M i , T i ) , where T i denotes the length of time for which the dynamics of M i are activated.
Without loss of generality, we assume the sum of the T i (i.e., the total environment time) is greater than the agent’s lifetime.
The properties of the MDP M i are deﬁned by the tuple ( S , A , P i , r i , γ ) , where S is the state space, A is the action space, P i : S ×
A × S → R are the transition dynamics, r i : S ×
A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.
Consistent with prior work, we assume r i is always known to the agent specifying the task; it is also easy to learn for settings where it is not known.
We use P and r as shorthand to refer to the current M i with respect to the agent.
The agent is denoted by a policy π : S
→ A and seeks to maximize its expected return starting from the current state s 0 : arg max π E s t +1 ∼P ,a t ∼ π
[ (cid:80) ∞ t =
γ t r ( s t , a t )] .
The policy π may be implemented as a parameterized function or an action-generating procedure.
We expect the agent to optimize for the current M i , rather than trying to predict the future dynamics; e.g., a robot may be moved to an arbitrary new MDP and expected to perform well, without anticipating this change in advance.
Skill Discovery.
Traditional single-task RL learns a single parametrized policy π θ ( ·| s ) .
For an agent to succeed at multiple tasks, we can increase the ﬂexibility of the agent by introducing a set of latent skills z ∈ [ − 1 , 1] dim ( z ) and learning a skill conditional policy π θ ( ·| s, z ) .
As in standard latent variable modeling, we assume a ﬁxed, simple prior over the skills p ( z ) , e.g., uniform.
The learning objective of the skill policy is to maximize some intrinsic notion of reward.
We denote the intrinsic reward as ˜ r
( s, z, s (cid:48) ) to distinguish it from the task-speciﬁc reward deﬁned previously.
For example, one useful reward proposed in DADS (Sharma et al., 2019) can be derived from a variational approximation to the mutual information between the skills and next states I ( s (cid:48) ; z | s ) as:
Initialize true replay buffer D , generated replay buffer ˆ D , dynamics model ensemble { f φ i } N − 1 i =0 , policy π θ , discriminator q ν , and skill-practice distribution p ψ if performing ofﬂine pretraining then
Learn dynamics model f φ and train policy π θ with UpdatePolicy until convergence while agent is alive at current state s do
Update dynamics model to maximize the log probability of transitions of D
Update policy models with UpdatePolicy ( D , ˆ D , f φ , π θ
[Equation]
q ν , p ψ ) Execute action from GetAction ( s, f φ , π θ ) and add environment transition to D
[Equation]
where z i ∼ p
( z ) .
Here q ν ( s (cid:48) | s, z ) is a tractable variational approximation for the intractable posterior p ( s (cid:48) | s, z ) .
Intuitively, this ˜ r learns predictable (via the numerator) and distinguishable (via the denominator) skills.
Model-Based Planning.
Whereas RL methods act in the environment according to a parameterized policy, model-based planning methods learn a dynamics model f φ ( s t +1 | s t , a t ) to approximate P and use Model Predictive Control (MPC) to generate an action via search over the model (Nagabandi et al., 2017; Chua et al., 2018).
At every timestep, MPC selects the policy π that maximizes the predicted H -horizon expected return from the current state s 0 for a speciﬁed reward function r :
[Equation]
We use Model Path Predictive Integral (MPPI)
(Williams et al., 2015) as our optimizer.
MPPI is a gradient-free optimization method that: (1) samples policies according to Gaussian noise on the optimization parameters, (2) estimates the policy returns, and (3) reweighs policies according to a
Boltzmann distribution on the predicted returns.
For our dynamics model, we use a probabilistic ensemble of N models { f φ i } N − 1 i =0 , where each model predicts the mean and variance of the next state.
The returns are estimated via trajectory sampling (Chua et al., 2018), where each policy is evaluated on each individual model for the entire H -length rollout and the returns are averaged.
L IFELONG S KILL -S PACE P LANNING
In this section, we present Lifelong Skill Planning (LiSP), our proposed approach for reset-free lifelong RL.
We provide an outline of LiSP in Algorithm 1.
The agent initially learns a dynamics model f φ and a skill policy π θ from any available ofﬂine data.
Thereafter, the agent continuously updates the model and policy based on online interactions in a reset-free lifelong environment.
The agent uses skill-space planning to act in the environment and avoid sink states.
In particular, our algorithmic contributions and components are as follows: • A primitive dynamics model f φ used for both planning and policy optimization • A low-level skill policy π θ trained from generated model rollouts on intrinsic rewards • A discriminator q ν for learning the intrinsic reward using Sharma et al. (2019) • A skill-practice distribution p ψ to generate a curriculum for training skills • Using the model-based disagreement penalty of Kidambi et al. (2020) for ofﬂine learning
3. M ODEL -B ASED S KILL D ISCOVERY
Our goal is to learn a skill-conditioned policy π θ .
In order to minimize interactions with the environment, we ﬁrst learn a model f φ to generate synthetic rollouts for policy training.
Since there is no start state distribution, the initialization of these rollouts is an important design choice.
We propose to learn a skill-practice distribution p ψ ( z | s ) to deﬁne which skills to use at a particular stage of
Hyperparameters: number of rollouts M , disagreement threshold α thres
Function UpdatePolicy( replay buffer D , generated replay buffer ˆ D , dynamics model f φ , policy π θ , discriminator q ν , skill-practice distribution p ψ ) :
for i = 1 to M do
Sample s i 0 uniformly from D and latent z i from skill-practice p ψ ( ·| s i 0 )
Generate s i 1 := f φ ( ·| s i 0 , π θ ( ·| s i 0 , z i )) and add transition ( s i 0 , a i , z i , s i 1 ) to ˆ D
Update discriminator q ν on { s i 0 , a i , z i , s i 1 } Mi =1 to maximize log q ν ( s 1 | s 0 , z )
Calculate intrinsic rewards ˜ r adjusted for ˆ D with q ν , α thres using Equations 1 and
Update π θ , q ν , p ψ using SAC with minibatches from ˆ D training.
p ψ acts as a “teacher” for π θ , automatically generating a curriculum for skill learning.
We include visualizations of the learned skills on 2D gridworld environments in Appendix C.2.
To actually learn the policy, we use the model to generate short rollouts, optimizing π θ with SAC, similar to model-based policy learning works that ﬁnd long rollouts to destabilize learning due to compounding model errors (Janner et al., 2019).
To initialize the rollout, we sample a state from the replay buffer D and a skill to practice via p ψ .
We learn p ψ by learning a model using SAC to optimize the same intrinsic reward ˜ r adjusted as π θ , intuitively only selecting skills which are most useful from the current state instead of arbitrary skills.
The actual next state prediction is given by the dynamics model f φ , where the model used to make the prediction is uniformly sampled from the ensemble.
Note that gradients do not go through f φ when training the other distributions.
This is summarized in Algorithm 2.
3.1. O FFLINE S KILL L EARNING
The key problem in ofﬂine RL is avoiding value overestimation, typically by constraining actions to the support of the dataset.
We can use our same algorithm to learn skills ofﬂine with a simple adjustment to the intrinsic reward based on the model disagreement (Kidambi et al., 2020).
For hyperparameters κ, α thres ∈ R + , we replace the intrinsic reward ˜ r with ˜ r adjusted , penalizing ˜ r with − κ if the expected model disagreement is greater than α thres .
This penalty encourages the policy to stay within the support of the training set by optimizing against an MDP constructed to underestimate the value function.
We approximate the expected (cid:96) 2 disagreement in the mean prediction of the next state, denoted µ φ i for model i , between members of the ensemble with a sample.
The disagreement estimates the epistemic uncertainty.
This is shown in Equation 3.
[Equation]
3. P LANNING IN S KILL S PACE FOR R ESET -F REE A CTING
As described in Section 1, we argue that the failures of RL in lifelong RL arise chieﬂy from the naive application of model-free RL.
In particular, it is imperative that the model not only be used for sample-efﬁcient policy optimization (as in MBPO), but also that the model be used to safely act in the environment.
The method in which acting is performed is important, serving both to exploit what the agent has learned and crucially to maintain the agent’s safety in reset-free settings.
In this work, we propose to use model-based planning via MPC.
We differ from prior MPC works by planning with the set of skills from Section 3.1, which allow us to utilize the broad capabilities of model-free RL while still enabling the beneﬁts of model-based planning.
Planning over skills constrains the MPC optimization to a subset of the action space which the policy is conﬁdent in.
The model-free policy learns to act in a reliable manner and is consequently more predictable than naive actions.
As a result, we are able to perform accurate planning for longer horizons than before (Chua et al., 2018;
Nagabandi et al., 2019b).
Note that the complexity of our proposed approach is the same as prior MPC works, i.e. it is linear in the length of the horizon.
We summarize this subroutine in Algorithm 3.
Hyperparameters: population size S , planning horizon H , planning iterations P , discount γ
Function GetAction( current state s 0 , dynamics model f φ , policy π θ ) : for P planning iterations do
Sample skills { z i } Si =1 ∼ [ − 1 , 1] dim ( z ) × H based on distribution of previous iteration
Estimate returns R = { (cid:80) H − 1 t =
γ t r ( s it , π θ ( ·| s it , z it ) , s it +1 ) }
Si =1 using trajectory sampling, with states s i sampled from f φ for skills
z i
Use MPPI update rule on R and z to generate new distribution of skills { z 0 } H − 1 t = return a ∼ π θ ( ·| s, z 0 )
We summarize the LiSP subroutines in Figure 2.
Skills are ﬁrst learned via Algorithm 2, wherein the skill discriminator generates the intrinsic rewards and the skill-practice distribution generates a skill curriculum.
We then plan using the skill policy and the dynamics model as per Algorithm 3.
E XPERIMENTAL E VALUATIONS
We wish to investigate the following questions with regards to the design and performance of LiSP:
• Does LiSP learn effectively in lifelong benchmarks with sink states and nonstationarity?
• What are the advantages of long-horizon skill-space planning over action-space planning?
• Why is the skill-practice distribution important for learning in this setting?
• Does LiSP learn a suitable set of skills fully ofﬂine for future use?
Lifelong learning benchmarks.
We extend several previously studied environments to the lifelong
RL setting and distinguish these benchmarks with the preﬁx Lifelong .
The key differences are that these environments have an online non-episodic phase where the agent must learn without resets and (optionally) an ofﬂine pretraining starting phase.
The agent’s replay buffer D is initialized with a starting dataset of transitions, allowing the agent to have meaningful behavior from the outset.
More details are in Appendix C and we will open source the implementations for wider use.
Unless stated otherwise, we run 3 seeds for each experiment and our plots show the mean and standard deviation. Comparisons.
We compare against select state-of-the-art algorithms to help illustrate the value of different design choices.
The closest baseline to ours is action-space MPC, also known as
PETS (Chua et al., 2018), which directly ablates for the beneﬁt of using skills for planning.
We also compare to SAC (Haarnoja et al., 2018), representing the performance of model-free RL on these tasks.
Finally we consider MOReL (Kidambi et al., 2020), the ofﬂine model-based algorithm which proposed the disagreement penalty discussed in Section 3.1.1. Note that the latter two are principally single-task algorithms, while MPC is more suited for multiple reward functions. We believe these represent a variety of RL algorithms that could be considered for the reset-free setting. 4. E VALUATION ON L IFELONG B ENCHMARKS
We begin by evaluating the overall LiSP framework (Algorithm 1) in non-episodic RL settings, particularly how LiSP interacts with sink states and its performance in nonstationary reset-free settings.
Nonstationary Mujoco locomotion tasks.
We evaluate LiSP on Hopper and Ant tasks from
Gym (Brockman et al., 2016); we call these Lifelong Hopper and Lifelong Ant.
The agents seek to achieve a target x-velocity, which changes over the agent’s lifetime.
Learning curves are shown in Figure 3.
Most of the LiSP seeds remain stable despite sink states and adapt instantly to the current tasks.
In line with Figure 1, the SAC and MOReL agents are unstable and do poorly.
The long-horizon planning capability of LiSP is crucial, signiﬁcantly outperforming SAC and MOReL, which lack the ability to plan.
Furthermore, planning over the space of skills is necessary to achieve improvement over MPC, which is not capable of accurate planning on its own.
LiSP outperforms all of the baselines tested.
[Figure] [Figure]
Minimizing resets with permanent
We evaluate our method in a lifelong 2D volcano gridworld environment where the agent navigates to reach goals while avoiding pitfalls which permanently trap the agent if there is no intervention.
Every 100 timesteps, the pitfalls and goals rearrange, which allow the agent to get untrapped; we can consider this as a “reset” if the agent was stuck as it required this intervention to free it.
We use limited pretraining for this environment, i.e. we train the model but not the policies, reﬂecting the ability of the agents to act safely while not fully trained.
Figure 4 shows the number of times the agent got stuck during training.
We ﬁnd all the model-based methods, including LiSP, are easily capable of avoiding resets, suggesting model-based algorithms are naturally suited for these settings and incorporating data to avoid failures.
We note that LiSP is also stable in the Lifelong Ant environment which likewise has permanent sink states, as the ant cannot recover after falling over.
4. A DVANTAGES OF L ONG H ORIZON S KILL -S PACE P LANNING
Next, we seek to clearly show the advantages of planning in the skill space, demonstrating the beneﬁts of using skills in Algorithm 3 and arguing for the use of skills more broadly in lifelong RL.
Constraining model error.
We can interpret skill-space planning as constraining the MPC optimization to a more accurate subset of the action space which the policy operates in.
In Figure 5, we consider Lifelong Hopper and look at the one-step dynamics model error when evaluated on actions generated by the skill policy vs uniformly from the action space.
While most samples from both have similar model error, the variance in the residual error for random actions is high.
[Figure]
Constraining planning search space.
This advantage in the one-step model error extends to longhorizon planning.
In the Lifelong Hopper environment, accurate long-horizon planning is critical in order to correctly execute a hopping motion, which has traditionally made MPC difﬁcult in this environment.
For good performance, we use a long horizon of 180 for planning.
In Figure 6, we ablate LiSP by planning with actions instead of skills (without gradient updates).
Accurate planning with actions is completely infeasible due to the accumulated unconstrained model errors.
[Figure]
Model updates are extremely stable.
Furthermore, as described in Figure 1, we found that SAC and MBPO fail in reset-free settings when denied access to resets if they continue gradient updates, which we attributed to gradient-based instability.
Here we try to understand the stability of RL algorithms; in particular, since we found that policy/critic updates are unstable, we can instead consider how model updates affect planning.
To do so, we consider SAC as before, as well as action-space
MPC over a short 5-step horizon trying to optimize the same SAC critic (which depends on the policy).
These can be viewed as simple ablations of LiSP, as LiSP still uses the dynamics model to predict states from actions, with an additional terminal value function.
We note that this is similar to work in Sikchi et al.
The results are shown in Figure 7.
If only the model is updated, the algorithm is very stable, showing the resilience of planning to model updates.
Alternatively, value function updates make planning unstable, supporting the idea that long-horizon planning improves stability vs relying on a value function.
However, even the short 5-step planner avoids the catastrophic failures that occur with no planning, as the additional model stability quickly neturalizes value instability.
This also motivates planning with a model using action input, rather than skills as in Sharma et al.
(2019), since predictions will remain stable as they do not depend on the policy. [Figure]
Hopper with MPC
[Figure]
4. O NLINE S KILL L EARNING WITH A M ODEL
Here, we show the importance of the skill-practice distribution from Algorithm 2 for learning setups that require hierarchical skills and involve short model-based rollouts.
Hierarchical skill learning.
We consider a 2D Minecraft environment where the agent must learn hierarchical skills to build tools, which in turn are used to create higher level tools.
We ablate against minimizing ˜ r adjusted and a baseline of not using a practice distribution; the former acts as a sanity check whereas the latter represents previous skill discovery algorithms. Note that we do not compare against MOReL here since the asymptotic performance will be similar to SAC, as they both rely on Q-learning policy updates. Similar to the volcano environment, we only do limited pretraining on a small set of starting data.
The learning curves are in Figure 8.
Learning skills with directed practice sampling is necessary for learning useful skills.
Without the skill-practice curriculum, the asymptotic performance of LiSP is signiﬁcantly limited.
Additionally, the fact that MPC performs signiﬁcantly worse than LiSP suggests that planning in the skill space – namely, planning in a space of expressive skills – greatly aids the search process and exploration.
[Figure]
Learning with short rollouts.
Algorithm 2 learns skills using one-step rollouts from arbitrary states.
These short rollouts signiﬁcantly reduce learning signal relative to sampling a skill from a ﬁxed start state distribution and running the skill for an entire episode: rather than being able to associate all the states along an episode with a skill, the agent will be forced to associate all states with all skills. Intuitively, the skill-practice distribution can alleviate this, associating states with certain important skills, which can make skill learning easier.
We investigate this by analyzing DADS-Off (Sharma et al., 2020) – an off-policy improved version of DADS – and resampling skills every K timesteps.
This simulates K -step rollouts from arbitrary starting states, approximating LiSP except generating data from the real environment.
We show the learned skills with various
K for Lifelong Ant in Figure 9.
Despite DADS-Off using off-policy data, resampling skills hurts performance severely.
When generating transitions with a model as in LiSP, it is unstable to run long model rollouts due to compounding errors, making the skill-practice distribution critical.
[Figure]
4. L EARNING S KILLS E NTIRELY F ROM O FFLINE D ATA
Finally, we evaluate the entire LiSP framework (Algorithm 1) in a fully ofﬂine setting without testtime updates, considering LiSP as an ofﬂine RL algorithm.
The most direct baseline to LiSP is action-space MPC (PETS) (Chua et al., 2018), which can learn ofﬂine for multiple test-time tasks; we consider PETS as a LiSP ablation that plans without skills.
We consider two versions: one that uses a long horizon for planning ( H = 180 , PETS-Long) and one with a short horizon ( H = 25 ,
PETS-Short).
The former is a direct ablation, while the latter is closer to the original work.
Learning skills ofﬂine for multiple tasks.
We consider three Lifelong Hopper tasks, where tasks deﬁne target velocities.
We use the same dataset as Section 4.1, collected from a forward hop task.
Our results are in Table 1.
While PETS-Long can sometimes perform well, it has high variance;
PETS-Short fails.
LiSP successfully learns a set of skills ofﬂine for all tasks with low variance.
[Table]
D ISCUSSION AND R ELATED W ORK
In this section, we provide an overview of other works related to LiSP, in similar ﬁelds to lifelong RL.
We include an in-depth discussion of the related works in Appendix A to highlight the relationship of our work with other subﬁelds of RL.
Non-Episodic RL.
The traditional approach to non-episodic RL is to explicitly learn a policy to reset the environment (Eysenbach et al., 2017; Han et al., 2015; Even-Dar et al., 2005).
This requires somewhat hard-to-deﬁne notions of what a reset should accomplish and still uses manual resets.
We do not learn reset policy, instead focusing on naturally safe acting via learning from ofﬂine data and effective planning.
Zhu et al.
(2020) and Co-Reyes et al.
(2020) propose solutions to lack of learning signal in reset-free settings but only consider environments without sink states; the latter requires control over the environment to form a training curriculum.
Lu et al.
and Lowrey et al.
(2018) highlight the beneﬁts of both planning and model-free RL for lifelong agents but require highly accurate world models.
Ofﬂine RL learns policies exclusively from ofﬂine data, which naturally lacks resets (Levine et al., 2020; Agarwal et al., 2019;
Wu et al., 2019; Yu et al., 2020; Kumar et al., 2020; Kidambi et al., 2020), but most work is restricted to single-task, stationary settings.
Model-Based Planning.
Existing works in model-based planning are restricted to short horizons (Chua et al., 2018; Wang & Ba, 2019; Nagabandi et al., 2019b).
Similarly to LiSP, some works (Henaff et al., 2019; Kahn et al., 2017) try to reduce model error for planning by penalizing deviations outside the training set.
We found embedding uncertainty into the cost function causes poor cost shaping; these works also still have relatively short horizons.
Mishra et al.
(2017) learns action priors used to generate actions for MPC that embed past actions in a latent space for constrained planning, similarly to how we skill constraint, but only considers a fairly simple manipulation task.
Some works (Lowrey et al., 2018; Lu et al., 2019; Sikchi et al., 2020), including in ofﬂine RL (Argenson & Dulac-Arnold, 2020), use terminal value functions to aid planning, allowing for successful short-horizon planning; as discussed previously, this does not directly translate to good performance in the lifelong setting due to instability and challenges in learning this function.
Skill Discovery.
The key difference from prior skill discovery work is the lack of episodes.
The most relevant work to ours is DADS (Sharma et al., 2019).
Post-training, the skills learned by DADS can be used for planning with the discriminator q ν .
Unlike their setup, we plan over the model with action predictions, which allows for accurate planning even if q ν is not accurate.
Most other works (Hansen et al., 2019; Campos et al., 2020; Warde-Farley et al., 2018; Mohamed & Rezende, 2015;
Achiam et al., 2018) are complementary methods that can help learn a better set of skills.
Some seek to use skills as pretraining for episodic learning, as opposed to our focus on safe reset-free acting.
C ONCLUSION
We presented LiSP, an algorithm for lifelong learning based on skill discovery and long-horizon skill-space planning.
To encourage future work in this space, we proposed new benchmarks that capture the key difﬁculties of lifelong RL in reset-free, nonstationary settings.
Our experiments showed that LiSP effectively learns in non-episodic settings with sink states, vastly improving over prior RL methods, and analyzed ablations to show the beneﬁts of each component.
A F URTHER D ISCUSSION AND R ELATED W ORK
Model-Based Planning.
Chua et al.
(2018) propose to use an ensemble of probabilistic dynamics models for planning, which we use, but by itself struggles to scale to higher-dimensional Mujoco tasks due to limited model accuracy; the use of model ensembling to mitigate errors has also been previously explored by Nagabandi et al.
(2017) and Kurutach et al.
Wang & Ba (2019) improves upon some of the weaknesses of Chua et al.
(2018) by planning in the parameter space of policies, solving some MuJoCo tasks asymptotically, but is still not accurate enough to plan over horizons long enough for environments that require a long sequence of coordinated actions.
Nagabandi et al.
(2019b) shows that an improved optimizer can improve MPC performance for short-horizon tasks.
We note that all of these methods struggle to plan for long horizons due to compounding model errors, and none have been able to perform MPC in the Hopper environment without a terminal value function (as in Sikchi et al.
and Argenson & Dulac-Arnold (2020)), so this is a novelty of our work.
Argenson & Dulac-Arnold (2020), Lu et al.
, Wang & Ba (2019), and Sikchi et al.
(2020) initialize the actions for MPC with a prior policy, but this is not the same as constraining the space.
Sometimes early termination and low Gaussian noise for random shooting is used to attempt to approximate a constraint, but this is not very effective in higher dimensional environments.
Consequently, they do not have as strong long-horizon beneﬁts, instead using other methods for attaining strong performance.
Some works (Hafner et al., 2019; Venkatraman et al., 2015; Mishra et al., 2017) try to improve long horizon planning accuracy with multi-step prediction losses, which is complementary to our work and could improve performance.
Furthermore, even though value functions for short horizon planning is not particularly promising, we note that terminal value functions still have other beneﬁts that can improve performance when combined with long-horizon planning, such as improved exploration or long-term reasoning (Lowrey et al., 2018; Lu et al., 2019).
Hierarchical RL.
Our work is somewhat similar to hierarchical option-critic architectures that “plan” without MPC (Sutton et al., 1999; Bacon et al., 2016; Nachum et al., 2018; Khetarpal et al., 2020), which is sometimes referred to as “background-time planning” (Sutton & Barto, 2018).
However, our skills are learned with an unsupervised reward and composed via MPC, which has the promise of more explicit long-term hierarchical beneﬁts, whereas there is some evidence (Nachum et al., 2019) that policy-based hierarchy only aids in exploration – not long-term reasoning – and is replaceable with monolithic exploration methods.
Our skill-practice distribution somewhat resembles the interest function of Khetarpal et al.
(2020); both are designed to associate certain skills with certain states, forming a type of curriculum.
In particular, they note that using a small number of skills improves learning early but can be asymptotically limiting, which has some similarity to curriculums generated from the skill-practice distribution.
We show the entropy of the skills generated for the 2D Minecraft environment in Figure 10; the skill-practice distribution automatically generates a curriculum where it purposefully practices less skills (low entropy) early in training, and more skills (high entropy) later.
Unlike Khetarpal et al.
(2020), our skill-practice distribution is not part of the policy update and is only used to generate a curriculum.
We also note that while this means that transitions generated from the model are not drawn from the prior distribution in the objective, since the critic Q ( s, z, a ) learns a target for the next state using the same z as Q ( s (cid:48) , z, π ( a | s (cid:48) , z )) , it is correct without importance sampling.
Safety.
Zhang et al.
(2020) perform risk-averse planning to embed safety constraints, but require a handcrafted safety function.
This is similar to arguments from the Safety Gym work (Achiam &
Amodei, 2019), which argues that the correct way to formulate safe RL is with a constrained optimization problem.
However, the safety in our work deals more with stability and control, rather than avoiding a cost function, so our notion of safety is generally different from these ideas.
Consequently, we found that long-horizon planning and trying to perform accurate value estimation at greater scale to be more promising than constraining against a cost function or using explicit risk aversion.
We note that some “empowerment”-style methods (Gregor et al., 2016; Karl et al., 2017;
Eysenbach et al., 2018; Sharma et al., 2019) optimize an objective that aims to resemble a notion of stability; however, this metric is hard to estimate accurately in general, so we do not use it for any form of constrained planning.
Zhao et al.
(2020) learn a more accurate estimator of the empowerment, which could be useful for future work in safety.
[Figure]
Nonstationarity.
Much work in nonstationarity focuses on catastrophic forgetting (Kirkpatrick et al., 2017;
Schwarz et al., 2018; Rusu et al., 2016), which is not a primary goal of our work as we are primarily concerned with acting competently in the current MDP without explicitly trying to remember all previous or any arbitrary MDPs.
Our approach to nonstationary lifelong learning follows Lu et al.
, but we do not require access to the ground truth dynamics P at world changes.
Lecarpentier & Rachelson (2019) tackles nonstationarity in zero-shot for simple discrete
MDPs by using risk-averse planning; our experiments generally suggested that accurate and scalable planning was more important than explicit risk aversion for our settings.
Other works (Nagabandi et al., 2019a; Finn et al., 2019; Rolnick et al., 2019) are typically nonstationary at the abstraction of episodes, which generally does not require competencies such as online safe adaptation, and instead try to adapt quickly, minimize regret, or avoid catastrophic forgetting as previously discussed.
Xie et al.
(2020) is closer to setting, as they consider nonstationarity within an episode, but they still require manual resets except in a 2D environment which does not contain sink states.
B F URTHER P LOTS FOR L EARNING D YNAMICS
In this section, we provide additional plots, seeking to give more insights on the learning of LiSP from the MuJoCo experiments from Section 4.1.
In particular, we can consider the intrinsic reward, which is a proxy for the diversity of skills.
Since the intrinsic reward is calculated under the model, which changes, high intrinsic reward under the model is not always the best indicator, whereas it is a more reliable metric when learned from real world transitions as in Sharma et al.
The intrinsic reward during the ofﬂine phases are:
[Figure]
C E NVIRONMENT D ETAILS
In this section, we provide additional details about our lifelong experiments and the environment setup.
Our environment settings were designed such that the optimal behaviors are similar to the optimal behaviors in the Gym version of these environments, and so that the difﬁculty of learning was similar (although the learning dynamics are different).
This is generally accomplished by having some part of the reward for stability, in addition to the standard reward for accomplishing the task, which also emphasizes the health of an agent by considering the reward; standard benchmarks lack this because the signal comes from a termination function.
While this extra shaping of the reward gives more signal than normal, these settings are still hard for standard RL algorithms as highlighted above, and even episodic RL without early termination using the standard rewards tends to fail when episodes are long relative to time-to-failure.
C. L IFELONG M U J O C O E NVIRONMENTS
In this work, we considered two MuJoCo tasks: Lifelong Hopper and Lifelong Ant.
The reward functions we use for these environments are as follows, where z is the height of the agent, x vel is the x-velocity, and ˆ x vel is a target x-velocity:
• Lifelong Hopper: − 5( z − 1 .
8) 2 − | x vel − ˆ x vel | + | ˆ x vel | • Lifelong Ant: − ( z − 0 .
7) 2 − | x vel − ˆ x vel | + | ˆ x vel |
For the experiments in Section 4.1, we change the target velocity every 1000 timesteps according to the following ordering (chosen arbitrarily):
• Lifelong Hopper: [0 , 1 , − 1 , 2 , − 1] • Lifelong Ant: [1 , − 1 , 1]
In principle, these can be adjusted as desired for various settings.
The dataset we used for both tasks was the replay buffer generated from a SAC agent trained to convergence.
While this is more data than typically used in ofﬂine RL (Fu et al., 2020), our setting is still difﬁcult and highly nontrivial due to sink states and gradient instability.
We hope future works will continue to explore non-episodic with varying datasets.
C. 2D G RIDWORLD E NVIRONMENTS AND S KILL V ISUALIZATIONS
In this work, we considered two continuous 2D gridworld tasks: volcano and 2D Minecraft.
The volcano environment allows us to consider the safety of agents, and Minecraft is a well-known video game setting that is attractive for hierarchical RL due to the nature of tools (Guss et al., 2019); our version is a simpler 2D version that is faster to train.
We visualize both these environments and skills learned on them below:
In the volcano environment, there is lava (shown in dark red), where the agent is penalized for standing in but does not inhibit its movement.
The green tile, located on the right side in the picture, denotes the target goal.
The reward is given by a negative L2 distance to this goal.
The white squares denote free tiles.
The primary square of interest is the brown tile, representing a pitfall, where the agent will be trapped if it falls in.
We can see that the learned skills correspond to avoiding the tile, representing how the skills themselves can embed safety, whereas planning in action space would not achieve the same effect.
The state is the agent’s x-y position, the position of the pitfall, as well as the position of the goal.
We initialize the dataset with some demonstrations of a trained SAC agent with noise applied to actions.
In the 2D Minecraft environment, the agent maintains an inventory of up to one of each item, and receives a reward whenever obtaining a new item (either through mining or crafting) in increasing magnitudes depending on the hierarchy of the item (how many items were needed to obtain beforehand to obtain the item).
There are four tiles of interest:
• A crafting table (bottom right), where the agent will craft all possible tools using the current materials in its inventory • A wood block (bottom left), where the agent will obtain a wood for visiting • A stone block (top left), where the agent will obtain a stone for visiting if it has a wooden pickaxe, consuming the pickaxe • An iron block (top right), where the agent will obtain an iron for visiting if it has a stone pickaxe, consuming the pickaxe and yielding the maximum reward in the environment
The skill progression is representing by the following ordering:
Walk to wood to mine 2.
Bring wood to craft table to craft a stick 3.
Craft wooden pickaxe with both a wood and a stick 4.
Walk to stone to mine using a wooden pickaxe 5.
Craft stone pickaxe with both a stone and a stick 6.
Walk to iron to mine using a stone pickaxe
There is skill reuse in that lower-level skills must be executed multiple times in order to accomplish higher-level skills, forming a hierarchy of skills in the environment.
The state is the position of the agent, the position of the blocks it can interact with, as well as its inventory, represented by a vector of 0’s or 1’s to represent ownership of each item.
The dataset consists of 2000 steps of manually provided demonstrations, which is a relatively low amount of demos.
C. P ERFORMANCE M ETRICS
To summarize performance easily in our experiments, we use a performance metric such that it is possible to achieve an optimal performance of 1 , and a representative “worst-case” performance is set to 0.
This allows for easy comparison between tasks and represents difﬁculty in harder tasks.
The weighting of the performance is similar to the standard reward functions, prioritizing stability with the height term and rewarding competence with a task-speciﬁc term, and easily represents the overall gap to a relatively perfect policy/desired behavior with a single number.
For Lifelong Hopper, take the average height z avg of the evaluation duration and the average x velocity x vel .
Note that it is impossible to make a similar normalization to 1 with returns, since it is impossible to attain the optimal height and velocity from every timestep, but the averages can be optimal.
For some target velocity ˆ x vel , the performance we use is given by:
[Equation]
The height is derived from approximately the height of an optimal SAC agent on the standard Hopper benchmark, representing the behavior we aim to achieve.
For Lifelong Ant, we use a similar metric:
[Equation]
D H YPERPARAMETERS
Our code will be released and made open source upon publication.
For all algorithms (as applicable) and environments, we use the following hyperparameters, roughly based on common values for the parameters in other works (note we classify the skill-practice distribution as a policy/critic here):
• Discount factor γ equal to 0.
• Replay buffer D size of 10• Dynamics model with three hidden layers of size 256 using tanh activations • Dynamics model ensemble size of 4 and learning rate 10 − 3 , training every 250 timesteps • Policy and critics with two hidden layers of size 256 using ReLU activations • Discriminator with two hidden layers of size 512 using ReLU activations • Policy, critic, discriminator learning rates of 3 × 10 − 4 training every timestep • Automatic entropy tuining for SAC • Batch size of 256 for gradient updates • MPC population size S set to• MPC planning iterations P set to• MPC number of particles for expectation calculation set to• MPC temperature of 0.
• MPC noise standard deviation of• For PETS, we use a planning horizon of either 25 or 180, as mentioned • For DADS, we ﬁnd it helpful to multiply the intrinsic reward by a factor of 5 for learning (for both DADS and LiSP usage)
For LiSP speciﬁcally, our hyperparameters are:
• Planning horizon of• Repeat a skill for three consecutive timesteps for planning (not for environment interaction) • Replan at every timestep • Number of rollouts per iteration
M set to• Generated replay buffer ˆ D size set to• Number of prior samples set to 16 in the denominator of the intrinsic reward • Number of discriminator updates per iteration set to• Number of policy updates per iteration set to• Number of skill-practice updates per iteration set to• Disagreement threshold α thres set to 0.05 for Hopper, 0.1 for Ant • Disagreement penalty κ set to