H IERARCHICAL R EINFORCEMENT L EARNING BY D ISCOVERING I NTRINSIC O PTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a hierarchical reinforcement learning method, HIDIO, that can learn task -agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks.
Unlike current hierarchical RL approaches that tend to formulate a goal-reaching framework or pre-deﬁne ad hoc lower-level policies, HIDIO encourages lower-level option learning that is independent of the task at hand, requiring no assumptions or knowledge about the task structure.
These options are learned through an intrinsic entropy minimization objective conditioned on the option trajectories.
The learned options are diverse , shortsighted, and non task-driven.
In experiments on robotic manipulation and navigation tasks with sparse rewards HIDIO achieves higher success rates with greater sample efﬁ - ciency than regular RL baselines and two state of-the-art hierarchical RL methods.
I NTRODUCTION
Imagine a wheeled robot learning to kick a soccer ball into a goal with sparse reward supervision.
In order to succeed, it must discover how to ﬁrst navigate in its environment, then touch the ball, and ﬁnally kick it into the goal, only receiving a positive reward at the end for completing the task.
This is a naturally difﬁcult problem for traditional reinforcement learning (RL) to solve, unless the task has been decomposed into temporally extended stages where each stage constitutes a much easier subtask.
In this paper we ask, how do we learn such decompositions automatically and utilize them to solve sparse reward problems?
Deep RL has made great strides solving a variety of tasks recently, with hierarchical RL (hRL) demonstrating promise in solving such sparse reward tasks (Sharma et al., 2019b; Le et al., 2018;
Merel et al., 2019; Ranchod et al., 2015).
In hRL, the task is decomposed into a hierarchy of subtasks, where policies at the top of the hierarchy call upon policies below to perform actions to solve their respective subtasks.
This abstracts away actions for the policies at the top levels of the hierarchy.
hRL makes exploration easier by potentially reducing the number of steps the agent needs to take to explore its state space.
Moreover, at higher levels of the hierarchy, temporal abstraction results in more aggressive, multi-step value bootstrapping when temporal-difference (TD) learning is employed.
These beneﬁts are critical in sparse reward tasks as they allow an agent to more easily discover reward signals and assign credit.
However, many existing methods make assumptions about the task structure ( e.g. , two stages: ﬁrst moving towards the object, and then picking it up), and/or the skills needed to solve the task ( e.g. , pre-programmed motor skills) (Florensa et al., 2016; Riedmiller et al., 2018;
Lee et al., 2019; Hausman et al., 2018; Lee et al., 2020; Sohn et al., 2018; Ghavamzadeh & Mahadevan, 2003; Nachum et al., 2018).
Instead, we seek to formulate a general method that can learn these abstractions, for any task, with little manual design.
The main contribution of this paper is HIDIO ( HI erarchical RL by D iscovering I ntrinsic O ptions), a hierarchical method that discovers task-agnostic intrinsic options to be executed in the environment, in a self-supervised manner.
These options are simultaneously learned to be applied to the environment to maximize reward.
We propose and empirically evaluate various instantiations of the option discovery process, comparing the resulting options with respect to their ﬁnal task performance.
We demonstrate that HIDIO is able to efﬁciently learn and discover diverse options to be utilized for higher task reward with superior sample efﬁciency compared to other hierarchical methods.
P RELIMINARIES
We consider the reinforcement learning (RL) problem in a Markov Decision Process (MDP).
Let s ∈ R S be the agent state.
We use the terms “state” and “observation” interchangeably to denote the environment input to the agent.
A state can be fully or partially observed.
Without loss of generality, we assume a continuous action space a ∈ R A for the agent.
Let π θ ( a | s ) be the policy distribution with learnable parameters θ , and P ( s t +1 | s t , a t ) the transition probability that measures how likely the environment transitions to s t +1 given that the agent samples an action by a t ∼ π θ ( ·| s t ) .
After the transition to s t +1 , the agent receives a deterministic scalar reward r ( s t , a t , s t +1 ) .
The objective of RL is to maximize the sum of discounted rewards with respect to θ :
[Equation]
where γ ∈ [0 , 1] is a discount factor.
We will omit P in the expectation for notational simplicity.
In the options framework (Sutton et al., 1999), the agent can switch between different options during an episode, where an option is translated to a sequence of actions by an option-conditioned policy with a termination condition.
For a two-level hierarchy, the expectation in Eq.
1 is taken over policies at both levels.
In the following, we will use the terms “goal”, “option”, and “skill” interchangeably.
H IERARCHICAL RL BY D ISCOVERING I NTRINSIC O PTIONS
K
[Figure]
We now introduce our hierarchical method for solving sparse reward tasks in general settings.
We allow the reward function to be deﬁned in any form, not necessarily deﬁned by an “ (cid:15) - region” in the state space as assumed in some sparse reward settings (Andrychowicz et al., 2017; Riedmiller et al., 2018).
For the purpose of this paper, we consider a hierarchy of two levels.
The higher-level policy (the scheduler π θ ), is trained to maximize environment reward, while the lower-level policy (the worker π φ ) is trained in a self-supervised manner to efﬁciently discover options that are utilized by π θ to accomplish tasks.
Importantly, the worker gets access to dense intrinsic rewards regardless of the sparsity of the extrinsic rewards.
Without loss of generality, we assume that each episode has a length of T and the scheduler outputs an option every K steps.
Given a scheduled option u ∈ [ − 1 , 1] D ( e.g. , tanh activated), the worker executes K steps before the scheduler outputs the next option.
Let the time horizon of the scheduler be H =
(cid:100) TK (cid:101) , and denote s h,k ≡ s hK + k and a h,
k ≡ a hK + k .
Formally, we deﬁne
Scheduler policy:
[Equation]
Worker [Equation] Note that given the notation and sampling process, we have s h,K ≡ s h +1 , 0 , namely, the last state of the current option u h is the initial state of the next option
u
h +1 .
The overall framework of our method is illustrated in Figure 1.
The pseudo code of HIDIO is in Appendix A.
3. L EARNING THE S CHEDULER
Every time the scheduler issues an option u h , it receives an reward R h computed by accumulating environment rewards over the next K steps.
Its objective is: max
[Equation]
E
[Equation]
θ (cid:34) H
[Equation]
1 (cid:88) h
= β h R h (cid:35) , where β = γ K and R h =
E π φ (cid:104)(cid:80) K − 1 k
= 0 γ k r (
s
h,k , a h,k , s h,k +1 ) (cid:105) (3)
This scheduler objective itself is not a new concept, as similar ones have been adopted by other hRL methods (Vezhnevets et al., 2017; Nachum et al., 2018; Riedmiller et al., 2018).
One signiﬁcant difference between our option with that of prior work is that there is no constraint on what u could represent.
In contrast, existing methods usually require their options to reside in a subspace of the state space, to be grounded to the environment, or to have known structures, so that the scheduler can compute rewards and termination conditions for the worker.
As a result, their options are taskdependent and might have difﬁculty adapting to new scenarios without being redesigned.
3. L EARNING THE W ORKER
The main focus of this paper is to investigate how to effectively learn the worker policy in a selfsupervised manner.
Many existing hRL works either pre-deﬁne or pre-train ad hoc worker policies (Riedmiller et al., 2018) to be composed by the scheduler, or jointly train the worker to reach goals that reside in a subspace of the (latent) state space (Vezhnevets et al., 2017; Nachum et al., 2018).
Our motivation is that it might be unnecessary to explicitly ask the worker to reach some “ (cid:15) -space” of goals.
As long as a goal can be translated to a short sequence of primitive actions, it does not need to be grounded with concrete meanings.
Our worker learning is inspired by similar ideas explored by
DIAYN (Eysenbach et al., 2019) and DADS (Sharma et al., 2019a) for unsupervised skill discovery, but their goals are ﬁxed over entire episodes.
Also different from them, our worker goes beyond using individual state features alone for identifying options.
Worker Objective We ﬁrst deﬁne a new MDP
so that for any h , k , and t :
1) s h,k :
= ( s h, 0 , . . .
, s h,k ) , 2) a h,k :=
( a h, 0 , . . . , a h,k ) , 3) r ( s t , a t , s t +1 ) :
= r ( s t , a t , s t +1 ) , and 4) P ( s t +1 | s t , a t ) :
= P ( s t +1 | s t , a t ) .
This new MDP equips the worker with historical state and action information since an option was scheduled.
In the following, we will call pairs { a h,k , s h,k +1 } option sub-trajectories .
The worker policy now takes option sub-trajectories as inputs: a h,k ∼ π φ ( ·| s h,k , a h,k − 1 , u h ) , 0 ≤ k < K , whereas the scheduler policy still operates in the original MDP.
Denote (cid:80) h,k ≡ (cid:80)
H − 1 h =
(cid:80) K − 1 k =0 for simplicity.
The worker objective is deﬁned on this new MDP, to minimize the entropy of the option u h conditioned on the option sub-trajectory { a h,k , s h,k +1 } :
[Equation]
(4) where the expectation is over the current π θ and π φ
but the maximization is only with respect to φ .
Intuitively, the ﬁrst term suggests that the worker is optimized to conﬁdently identify an option given a sub-trajectory.
However, it alone will not guarantee the diversity of options because potentially even very similar sub-trajectories can be mapped to different options if the resulting sub-trajectory space has a very high “resolution”.
As a result, the conditional entropy might not be able to generate useful skills to be exploited by the scheduler for task solving.
To combat this degenerate solution, we add a second term which computes the entropy of the worker policy.
Intuitively, while the worker generates identiﬁable sub-trajectories corresponding to a given option, it should act as randomly as possible to separate sub-trajectories of different options, lowering the “resolution” of the subtrajectory space to encourage diversity.
Because directly estimating the posterior p ( u h | a h,k , s h,k +1 ) is intractable, we approximate it with a parameterized posterior log q ψ ( u h | a h,k , s h,k +1 ) to obtain a lower bound (Barber & Agakov, 2003), where q ψ is a discriminator to be learned.
Then we can maximize this lower bound instead:
[Equation]
Similar to VIC (Gregor et al., 2016) and DIAYN (Eysenbach et al., 2019), the discriminator is trained by maximizing likelihoods of options given sampled sub-trajectories.
The worker is trained via max-entropy RL (Soft Actor-Critic (SAC) (Haarnoja et al., 2018)) with the intrinsic reward r loh,
k +1 :
= log q ψ ( · ) − β log π φ ( · ) .
β is a constant weight which is ﬁxed to 0 .
01 in all our experiments.
Note that there are at least two differences between Eq.
5 and the skill discovery objectives in
VIC (Gregor et al., 2016) and DIAYN (Eysenbach et al., 2019).
First, both VIC and DIAYN assume that a sampled skill will last through an entire episode, and the skill is always sampled at the beginning of an episode.
Thus their skill trajectories “radiate” from the initial state set.
In contrast, our worker policy learns skills that are reset every K steps within an episode, and they can have more diverse semantics depending on the various states visited by the agent.
This is especially helpful for some tasks where new skills need to be discovered after the agent reaches unseen areas in later stages of training.
Second, we extend the discriminator to predict based on option sub-trajectories { a h,k , s h,k +1 } instead of solely on individual states s h,k , and we investigate six possible instantiations of the discriminator.
As observed in the experiments, individual states are actually not the optimal features for identifying options.
Shortsighted Worker
It’s challenging for the worker to accurately predict values over a long horizon, since its rewards are densely computed by a complex nonlinear function q ψ .
Also each option only lasts at most K steps.
Thus we set the discount η for the worker in two shortsighted ways:
Hard : η = 0 if t = hK
and η = otherwise.
Basically this version truncates the temporal correlation gradients) between adjacent options.
The beneﬁt of this might be faster and easier value learning because the value is bootstrapped over at most K steps ( K (cid:28) T ).
Soft : η = 1 − 1 K , which considers rewards of roughly K steps ahead.
The worker policy still needs to take into account the identiﬁcation of future option sub-trajectories, but their importance quickly decays.
We will evaluate both versions and compare their performance in Section 4.1.
3. I NSTANTIATING THE D ISCRIMINATOR
We conduct a comprehensive study of various ways of instantiating the discriminator q ψ in order to compute useful intrinsic rewards for the worker.
Previous work has utilized individual states (Eysenbach et al., 2019; Jabri et al., 2019) or full observation trajectories (Warde-Farley et al., 2019;
Sharma et al., 2019a; Achiam et al., 2018) for skill discrimination.
Because our discriminator takes option sub-trajectories for prediction, in principle we can instantiate our discriminator on any function of an option sub-trajectory.
In this paper, we investigate six sub-trajectory feature extractors f ψ :
Feature extractor
Name
Formulation
Explanation f ψ ( a h,k , s h,k +1 ) =
State
M LP ( s t,k +1 )
Next state alone
Action
M LP ([ s h, 0 , a h,k ])
Action in context
StateDiff
M LP ( s h,k +
− s h,k )
Difference between state pairs
StateAction
M LP ([ a h,k , s h,k +1 ])
Action and next state
StateConcat
M LP ([ s h,k +1 ])
Concatenation of states
ActionConcat
M LP ([ s h, 0 , a h,k ])
Concatenation of actions where the operator [ · ] denotes concatenation and M LP denotes a multilayer perceptron 1 .
Our State feature extractor is most similar to DIAYN (Eysenbach et al., 2019), and StateConcat is similar to (Warde-Farley et al., 2019; Sharma et al., 2019a; Achiam et al., 2018). However we note that unlike these works, the reachability of our option sub-trajectories is determined by the scheduler in the context of hRL.
The other four feature extractors have not been evaluated before.
With the extracted feature, the log-probability of predicting an option is simply computed as the negative squared L norm: log q ψ ( u h | a h,k , s h,k +1 ) = −(cid:107)
f ψ ( a h,k , s h,k +1 ) − u h (cid:107) 22 , through which we implicitly assume the discriminator’s output distribution to be a multi-variate Gaussian with zero means and unit variances.
3. O FF - POLICY T RAINING
The scheduler and worker objectives (Eq. 3 and Eq. 5) are trained jointly.
In principle, on-policy training such as A2C (Clemente et al., 2017) is needed due to the interplay between the scheduler and worker.
However, to reuse training data and improve sample efﬁciency, we employ off-policy training (SAC (Haarnoja et al., 2018)) for both objectives with some modiﬁcations.
Modiﬁed worker objective In practice, the expectation over the scheduler π θ in Eq.
5 is replaced with the expectation over its historical versions.
Speciﬁcally, we sample options
u h from a replay buffer, together with sub-trajectories { a h,k , s h,k +1 } .
This type of data distribution modiﬁcation is conventional in off-policy training (Lillicrap et al., 2016).
Intrinsic reward relabeling We always recompute the rewards in Eq.
5 using the up-to-date discriminator for every update of φ , which can be trivially done without any additional interaction with the environment.
Importance correction
The data in the replay buffer was generated by historical worker policies.
Thus a sampled option sub-trajectory will be outdated under the same option, causing confusion to the scheduler policy.
To resolve this issue, when minimizing the temporal-difference (TD) error between the values of s h, 0 and s h
+1 , 0 for the scheduler, an importance ratio can be multiplied: (cid:81)[-K −k =
-]
[Equation]
s h,k , u h ) .
A similar correction can also be applied to the discriminator loss.
However, in practice we ﬁnd that this ratio has a very high variance and does not help.
Empirically, even without correction our method is able to perform well 2 ; similar observations were made in Nachum et al. ; Fedus et al.
We leave further reduction of off-policy biases to future work.
E XPERIMENTS
Environments We evaluate success rate and sample efﬁciency across two environment suites, as shown in Figure 2.
Important details are presented here with more information in appendix Section
B. The ﬁrst suite consists of two 7-DOF reaching and pushing environments evaluated in Chua et al.
They both emulate a one-armed PR2 robot.
The tasks have sparse rewards: the agent gets a reward of 0 at every timestep where the goal is not achieved, and 1 upon achieved.
There is also a small L 2 action penalty applied.
In 7-DOF R EACHER , the goal is achieved when the gripper reaches a 3D goal position.
In 7-DOF P USHER , the goal is to push an object to a 3D goal position.
A success of an episode is deﬁned to be if the goal is achieved at the ﬁnal step of the episode.
We also propose another suite of environments called S OCIAL R OBOT .
We construct two sparse reward robotic navigation and manipulation tasks, G OAL T ASK and K ICK B ALL .
In G OAL T ASK , the agent gets a reward of 1 when it successfully navigates to a goal, -1 if the goal becomes too far, -0.
every time it is too close to a distractor object, and 0 otherwise.
In K ICK B ALL , the agent receives a reward of 1 for successfully pushing a ball into the goal, 0 otherwise, and has the same distractor object penalty.
At the beginning of each episode, both the agent and the ball are spawned randomly.
Both environments contain a small L 2 action penalty, and terminate an episode upon a success.
Comparison methods One baseline algorithm for comparison is standard SAC (Haarnoja et al., 2018), the building block of our hierarchical method.
To verify if our worker policy can just be re- 3 Num Timesteps 1e 0. S u cc e ss Pusher Success Plot   3 Num Timesteps 1e 0.
S u cc e ss
Reacher Success Plot     5 Num Timesteps 1e 0.
S u cc e ss
GoalTask Success Plot     5 Num Timesteps 1e 0.0 0.
S u cc e ss
KickBall Success Plot
ActionStateActionActionConcatStateConcatStateDiffState placed with a na¨ıve action repetition strategy, we compare with SAC+ActRepeat with an action repetition for the same length K as our option interval.
We also compare against HIRO (Nachum et al., 2018), a data efﬁcient hierarchical method with importance-based option relabeling, and HiPPO (Li et al., 2020) which trains the lower level and higher level policies together with one uniﬁed
PPO-based objective.
Both are state-of-the-art hierarchical methods proposed to solve sparse reward tasks.
Similar to our work, HiPPO makes no assumptions about options, however it utilizes a discrete option space and its options are trained with environment reward.
A comprehensive hyperparameter search is performed for every method, with a far greater search space over HiPPO and HIRO than our method HIDIO to ensure maximum fairness in comparison; details are presented in Appendix C.
For every evaluation point during training, we evaluate the
4. W ORKER D ESIGN C HOICES
We ask and answer questions about the design choices in HIDIO speciﬁc to the worker policy π φ .
What sub-trajectory feature results in good option discovery?
We evaluate all six features proposed in Section 3.3 in all four environments.
These features are selected to evaluate how different types of subtrajectory information affect option discovery and ﬁnal performance.
They encompass varying types of both local and global subtrajectory information.
We plot comparisons of sample efﬁciency and ﬁnal performance in Figure 3 across all environments (solid lines), ﬁnding that Action , StateAction , and StateDiff are generally among the top performers.
StateAction includes the current action and next state, encouraging π φ to differentiate its options with different actions even at similar states.
Similarly, Action includes the option initial state and current action, encouraging option diversity by differentiating between actions conditioned on initial states.
Meanwhile StateDiff simply encodes the difference between the next and current state, encouraging π φ to produce options with different state changes at each step.
How do soft shortsighted workers ( Soft ) compare against hard shortsighted workers ( Hard )?
In Figure 3, we plot all features with Soft in dotted lines.
We can see that in general there is not much difference in performance between Hard and Soft except some extra instability of Soft in
R EACHER regarding the StateConcat and State features.
One reason of this similar general performance could be that since our options are very short-term in Hard , the scheduler policy has the opportunity of switching to a good option before the current one leads to bad consequences.
In a few cases, Hard seems better learned, perhaps due to an easier value bootstrapping for the worker.
4. C OMPARISON RESULTS
We compare our three best sub-trajectory features of Hard , in Section 4.1, against the SAC baselines and hierarchical RL methods across all four environments in Figure 4.
Generally we see that HIDIO (solid lines) achieves greater ﬁnal performance with superior sample efﬁciency than the compared methods.
Both SAC and SAC+ActRepeat perform poorly across all environments, and all baseline methods perform signiﬁcantly worse than HIDIO on R EACHER , G OAL T ASK , and K ICK B ALL . 3 Num Timesteps 1e 0. S u cc e ss
Pusher Success Plot   3 Num Timesteps 1e 0. S u cc e ss Reacher Success Plot     5 Num Timesteps 1e 0. S u cc e ss
GoalTask Success Plot     5 Num Timesteps 1e 0.0 0.
S u cc e ss
KickBall Success Plot
SACSAC+ActRepeat
HiPPOHIROActionStateActionStateDiff
In P USHER , HiPPO displays competitive performance, rapidly improving from the start.
However, all three HIDIO instantiations achieve nearly 100% success rates while HiPPO is unable to do so.
Furthermore, HIRO and SAC+ActRepeat take much longer to start performing well, but never achieve similar success rates as HIDIO.
HIDIO is able to solve R EACHER while HiPPO achieves only about a 60% success rate at best.
Meanwhile, HIRO, SAC+ActRepeat, and SAC are unstable or non-competitive.
R EACHER is a difﬁcult exploration problem as the arm starts far from the goal position, and we see that HIDIO’s automatically discovered options ease exploration for the higher level policy to consistently reach the goal.
HIDIO performs well on G OAL T ASK , achieving 60-80% success rates, while the task is too challenging for every other method.
In K ICK B ALL , the most challenging task, HIDIO achieves 30-40% success rates while every other method fails to learn again, highlighting the need for the intrinsic option discovery of HIDIO in these environments.
In summary, HIDIO demonstrates greater sample efﬁciency and ﬁnal reward gains over all other baseline methods.
Regular RL (SAC) fails on all four environments, and while HiPPO is a strong baseline on P USHER and R EACHER , it is still outperformed in both by HIDIO.
All other methods fail on G OAL T ASK and K ICK B ALL , while HIDIO is able to learn and perform better in both.
This demonstrates the importance of the intrinsic, short-term option discovery employed by HIDIO, where the options are diverse enough to be useful for both exploration and task completion.
4. J OINT π φ AND π θ TRAINING
[Figure]
We ask the next question: is jointly training π θ and π φ necessary?
To answer this, we compare HIDIO against a pre-training baseline where we ﬁrst pretrain π φ , with uniformly sampled options u for a portion ρ of total numbers of time steps, and then ﬁx π φ while training π θ for the remaining (1 − ρ ) time steps.
This is essentially using pre-trained skills for downstream higher-level tasks as demonstrated in DIAYN (Eysenbach et al., 2019).
We conduct this experiment with the StateAction feature extractor on both K ICK B ALL and P USHER , with ρ = { 116 , 18 , 14 } .
The results are shown in Figure 6.
We can see that in P USHER , fewer pretraining timesteps are more sample efﬁcient, as the environment is simple and options can be learned from a small amount of samples.
The nature of P USHER also only requires stationary options that can be learned independent of the scheduler policy evolution.
Nevertheless, the pretraining baselines seem less stable.
In K ICK B ALL , the optimal pretraining baseline is on ρ = 1 8 of the total timesteps.
However without the joint training scheme of HIDIO, the learned skills are unable to be used as efﬁciently for the difﬁcult obstacle avoidance, navigation, and ball manipulation subtasks required for performing well.
4. O PTION B EHAVIORS
Finally, since options discovered by HIDIO in our sparse reward environments help it achieve superior performance, we ask, what do useful options look like?
To answer this question, after training, we sample options from the scheduler π θ to visualize their behaviors in different environments in
Figure 5.
For each sampled option u , we ﬁx it until the end of an episode and use the worker π φ to output actions given u .
We can see that the options learned by HIDIO are low-level navigation and manipulation skills useful for the respective environments.
R ELATED W ORK
Hierarchical RL Much of the previous work in hRL makes assumptions about the task structure and/or the skills needed to solve the task. While obtaining promising results under speciﬁc settings, they may have difﬁculties with different scenarios.
For example, SAC-X (Riedmiller et al., 2018) requires manually designing auxiliary subtasks as skills to solve a given downstream task.
SNN4HRL (Florensa et al., 2016) is geared towards tasks with pre-training and downstream components.
Lee et al.
(2019; 2020) learns to modulate or compose given primitive skills that are customized for their particular robotics tasks.
Ghavamzadeh & Mahadevan (2003) and Sohn et al.
(2018) operate under the assumption that tasks can be manually decomposed into subtasks.
The feudal reinforcement learning proposal (Dayan & Hinton, 1993) has inspired another line of works (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2019) which make higher-level manager policies output goals for lower-level worker policies to achieve.
Usually the goal space is a subspace of the state space or deﬁned according to the task so that lower-level rewards are easy to compute.
This requirement of manually “grounding” goals in the environment poses generalization challenges for tasks that cannot be decomposed into state or goal-reaching.
Our method HIDIO does not make any assumptions about the speciﬁc task at hand.
It follows from the options framework (Sutton et al., 1999), which has recently been applied to continuous domains (Bacon et al., 2017), spawning a diverse set of recent hierarchical options methods (Bagaria & Konidaris, 2020; Klissarov et al., 2017; Riemer et al., 2018; Tiwari & Thomas, 2019; Jain et al., 2018).
HIDIO automatically learns intrinsic options that avoids having explicit initiation or termination policies dependent on the task at hand.
HiPPO (Li et al., 2020), like HIDIO, also makes no major assumptions about the task, leading to a more general hRL algorithm.
Self-supervised skill discovery There are also plenty of prior works which attempt to learn skills without task reward.
DIAYN (Eysenbach et al., 2019) and VIC (Gregor et al., 2016) learn skills by maximizing the mutual information between trajectory states and their corresponding skills.
VALOR (Achiam et al., 2018) learns options by maximizing the probability of options given their resulting observation trajectory.
DADS (Sharma et al., 2019a) learns skills that are predictable by dynamics models.
DISCERN (Warde-Farley et al., 2019) maximizes the mutual information between goal and option termination states to learn a goal-conditioned reward function.
Eysenbach et al.
Achiam et al.
; Sharma et al.
(2019a) demonstrate these pre-trained skills to be useful for hRL.
However, usually these methods pre-train skills in an initial stage that is separate from downstream task learning.
Little work has been done to integrate these skill discovery processes into hRL.
C ONCLUSION
Under the context of solving difﬁcult sparse reward tasks, we propose a new hierarchical reinforcement learning method, HIDIO, which can learn task-agnostic options in a self-supervised manner and simultaneously learn to utilize them to solve tasks.
We evaluate several different instantiations of the discriminator of HIDIO for providing intrinsic rewards for training the lower-level worker policy.
We demonstrate the effectiveness of HIDIO compared against other reinforcement learning methods in achieving high rewards with better sample efﬁciency across a variety of robotic naviga-
tion
and manipulation tasks
A P SEUDO CODE FOR HIDIO
Input:
T
Episode length
M
Batches per iteration π φ ( a h,k
| s h,k , u h )
Worker B
Batch size α
Learning rate q ψ ( u h | a h,k , s h,k +1 )
Discriminator K
Option interval
P ( s t +1 | s t , a t )
Environment dynamics π θ ( u h | s h, 0 )
Scheduler
Output: Learned parameters θ , φ , and ψ .
Initialize: Random model parameters θ , φ , and ψ ; empty replay buffers D scheduler and D worker .
while termination not met do /* Data collection */ for scheduler step h =
TK − 1 do
Sample an option u h ∼ π θ ( ·| s h, 0 ) .
for worker step k =
..K − 1 do
Sample an action a h,k ∼ π φ ( ·| s h,k , u h ) .
Step through the environment s h,k +1 ∼ P ( ·| s h,k , a h,k ) .
a h,k , s h,k +1 ← [ a h,k − 1 , a h,k ] , [ s h,k , s h,k +
D worker ← D worker ∪ ( u h , a h,k , s h,k +1 ) end
R h ← (cid:80) K − 1 k =
r
( s h,k , a h,k , s h,k +1 )
D scheduler ← D scheduler ∪ ( s h, 0 , u h , s h +1 , 0 , R h ) end /*
Model training */ for batch m =
M − 1 do /*
Scheduler training */
Uniformly sample transitions { ( s t , u t , s t +1 ) }
Bb =1 ∼ D scheduler .
Compute gradient ∆ θ according to Eq. 3.
Update models θ ← θ + α ∆ θ .
/* Worker training */
Uniformly sample transitions { ( u h , a h,k , s h,k +1 ) }
Bb =1 ∼ D worker .
Compute intrinsic rewards r loh,k ← q ψ u
| a
,k , s
,k +1 ) .
Compute gradient ∆ ψ and ∆ φ according to Eq.
Update models φ ← φ + α ∆ φ and ψ ← ψ + α ∆ ψ .
end end
B M ORE ENVIRONMENT DETAILS B.0. P USHER AND R EACHER
These environments both have a time horizon of 100 with no early termination: each episode always runs for 100 steps regardless of goal achievement.
For both, a success is when the agent achieves the goal at the ﬁnal step of an episode.
In R EACHER , observations are 17-dimensional, including the positions, angles, and velocities of the robot arm, and in P USHER observations also include the 3D object position.
Both include the goal position in the observation space.
Actions are 7-dimensional vectors for joint velocity control.
The action range is [ − 20 , 20] in R EACHER and [ − 2 , 2] in P USHER .
There is an action penalty in both environments: at every timestep the squared L 2 norm of the agent action is subtracted from the reward.
In P USHER , this penalty is multiplied by a coefﬁcient of 0 .
In R EACHER , it’s multiplied by 0 . 0001 .
B.0. G OAL T ASK AND K ICK B ALL
For both S OCIAL R OBOT environments, an episode terminates early when either a success is reached or the goal is out of range.
For each episode, the positions of all objects (including the agent) are randomly picked.
Observations are 18-dimensional, including ego-centric distances and directions from the agent to different objects.
In K ICK B ALL , the agent receives a reward of 1 for successfully pushing a ball into the goal (episode termination) and 0 otherwise.
At the beginning of each episode, the ball is spawned randomly inside the neighborhood of the agent.
Three distractor objects are included on the ground to increase task difﬁculty.
In G OAL T ASK , the number of distractor objects increases to 5.
Both environments contain a small L 2 action penalty: at every time step the squared
L 2 norm of the agent action, multiplied by 0 .
01 , is subtracted from the reward.
G OAL T ASK has a time horizon of 100 steps, while K ICK B ALL ’s horizon is 200.
Observations are 30-dimensional, including absolute poses and velocities of the goal, the ball, and the agent.
Both G OAL T ASK and
K ICK B ALL use the same navigation robot P IONEER 2 DX which has 2-dimensional actions that control the angular velocities (scaled to [ − 1 , 1] ) of the two wheels.
C H YPERPARAMETERS
To ensure a fair comparison across all methods, we perform a hyperparameter search over the following values for each algorithm and suite of environments.
C. P USHER AND R EACHER
Shared hyperparameters across all methods are listed below (where applicable, and except when overridden by hyperparameters listed for each individual method).
For all methods, we take the hyperparameters that perform best across 3 random seeds in terms of the area under the evaluation success curve (AUC) in the P USHER environment.
• Number of parallel actors/environments per rollout:• Steps per episode:• Batch size:• Learning rate: 10 − 4 for all network modules • Policy/Q network hidden layers: (256, 256, 256) with ReLU non-linearities • Polyak averaging coefﬁcient for target Q: 0.
• Target Q update interval (training iterations):• Training batches per iteration:• Episodes per evaluation:• Initial environment steps for data collection before training:
Rollouts and training iterations are performed alternatively, one after the other.
The rollout length searched below refers to how many time steps in each environment are taken per rollout/training iteration, effectively controlling the ratio of gradient steps to environment steps.
A smaller rollout length corresponds to a higher ratio.
This ratio is also searched over for H I PPO and HIRO.
Other hyperparameters searched separately for each algorithm are listed below, and selected ones are bolded .
C.1. SAC
• Target entropy min prob ∆ 3 : { 0.1, 0.2 , 0.3 } • Replay buffer length per parallel actor: { 50000, 200000 } • Rollout Length: { 12, 25, 50 , 100 }
C.1. SAC W / A CTION R EPETITION
• Action repetition length 4 :• Rollout Length: { 4 , 8, 16, 33 }
Other hyperparameters are kept the same as the optimal SAC ones.
C.1. HIDIO
The hyperparameters of HIDIO were mostly heuristically chosen due to the hyperparameter search space being too large.
• Latent option u vector size: { 8 , 12 } • Policy/Q network hidden layers for π φ : (128, 128, 128) • Steps per option ( K ):• π φ has a ﬁxed entropy coefﬁcient α of 0.01.
Target entropy min prob ∆ for π θ is 0.2.
• Discriminator network hidden layers: (64, 64) • Replay buffer length per parallel actor: { 50000, 200000 } • Rollout Length: { 25 , 50, 100 } C
HIRO
• Steps per option: { 3 , 5, 8 } • Replay buffer size (total): { 500000, 2000000 } • Meta action space (actions are relative, e.g. ,
• Policy stddev noise: { 0.1 , 0.3, 0.5 } • Number of gradient updates per training iteration: { 100, 200 , 400 } C.1.
H I PPO
For most hyperparameters, the search ranges chosen were derived after discussion with the ﬁrst author of HiPPO.
• Learning rate: 3 × 10 −• Policy network hidden layers: (256, 256) • Skill selection network hidden layers: { (32, 32), (128, 64) } • Latent skill vector size: { 5, 10 , 15 } • PPO clipping parameter: { 0.05, 0.
} • Time commitment range: { (2, 5), (3, 7) } • Policy training steps per epoch: { 25 , 50, 100 }
C. S OCIAL R OBOT
For all methods, we select the hyperparameters with the best area under the evaluation success curve (AUC) in the K ICK B ALL environment, and apply them to both K ICK B ALL and G OAL T ASK .
The shared hyperparameters are as follows (if applicable to the algorithm, and except when overridden by the respective algorithm’s list of hyperparameters):
• Number of parallel actors/environments per rollout:• Steps per episode: 100 (G OAL T ASK ), 200 (K ICK B ALL ) • Batch size:• Learning rate: 5 × 10 − 4 for all network modules • Policy/Q network hidden layers: (256, 256, 256) with ReLU non
-linearities • Polyak averaging coefﬁcient for target Q: 0.
• Target Q update interval (training iterations):• Training batches per iteration:
• Episodes per evaluation:• Evaluation interval (training iterations):• Initial environment steps for data collection before training:
The training terminology here generally follows section C.1.
C.2. SAC
• Target entropy min prob ∆ : { 0.1, 0.2 , 0.3 } • Replay buffer length per parallel actor: { 20000, 100000 } • Rollout length: { 12, 25, 50 , 100 }
C
SAC W / A CTION R EPETITION
• Action repetition length 5 :• Rollout Length: { 4, 8 , 16, 33 }
Other hyperparameters are kept the same as the optimal SAC ones.
C.2. HIDIO
Due to the large hyperparameter search space, we only search over the option vector size and rollout length, and select everything else heuristically.
• Latent option u vector size: { 4 , 6 } • Policy/Q network hidden layers for π φ (128, 128, 128) • Steps per option ( K ):• π φ has a ﬁxed entropy coefﬁcient α of 0.01.
Target entropy min prob ∆ for π θ is 0.2.
• Discriminator network hidden layers: (32, 32) • Replay buffer length per parallel actor:• Rollout Length: { 50, 100 }
C
HIRO
• Learning rate: 3 × 10 −• Steps per option: { 3 , 5, 8 } • Replay buffer size (total): { 500000 , 2000000 } • Meta action space (actions are relative, e
g. , meta-action is current obs + action ): – G OAL T ASK : (-np.ones(obs space) * 2, np.ones(obs space) * 2) – K ICK B ALL : (-np.ones(obs space - goal space) * 2, np.ones(obs space - goal space)
* 2) (because the goal position is given but will not change in the observation space)
• Policy stddev noise { 0.1 , 0.3, 0.5 } • Number of gradient updates per training iteration: { 100, 200, 400 }
C.
H
PPO
• Learning rate: 3 × 10 −• Policy network hidden layers: { (64, 64) , (256, 256) } • Skill selection network hidden layers: { (32, 32) , (128, 64) }
• Latent skill vector size: { 4, 8 } • PPO clipping parameter: { 0.05 , 0.
} • Time commitment range: { (2, 5) , (3, 7) } • Policy training steps per epoch: { 25, 50 , 100 }