S UB -P OLICY A DAPTATION FOR H IERARCHICAL R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards.
Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task.
Leaving the skills ﬁxed can lead to signiﬁcant sub-optimality in the transfer setting.
In this work, we propose a novel algorithm to discover a set of skills and continuously adapt them along with the higher level even when training on a new task.
Our main contributions are two-fold.
First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efﬁciently train all levels of the hierarchy jointly.
Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes.
Code and videos are available here
I NTRODUCTION
Reinforcement learning (RL) has made great progress in a variety of domains, from playing games such as Pong and Go (Mnih et al., 2015; Silver et al., 2017) to automating robotic locomotion (Schulman et al., 2015; Heess et al., 2017), dexterous manipulation (Florensa et al., 2017b; OpenAI et al., 2018), and perception (Nair et al., 2018; Florensa et al., 2018).
Yet, most work in RL is still learning from scratch when faced with a new problem.
This is particularly inefﬁcient when tackling multiple related tasks that are hard to solve due to sparse rewards or long horizons.
A promising technique to overcome this limitation is hierarchical reinforcement learning (HRL) (Sutton et al., 1999).
In this paradigm, policies have several modules of abstraction, allowing to reuse subsets of the modules.
The most common case consists of temporal hierarchies (Precup, 2000;
Dayan & Hinton, 1993), where a higher-level policy (manager) takes actions at a lower frequency, and its actions condition the behavior of some lower level skills or sub-policies.
When transferring knowledge to a new task, most prior works ﬁx the skills and train a new manager on top.
Despite having a clear beneﬁt in kick-starting the learning in the new task, having ﬁxed skills can considerably cap the ﬁnal performance on the new task (Florensa et al., 2017a).
Little work has been done on adapting pre-trained sub-policies to be optimal for a new task.
In this paper, we develop a new framework for simultaneously adapting all levels of temporal hierarchies.
First, we derive an efﬁcient approximated hierarchical policy gradient.
The key insight is that, despite the decisions of the manager being unobserved latent variables from the point of view of the Markovian environment, from the perspective of the sub-policies they can be considered as part of the observation.
We show that this provides a decoupling of the manager and sub-policy gradients, which greatly simpliﬁes the computation in a principled way.
It also theoretically justiﬁes a technique used in other prior works (Frans et al., 2018).
Second, we introduce a sub-policy speciﬁc baseline for our hierarchical policy gradient.
We prove that this baseline is unbiased, and our experiments reveal faster convergence, suggesting efﬁcient gradient variance reduction.
Then we introduce a more stable way of using this gradient, Hierarchical Proximal Policy Optimization (HiPPO).
This method helps us take more conservative steps in our policy space (Schulman et al., 2017), critical in hierarchies because of the interdependence of each layer.
Results show that HiPPO is highly efﬁcient when learning from scratch, i.e. adapting randomly initialized skills, and when adapting pretrained skills on a new task.
Finally, we evaluate the beneﬁt of randomizing the time-commitment of the sub-policies, and show it helps both in terms of ﬁnal performance and zero-shot adaptation on similar tasks.
P RELIMINARIES AND PROBLEM STATEMENT
We deﬁne a discrete-time ﬁnitehorizon discounted Markov decision process (MDP) by a tuple M =
( S , A , P , r, ρ 0 , γ, H ) , where S is a state set, A is an action set, P :
S ×
A × S → R + is the transition probability distribution, γ ∈ [0 , 1] is a discount factor, and H the horizon.
Our objective is to ﬁnd a stochastic policy π θ that maximizes the expected discounted return
We use τ = ( s 0 , a 0 , ..., ) to denote the entire state-action trajectory, where s 0 ∼ ρ 0 ( s 0 ) , a t ∼ π θ ( a t
| s t ) , and s t +1 ∼ P ( s t +1 | s t , a t ) .
In this work, we propose a method to learn a hierarchical policy and efﬁciently adapt all the levels in the hierarchy to perform a new task.
We study a quite extended type of hierarchical policies composed of a higher level, or manager π θ h
( z t | s t ) , and a lower level, or sub-
policy π θ l ( a t (cid:48) | z t , s t (cid:48) ) .
The higher level does not take actions in the environment directly, but rather outputs a command, or latent variable z t ∈ Z , that conditions the behavior of the lower level.
We focus on the common case where Z = Z n making the manager choose among n sub-policies, or skills, to execute.
The manager typically operates at a lower frequency than the sub-policies, only observing the environment every p time-steps.
When the manager receives a new observation, it decides which low level policy to commit to for p environment steps by the means of a latent code z .
Figure 1 depicts this framework where the high level frequency p is a random variable, which is one of the contribution of this paper as described in Section 4.4.
Note that the class of hierarchical policies we work with is more restrictive than others like the Option framework, where notably the time-commitment is also decided by the policy.
Nevertheless, we show that this loss in policy expressivity acts as a regularizer and does not prevent our algorithm from surpassing other state-of-the art methods.
R ELATED W ORK
There has been growing interest in HRL for the past few decades (Sutton et al., 1999; Precup, 2000), but only recently has it been applied to high-dimensional continuous domains as we do in this work (Kulkarni et al., 2016; Daniel et al., 2016).
To obtain the lower level policies, or skills, most methods exploit some additional assumptions, like access to demonstrations (Le et al., 2018; Merel et al., 2019; Ranchod et al., 2015; Sharma et al., 2018), policy sketches (Andreas et al., 2017), or task decomposition into sub-tasks (Ghavamzadeh & Mahadevan, 2003; Sohn et al., 2018).
Other methods use a different reward for the lower level, often constraining it to be a “goal reacher” policy, where the signal from the higher level is the goal to reach (Nachum et al., 2018; Levy et al., 2019;
Vezhnevets et al., 2017).
These methods are very promising for state-reaching tasks, but might require access to goal-reaching reward systems not deﬁned in the original MDP, and are more limited when training on tasks beyond state-reaching.
Our method does not require any additional supervision, and the obtained skills are not constrained to be goal-reaching.
When transferring skills to a new environment, most HRL methods keep them ﬁxed and simply train a new higher-level on top (Hausman et al., 2018; Heess et al., 2016).
Other work allows for building on previous skills by constantly supplementing the set of skills with new ones (Shu et al., 2018), but they require a hand-deﬁned curriculum of tasks, and the previous skills are never ﬁne-tuned.
Our algorithm allows for seamless adaptation of the skills, showing no trade-off between leveraging the power of the hierarchy and the ﬁnal performance in a new task.
Other methods use invertible functions as skills (Haarnoja et al., 2018), and therefore a ﬁxed skill can be fully over-written when a new layer of hierarchy is added on top.
This kind of “ﬁne-tuning” is promising, although similar to other works (Peng et al., 2019), they do not apply it to temporally extended skills as we do here.
One of the most general frameworks to deﬁne temporally extended hierarchies is the options framework (Sutton et al., 1999), and it has recently been applied to continuous state spaces (Bacon et al., 2017).
One of the most delicate parts of this formulation is the termination policy, and it requires several regularizers to avoid skill collapse (Harb et al., 2017; Vezhnevets et al., 2016).
This modiﬁcation of the objective may be difﬁcult to tune and affects the ﬁnal performance.
Instead of adding such penalties, we propose having skills of a random length, not controlled by the agent during training of the skills.
The beneﬁt is two-fold: no termination policy to train, and more stable skills that transfer better.
Furthermore, these works only used discrete action MDPs.
We lift this assumption, and show good performance of our algorithm in complex locomotion tasks.
There are other algorithms recently proposed that go in the same direction, but we found them more complex, less principled (their per-action marginalization cannot capture well the temporal correlation within each option), and without available code or evidence of outperforming non-hierarchical methods (Smith et al., 2018).
The closest work to ours in terms of ﬁnal algorithm structure is the one proposed by Frans et al.
Their method can be included in our framework, and hence beneﬁts from our new theoretical insights.
We introduce a modiﬁcation that is shown to be highly beneﬁcial: the random timecommitment mentioned above, and ﬁnd that our method can learn in difﬁcult environments without their complicated training scheme.
E FFICIENT H IERARCHICAL P OLICY G RADIENTS
When using a hierarchical policy, the intermediate decision taken by the higher level is not directly applied in the environment.
Therefore, technically it should not be incorporated into the trajectory description as an observed variable, like the actions.
This makes the policy gradient considerably harder to compute.
In this section we ﬁrst prove that, under mild assumptions, the hierarchical policy gradient can be accurately approximated without needing to marginalize over this latent variable.
Then, we derive an unbiased baseline for the policy gradient that can reduce the variance of its estimate.
Finally, with these ﬁndings, we present our method, Hierarchical Proximal Policy
Optimization (HiPPO), an on-policy algorithm for hierarchical policies, allowing learning at all levels of the policy jointly and preventing sub-policy collapse.
4. A PPROXIMATE H IERARCHICAL P OLICY G RADIENT
Policy gradient algorithms are based on the likelihood ratio trick (Williams, 1992) to estimate the gradient of returns with respect to the policy parameters as
[Equation]
In a temporal hierarchy, a hierarchical policy with a manager π θ h
( z t | s t ) selects every p time-steps one of n sub-policies to execute.
These sub-policies, indexed by z ∈ Z n , can be represented as a single conditional probability distribution over actions π θ l ( a t | z t , s t ) .
This allows us to not only use a given set of sub-policies, but also leverage skills learned with Stochastic Neural Networks (SNNs) (Florensa et al., 2017a).
Under this framework, the probability of a trajectory τ =
( s 0 , a 0 , s 1 , . .
, s H ) can be written as
[Equation]
The mixture action distribution, which presents itself as an additional summation over skills, prevents additive factorization when taking the logarithm, as from Eq.
1 to 2.
This can yield numerical instabilities due to the product of the p sub-policy probabilities.
For instance, in the case where all the skills are distinguishable all the sub-policies’ probabilities but one will have small values, resulting in an exponentially small value.
In the following Lemma, we derive an approximation of the policy gradient, whose error tends to zero as the skills become more diverse, and draw insights on the interplay of the manager actions.
Lemma 1.
If the skills are sufﬁciently differentiated, then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability.
Let π
θ h
( z | s ) and π θ l ( a | s, z ) be Lipschitz functions w.r.t.
their parameters, and assume that 0 < π θ l ( a | s, z j ) < (cid:15) ∀ j (cid:54) =
kp , then
[Equation]
Proof.
See Appendix.
Our assumption can be seen as having diverse skills.
Namely, for each action there is just one sub-policy that gives it high probability.
In this case, the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability.
Many algorithms to extract lowerlevel skills are based on promoting diversity among the skills (Florensa et al., 2017a;
Eysenbach et al., 2019), therefore usually satisfying our assumption.
We further analyze how well this assumption holds in our experiments section and Table 2.
4. U NBIASED S UB -P OLICY B ASELINE
The policy gradient estimate obtained when applying the log-likelihood ratio trick as derived above is known to have large variance.
A very common approach to mitigate this issue without biasing the estimate is to subtract a baseline from the returns (Peters & Schaal, 2008).
It is well known that such baselines can be made state-dependent without incurring any bias.
However, it is still unclear how to formulate a baseline for all the levels in a hierarchical policy, since an action dependent baseline does introduce bias in the gradient (Tucker et al., 2018).
It has been recently proposed to use latent-conditioned baselines (Weber et al., 2019).
Here we go further and prove that, under the assumptions of Lemma 1, we can formulate an unbiased latent dependent baseline for the approximate gradient (Eq. 5).
Lemma 2.
For any functions b h : S → R and b l : S ×
Z
→ R we have:
[Equation]
Proof.
See Appendix.
Now we apply Lemma 1 and Lemma 2 to Eq. 1.
By using the corresponding value functions as the function baseline, the return can be replaced by the Advantage function
A ( s kp , z kp ) (see details in
Schulman et al.
and we obtain the following approximate policy gradient expression:
[Equation]
This hierarchical policy gradient estimate can have lower variance than without baselines, but using it for policy optimization through stochastic gradient descent still yields an unstable algorithm.
In the next section, we further improve the stability and sample efﬁciency of the policy optimization by incorporating techniques from Proximal Policy Optimization (Schulman et al., 2017).
4. H IERARCHICAL P ROXIMAL P OLICY O PTIMIZATION
Using an appropriate step size in policy space is critical for stable policy learning.
Modifying the policy parameters in some directions may have a minimal impact on the distribution over actions, whereas small changes in other directions might change its behavior drastically and hurt training efﬁciency (Kakade, 2002).
Trust region policy optimization (TRPO) uses a constraint on the KLdivergence between the old policy and the new policy to prevent this issue (Schulman et al., 2015).
1: Input: skills π θ l ( a | s, z ) , manager π θ h
( z | s ) , timecommitment bounds P min and P max , horizon H 2: Reset environment: s 0 ∼ ρ 0 , t = 0 .
3: while t < H do 4:
Sample time-commitment p ∼ Cat ([ P min , P max ]) 5:
Sample skill z
t ∼ π θ h ( ·| s t ) 6: for t (cid:48)
=
t . . .
( t + p ) do 7:
Sample action a t (cid:48) ∼ π θ l ( ·| s t (cid:48) , z t ) 8:
Observe new state s t (cid:48)
+1 and reward r t (cid:48) 9: end for 10: t ← t + p 11: end while 12: Output: ( s 0 , z 0 , a 0 , s 1 , a 1 , . . .
, s H , z H , a H , s H +1 )
Algorithm
1: Input: skills π θ l ( a | s, z ) , manager π θ h
( z | s ) , horizon H , learning rate α 2: while not done do 3: for actor
= 1, 2, ..., N do 4:
Obtain trajectory with
HiPPO
Rollout 5:
Estimate advantages ˆ A ( a t (cid:48) , s t (cid:48) , z t ) and ˆ A ( z t , s t ) 6: end for 7: θ ← θ + α ∇ θ L CLIPHiPPO ( θ ) 8: end while
Unfortunately, hierarchical policies are generally represented by complex distributions without closed form expressions for the KL-divergence.
Therefore, to improve the stability of our hierarchical policy gradient we turn towards Proximal Policy Optimization (PPO) (Schulman et al., 2017).
PPO is a more ﬂexible and compute-efﬁcient algorithm.
In a nutshell, it replaces the KL-divergence constraint with a cost function that achieves the same trust region beneﬁts, but only requires the computation of the likelihood.
Letting w t ( θ ) =
π θ ( a t | s t [Equation] π [Equation]
( a t | s t ) , the PPO objective is:
[Equation]
Since the likelihood ratio w t ( θ ) comes from importance sampling, we can adapt our approximated hierarchical policy gradient with the same approach.
Letting w h,kp ( θ ) =
[Equation]
s [Equation] π
[Equation]
and w l,t ( θ ) = [Equation] θl,old
[Equation]
| s t [Equation] kp ) , and using the super-index clip to denote the clipped objective version, we obtain the new surrogate objective:
[Equation]
We call this algorithm Hierarchical Proximal Policy Optimization (HiPPO).
Next, we introduce a critical additions: a switching of the time-commitment between skills.
4. V ARYING T IME - COMMITMENT
Most hierarchical methods either consider a ﬁxed time-commitment to the lower level skills (Florensa et al., 2017a; Frans et al., 2018), or implement the complex options framework (Precup, 2000; Bacon et al., 2017).
In this work we propose an in-between, where the time-commitment to the skills is a random variable sampled from a ﬁxed distribution Categorical ( T min , T max ) just before the manager takes a decision.
This modiﬁcation does not hinder ﬁnal performance, and we show it improves zero-shot adaptation to a new task.
This approach to sampling rollouts is detailed in
Algorithm 1.
The full algorithm is detailed in Algorithm 2.
E XPERIMENTS
We designed our experiments to answer the following questions: 1) How does HiPPO compare against a ﬂat policy when learning from scratch? 2) Does it lead to policies more robust to environment changes?
3) How well does it adapt already learned skills?
and 4) Does our skill diversity assumption hold in practice?
[Figure] [Figure]
5. T ASKS
We evaluate our approach on a variety of robotic locomotion and navigation tasks.
The Block environments, depicted in Fig.
2a-2b, have walls of random heights at regular intervals, and the objective is to learn a gait for the Hopper and Half-Cheetah robots to jump over them.
The agents observe the height of the wall ahead and their proprioceptive information (joint positions and velocities), receiving a reward of +1 for each wall cleared.
The Gather environments, described by
Duan et al.
(2016), require agents to collect apples (green balls, +1 reward) while avoiding bombs (red balls, -1 reward).
The only available perception beyond proprioception is through a LIDAR-type sensor indicating at what distance are the objects in different directions, and their type, as depicted in the bottom left corner of Fig.
2c-2d.
This is challenging hierarchical task with sparse rewards that requires simultaneously learning perception, locomotion, and higher-level planning capabilities.
We use the Snake and Ant robots in Gather.
Details for all robotic agents are provided in Appendix B.
5. L EARNING FROM S CRATCH AND T IME -C OMMITMENT
In this section, we study the beneﬁt of using our HiPPO algorithm instead of standard PPO on a ﬂat policy (Schulman et al., 2017).
The results, reported in Figure 3, demonstrate that training from scratch with HiPPO leads to faster learning and better performance than ﬂat PPO.
Furthermore, we show that the beneﬁt of HiPPO does not just come from having temporally correlated exploration:
PPO with action repeat converges at a lower performance than our method.
HiPPO leverages the time-commitment more efﬁciently, as suggested by the poor performance of the ablation where we set p = 1 , when the manager takes an action every environment step as well.
Finally, Figure 4 shows the effectiveness of using the presented skill-dependent baseline.
5. C OMPARISON TO O THER M ETHODS
We compare HiPPO to current state-of-the-art hierarchical methods.
First, we evaluate HIRO (Nachum et al., 2018), an off-policy RL method based on training a goal-reaching lower level policy.
Fig.
5 shows that HIRO achieves poor performance on our tasks.
As further detailed in Appendix
D, this algorithm is sensitive to access to ground-truth information, like the exact ( x, y ) position of the robot in Gather.
In contrast, our method is able to perform well directly from the raw sensory inputs described in Section 5.1.
We evaluate Option-Critic (Bacon et al., 2017), a variant of the options framework (Sutton et al., 1999) that can be used for continuous action-spaces.
It fails to learn, and we hypothesize that their algorithm provides less time-correlated exploration and learns
[Figure] [Figure]
less diverse skills.
We also compare against MLSH (Frans et al., 2018)
[Figure]
which repeatedly samples new environment conﬁgurations to learn primitive skills.
We take these hyperparameters from their
Ant Twowalk experiment: resetting the environment conﬁguration every 60 iterations, a warmup period of 20 during which only the manager is trained
[Figure]
and a joint training period of 40 during which both manager and skills are trained.
Our results show that such a training scheme does not provide any beneﬁts.
Finally
[Figure]
we provide a comparison to a direct application of our Hierarchical Vanilla
Policy Gradient (HierVPG) algorithm
[Figure]
and we see that the algorithm is unstable without PPO’s trust-region-like technique
[Figure]
5. R OBUSTNESS TO D YNAMICS P ERTURBATIONS
We investigate the robustness of HiPPO to changes in the dynamics of the environment.
We perform several modiﬁcations to the base Snake Gather and Ant Gather environments.
One at a time, we change the body mass, dampening of the joints, body inertia
[Table]
and friction characteristics of both robots.
The results, presented in Table 1, show that HiPPO with randomized period
Categorical ([ T min , T max ]) is able to better handle these dynamics changes.
In terms of the drop in policy performance between the training environment and test environment, it outperforms HiPPO with ﬁxed period on 6 out of 8 related tasks.
These results suggest that the randomized period exposes the policy to a wide range of scenarios, which makes it easier to adapt when the environment changes.
[Table]
5. A DAPTATION OF P RE -T RAINED S KILLS
For the Block task, we use DIAYN (Eysenbach et al., 2019) to train 6 differentiated subpolicies in an environment without any walls.
Here, we see if these diverse skills can improve performance on a downstream task that’s out of the training distribution.
For Gather, we take 6 pretrained
[Figure]
subpolicies encoded by a Stochastic Neural Network (Tang & Salakhutdinov, 2013) that was trained in a diversity-promoting environment (Florensa et al., 2017a).
We ﬁne-tune them with HiPPO on the
Gather environment, but with an extra penalty on the velocity of the Center of Mass.
This can be understood as a preference for cautious behavior.
This requires adjustment of the sub-policies, which were trained with a proxy reward encouraging them to move as far as possible (and hence quickly).
Fig.
6 shows that using HiPPO to simultaneously train a manager and ﬁne-tune the skills achieves higher ﬁnal performance than ﬁxing the sub-policies and only training a manager with PPO.
The two initially learn at the same rate, but HiPPO’s ability to adjust to the new dynamics allows it to reach a higher ﬁnal performance.
Fig.
6 also shows that HiPPO can ﬁne-tune the same given skills better than Option-Critic (Bacon et al., 2017), MLSH (Frans et al., 2018), and HIRO (Nachum et al., 2018).
5. S KILL D IVERSITY A SSUMPTION
In Lemma 1, we derived a more efﬁcient and numerically stable gradient by assuming that the sub-policies are diverse.
In this section, we empirically test the validity of our assumption and the quality of our approximation.
We run the HiPPO algorithm on Ant Gather and Snake Gather both from scratch and with given pretrained skills, as done in the previous section.
In Table 2, we report the average maximum probability under other sub-policies, corresponding to (cid:15) from the assumption.
In all settings, this is on the order of magnitude of 0.1.
Therefore, under the p ≈ 10 that we use in our experiments, the term we neglect has a factor (cid:15) p − 1 = 10 − 10 .
It is not surprising then that the average cosine similarity between the full gradient and our approximation is almost 1, as reported in Table 2.
Gather
Algorithm
Cosine Sim.
[Equation]
[Table]
C ONCLUSIONS AND F UTURE W ORK
In this paper, we examined how to effectively adapt temporal hierarchies.
We began by deriving a hierarchical policy gradient and approximation of it.
We then proposed a new method, HiPPO, that can stably train multiple layers of a hierarchy jointly.
The adaptation experiments suggest that we can optimize pretrained skills for downstream environments, and learn emergent skills without any unsupervised pre-training.
We also demonstrate that HiPPO with randomized period can learn from scratch on sparse-reward and long time horizon tasks, while outperforming non-hierarchical methods on zero-shot transfer.
A H YPERPARAMETERS AND A RCHITECTURES
The Block environments used a horizon of 1000 and a batch size of 50,000, while Gather used a batch size of 100,000.
Ant Gather has a horizon of 5000, while Snake Gather has a horizon of 8000 due to its larger size.
For all experiments, both PPO and HiPPO used learning rate 3 × 10 − 3 , clipping parameter (cid:15)
= 0 .
1 , 10 gradient updates per iteration, and discount γ = 0 . 999 .
The learning rate, clipping parameter, and number of gradient updates come from the OpenAI Baselines implementation.
HiPPO used n = 6 sub-policies.
HiPPO uses a manager network with 2 hidden layers of 32 units, and a skill network with 2 hidden layers of 64 units.
In order to have roughly the same number of parameters for each algorithm, ﬂat PPO uses a network with 2 hidden layers with 256 and 64 units respectively.
For HiPPO with randomized period, we resample p ∼ Uniform { 5 , 15 } every time the manager network outputs a latent, and provide the number of timesteps until the next latent selection as an input into both the manager and skill networks.
The single baselines and skill-dependent baselines used a MLP with 2 hidden layers of 32 units to ﬁt the value function.
The skill-dependent baseline receives, in addition to the full observation, the active latent code and the time remaining until the next skill sampling.
All runs used ﬁve random seeds.
B R OBOT A GENT D ESCRIPTION
Hopper is a 3-link robot with a 14-dimensional observation space and a 3-dimensional action space.
Half-Cheetah has a 20-dimensional observation space and a 6-dimensional action space.
We evaluate both of these agents on a sparse block hopping task.
In addition to observing their own joint angles and positions, they observe the height and length of the next wall, the x-position of the next wall, and the distance to the wall from the agent.
We also provide the same wall observations for the previous wall, which the agent can still interact with.
Snake is a 5-link robot with a 17-dimensional observation space and a 4-dimensional action space.
Ant is a quadrupedal robot with a 27-dimensional observation space and a 8-dimensional action space.
Both Ant and Snake can move and rotate in all directions, and Ant faces the added challenge of avoiding falling over irrecoverably.
In the Gather environment, agents also receive 2 sets of 10-dimensional lidar observations, whcih correspond to separate apple and bomb observations.
The observation displays the distance to the nearest apple or bomb in each 36 ◦ bin, respectively.
All environments are simulated with the physics engine MuJoCo
(Todorov et al., 2012).
C P ROOFS
Lemma 1.
If the skills are sufﬁciently differentiated, then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability.
Concretely, if π θ h ( z | s ) and π θ l ( a | s, z ) are Lipschitz in their parameters, and
0 <
π θ l ( a t | s t , z j ) < (cid:15)
∀ j (cid:54)
=
kp , then
[Equation]
Proof.
From the point of view of the MDP, a trajectory is a sequence τ = ( s 0 , a 0 , s 1 , a 1 , . . .
, a H − 1 , s H ) .
Let’s assume we use the hierarchical policy introduced above, with a higher-level policy modeled as a parameterized discrete distribution with n possible outcomes π θ h
( z | s ) =
Categorical θ h ( n ) .
We can expand P ( τ ) into the product of policy and environment dynamics terms, with z j denoting the j th possible value out of the n choices,
P ( τ
[Equation]
t =
kp π θ l ( a t | s t , z j ) (cid:105)(cid:19)(cid:20) P ( s 0 )
H (cid:89) t =
P ( s t +1 | s t , a t ) (cid:21)
Taking the gradient of log P ( τ ) with respect to the policy parameters θ
=
[ θ h , θ l ] , the dynamics terms disappear, leaving:
[Equation]
The sum over possible values of z prevents the logarithm from splitting the product over the p -step sub-trajectories.
This term is problematic, as this product quickly approaches 0 as p increases, and suffers from considerable numerical instabilities.
Instead, we want to approximate this sum of products by a single one of the terms, which can then be decomposed into a sum of logs.
For this we study each of the terms in the sum: the gradient of a sub-trajectory probability under a speciﬁc latent ∇ θ (cid:16) π
[Equation] [Equation]
(cid:81)
[Equation]
k +
[Equation]
p −
[Equation]
π θ l ( a t | s t , z j ) (cid:17) .
Now we can use the assumption that the skills are easy to distinguish, 0 < π θ l ( a t | s t , z j ) < (cid:15) ∀ j (cid:54)
=
kp .
Therefore, the probability of the sub-trajectory under a latent different than the one that was originally sampled z j (cid:54) =
z kp , is upper bounded by (cid:15) p .
Taking the gradient, applying the product rule, and the Lipschitz continuity of the policies, we obtain that for all z j (cid:54) =
z kp ,
[Equation]
Thus, we can across the board replace the summation over latents by the single term corresponding to the latent that was sampled at that time.
[Equation]
Interestingly, this is exactly ∇ θ P ( s 0 , z 0 , a 0 , s 1 , . . . ) .
In other words, it’s the gradient of the probability of that trajectory, where the trajectory now includes the variables z as if they were observed.
Lemma 2.
For any functions b h : S → R and b l : S ×
Z
→ R we have:
[Equation]
Proof.
We can use the tower property as well as the fact that the interior expression only depends on s kp and z kp :
[Equation]
Then
[Equation]
we can write out the deﬁnition of the expectation and undo the gradient-log trick to prove that the baseline is unbiased
[Equation]
Subtracting a state- and subpolicy- dependent baseline from the second term is also unbiased, i.e.
[Equation]
We’ll follow the same strategy to prove the second equality: apply the tower property, express the expectation as an integral, and undo the gradient-log trick.
[Equation]
[Figure]
D HIRO SENSITIVITY TO OBSERVATION - SPACE
In this section we provide a more detailed explanation of why HIRO (Nachum et al., 2018) performs poorly under our environments.
As explained in our related work section, HIRO belongs to the general category of algorithms that train goal-reaching policies as lower levels of the hierarchy (Vezhnevets et al., 2017; Levy et al., 2017).
These methods rely on having a goal-space that is meaningful for the task at hand.
For example, in navigation tasks they require having access to the ( x, y ) position of the agent such that deltas in that space can be given as meaningful goals to move in the environment.
Unfortunately, in many cases the only readily available information (if there’s no GPS signal or other positioning system installed) are raw sensory inputs, like cameras or the LIDAR sensors we mimic in our environments.
In such cases, our method still performs well because it doesn’t rely on the goal-reaching extra supervision that is leveraged (and detrimental in this case) in HIRO and similar methods.
In Figure 7, we show that knowing the ground truth location is critical for its success.
We have reproduced the HIRO results in Fig.
7 using the published codebase, so we are convinced that our results showcase a failure mode of HIRO.
E H YPERPARAMETER S ENSITIVITY P LOTS
[Figure] [Figure]