Original: H IERARCHICAL RL U SING AN E NSEMBLE OF P ROPRIOCEPTIVE P ERIODIC P OLICIES
Revised: H IERARCHICAL RL U SING AN E NSEMBLE OF P ROPRIOCEPTIVE P ERIODIC P OLICIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks.
Revised: In this paper we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks.

Original: The agent is split into a low-level and a high-level policy.
Revised: The agent is split into a low-level and a high-level policy.

Original: The low-level policy only accesses internal, proprioceptive dimensions of the state observation.
Revised: The low-level policy only accesses internal, proprioceptive dimensions of the state observation.

Original: The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions.
Revised: The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions.

Original: Furthermore, it is induced to be periodic with the use a “phase function.”
Revised: Furthermore, it is induced to be periodic with the use a “phase function.”

Original: The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time.
Revised: The high-level policy is trained using a sparse, task-dependent reward and operates by choosing which of the low-level policies to run at any given time.

Original: Using this approach, we solve difﬁcult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods.
Revised: Using this approach, we solve difﬁcult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The notion of hierarchy is fundamental to AI systems.
Revised: The notion of hierarchy is fundamental to AI systems.

Original: Effective approaches to perceptual tasks such as vision or audio rely on hierarchical neural network architectures (e.g. convnets and wavenets).
Revised: Effective approaches to perceptual tasks such as vision or audio rely on hierarchical neural network architectures (e.g. convnets and wavenets).

Original: Hierarchical methods are also a natural approach to solving reinforcement learning problems where reward is sparse.
Revised: Hierarchical methods are also a natural approach to solving reinforcement learning problems where reward is sparse.

Original: Consider an environment where a legged robot has to navigate a maze through winding paths to one of several possible goals.
Revised: Consider an environment where a legged robot has to navigate a maze through winding paths to one of several possible goals.

Original: The sequence of actions (at the level of joint controls) required for this agent to ever see a reward is too long and complex for na¨ıve reinforcement learning methods to ﬁnd.
Revised: The sequence of actions (at the level of joint controls) required for this agent to ever see a reward is too long and complex for na¨ıve reinforcement learning methods to ﬁnd.

Original: Hierarchical controllers provide an intuitive solution to this problem: ﬁrst learn a low-level controller that is able to handle low-level tasks such as locomotion and balance, and then learn a high-level controller that controls the inputs to that low-level controller and handles the high-level planning tasks (e.g. which path to take in the maze).
Revised: Hierarchical controllers provide an intuitive solution to this problem: ﬁrst learn a low-level controller that is able to handle low-level tasks such as locomotion and balance, and then learn a high-level controller that controls the inputs to that low-level controller and handles the high-level planning tasks (e.g. which path to take in the maze).

Original: While conceptually straightforward, hierarchical approaches have been difﬁcult to realize in practice for several reasons: (i) the lack of constraints on intermediate representations; (ii) the difﬁculty of learning a high-level controller when the low-level policies shifts, and (iii) the continued problem of sample-inefﬁciency when training both low-level and high-level controller jointly with sparse rewards.
Revised: While conceptually straightforward, hierarchical approaches have been difﬁcult to realize in practice for several reasons: (i) the lack of constraints on intermediate representations; (ii) the difﬁculty of learning a high-level controller when the low-level policies shifts, and (iii) the continued problem of sample-inefﬁciency when training both low-level and high-level controller jointly with sparse rewards.

Original: While there has been some progress recently, there is still no general hierarchical reinforcement learning method that is robust across domains, easy to implement, and effective.
Revised: While there has been some progress recently, there is still no general hierarchical reinforcement learning method that is robust across domains, easy to implement, and effective.

Original: The approach in this work avoids these problems by simplifying the high-level optimization problem to choosing from a set of independently trained low-level controllers.
Revised: The approach in this work avoids these problems by simplifying the high-level optimization problem to choosing from a set of independently trained low-level controllers.

Original: The problem then becomes how to train a variety of controllers which are generally useful for an agent acting in the world and are sufﬁciently diverse so that the high-level controller will always have access to the building blocks it needs to solve the high-level task.
Revised: The problem then becomes how to train a variety of controllers which are generally useful for an agent acting in the world and are sufﬁciently diverse so that the high-level controller will always have access to the building blocks it needs to solve the high-level task.

Original: The basic method of learning our low-level skills is as follows.
Revised: The basic method of learning our low-level skills is as follows.

Original: First, as with Heess et al. (2016), we factorize the state space into components that describe the internal and external properties of the agent.
Revised: First, as with Heess et al. (2016), we factorize the state space into components that describe the internal and external properties of the agent.

Original: The former carries information needed for proprioception (e.g. internal joint angles) and is exclusively used by the low-level policy and the latter is used exclusively by the high-level policy.
Revised: The former carries information needed for proprioception (e.g. internal joint angles) and is exclusively used by the low-level policy and the latter is used exclusively by the high-level policy.

Original: Second, we provide a weak supervisory signal to the low-level policies that rewards them for changes in the external part of the space.
Revised: Second, we provide a weak supervisory signal to the low-level policies that rewards them for changes in the external part of the space.

Original: We do not specify how; the low-level agent is rewarded for any change in the perceptual features.
Revised: We do not specify how; the low-level agent is rewarded for any change in the perceptual features.

Original: The intuition is that a policy is useful to an agent if following it changes the external world or the agent’s place in it, (e.g. by changing position or moving objects).
Revised: The intuition is that a policy is useful to an agent if following it changes the external world or the agent’s place in it, (e.g. by changing position or moving objects).

Original: Finally, we build into the agent a notion of repeated cycles of action.
Revised: Finally, we build into the agent a notion of repeated cycles of action.

Original: Biological systems are known to possess central pattern generators (CPGs) that produce rhythmic outputs, assisting with perception, locomotion, and self-regulation (Marder & Bucher, 2001).
Revised: Biological systems are known to possess central pattern generators (CPGs) that produce rhythmic outputs, assisting with perception, locomotion, and self-regulation (Marder & Bucher, 2001).

Original: Taking inspiration from this, we give our agent access to a “phase function” and reward it for acting in consistent, cyclic ways.
Revised: Taking inspiration from this, we give our agent access to a “phase function” and reward it for acting in consistent, cyclic ways.

Original: Using this method of training, we train a number of low-level controllers on Mujoco locomotion tasks and are able to learn a diverse and effective set of low-level policies.
Revised: Using this method of training, we train a number of low-level controllers on Mujoco locomotion tasks and are able to learn a diverse and effective set of low-level policies.

Original: We then show that these policies are easily controllable by our high-level controller and sufﬁcient for solving the ﬁnal task.
Revised: We then show that these policies are easily controllable by our high-level controller and sufﬁcient for solving the ﬁnal task.

Original: We show impressive results on a variety of difﬁcult Mujoco maze and navigation tasks with sparse rewards and demonstrate that our method compares favorably to other recent hierarchical RL approaches.
Revised: We show impressive results on a variety of difﬁcult Mujoco maze and navigation tasks with sparse rewards and demonstrate that our method compares favorably to other recent hierarchical RL approaches.

Original: Finally, we demonstrate that our method is robust to choice of RL algorithm.
Revised: Finally, we demonstrate that our method is robust to choice of RL algorithm.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There is an extensive literature on hierarchical approaches to reinforcement learning (Sutton et al., 1999; Dayan & Hinton, 1992; Dietterich, 1999).
Revised: There is an extensive literature on hierarchical approaches to reinforcement learning (Sutton et al., 1999; Dayan & Hinton, 1992; Dietterich, 1999).

Original: Options (Sutton et al., 1999) are perhaps the most standard framing for hierarchical RL, formalizing the notion of a subroutine in an MDP.
Revised: Options (Sutton et al., 1999) are perhaps the most standard framing for hierarchical RL, formalizing the notion of a subroutine in an MDP.

Original: Our lowlevel networks are options with a deterministic return condition.
Revised: Our lowlevel networks are options with a deterministic return condition.

Original: While earlier works used prespeciﬁed subroutines, and focused on learning when to call them, recent works, for example Bacon et al.
Revised: While earlier works used prespeciﬁed subroutines, and focused on learning when to call them, recent works, for example Bacon et al.

Original: ; Vezhnevets et al.
Revised: ; Vezhnevets et al.

Original: ; Nachum et al.
Revised: ; Nachum et al.

Original: (2018), have demonstrated success in learning multiple levels of hierarchy end-to-end from only task reward.
Revised: (2018), have demonstrated success in learning multiple levels of hierarchy end-to-end from only task reward.

Original: Still, in more complex settings such as the ones discussed in this work, with high dimensional action spaces, and especially with sparse rewards, performing unsupervised pre-training, or using auxiliary tasks during learning or pre-training have led to better results (Heess et al., 2016; Frans et al., 2017;
Revised: Still, in more complex settings such as the ones discussed in this work, with high dimensional action spaces, and especially with sparse rewards, performing unsupervised pre-training, or using auxiliary tasks during learning or pre-training have led to better results (Heess et al., 2016; Frans et al., 2017;

Original: Florensa et al., 2017; Haarnoja et al., 2018b; Eysenbach et al., 2018; Hausman et al., 2018).
Revised: Florensa et al., 2017; Haarnoja et al., 2018b; Eysenbach et al., 2018; Hausman et al., 2018).

Original: One line of work in this direction starts from a variational inference approach to intrinsic motivation and exploration (Mohamed & Rezende, 2015; Gregor et al., 2016;
Revised: One line of work in this direction starts from a variational inference approach to intrinsic motivation and exploration (Mohamed & Rezende, 2015; Gregor et al., 2016;

Original: Florensa et al., 2017; Eysenbach et al., 2018).
Revised: Florensa et al., 2017; Eysenbach et al., 2018).

Original: They use an actor parameterized by state and a latent vector in such a way that the latent vector is predictable from a ﬁnal state or a sequence of states the actor visits, but otherwise, the actions have high entropy.
Revised: They use an actor parameterized by state and a latent vector in such a way that the latent vector is predictable from a ﬁnal state or a sequence of states the actor visits, but otherwise, the actions have high entropy.

Original: The latter of these works use this approach for designing hierarchical policies.
Revised: The latter of these works use this approach for designing hierarchical policies.

Original: In Florensa et al.
Revised: In Florensa et al.

Original: (2017), they use a pre-training task, and the prediction cost is an auxiliary reward.
Revised: (2017), they use a pre-training task, and the prediction cost is an auxiliary reward.

Original: The main differences from our work is that we do not try to use stochastic neural networks, and we do not try to compress the one-hot representation of the low level networks into a dense vector during low level training.
Revised: The main differences from our work is that we do not try to use stochastic neural networks, and we do not try to compress the one-hot representation of the low level networks into a dense vector during low level training.

Original: Instead we keep them fully separate.
Revised: Instead we keep them fully separate.

Original: We also do not attempt to impose any regularization to encourage the low level networks to be diverse.
Revised: We also do not attempt to impose any regularization to encourage the low level networks to be diverse.

Original: These can be considered simpliﬁcations; what we show is that the simple thing works quite well.
Revised: These can be considered simpliﬁcations; what we show is that the simple thing works quite well.

Original: In Eysenbach et al.
Revised: In Eysenbach et al.

Original: the low-level policy is trained without using any environmental reward by pre-training using the unsupervised variational objective; they then training a high-level actor to issue commands via the latent vector.
Revised: the low-level policy is trained without using any environmental reward by pre-training using the unsupervised variational objective; they then training a high-level actor to issue commands via the latent vector.

Original: This latter work is in some senses more general than ours, in that a user is not required to separate out the proprioceptive variables from the external sensory variables.
Revised: This latter work is in some senses more general than ours, in that a user is not required to separate out the proprioceptive variables from the external sensory variables.

Original: In addition, the formulation directly encourages diverse and controllable low-level policies.
Revised: In addition, the formulation directly encourages diverse and controllable low-level policies.

Original: Nevertheless, both these works were validated mostly in the same settings we use here, where the separation between variables is natural, and the reward in equation 3.2 is well motivated.
Revised: Nevertheless, both these works were validated mostly in the same settings we use here, where the separation between variables is natural, and the reward in equation 3.2 is well motivated.

Original: In these settings our method is simpler, and performs well in comparison (see Figure 7a).
Revised: In these settings our method is simpler, and performs well in comparison (see Figure 7a).

Original: Our work is most closely related to Heess et al.
Revised: Our work is most closely related to Heess et al.

Original: We operate under the same basic philosophy, separating the agent into a low-level component that only sees “proprioceptive” information, and works at a ﬁne time scale, and a coarse agent that has access to “sensory” information.
Revised: We operate under the same basic philosophy, separating the agent into a low-level component that only sees “proprioceptive” information, and works at a ﬁne time scale, and a coarse agent that has access to “sensory” information.

Original: Like that work, we evaluate or method in the setting of locomotive agents.
Revised: Like that work, we evaluate or method in the setting of locomotive agents.

Original: However, there are several differences between Heess et al.
Revised: However, there are several differences between Heess et al.

Original: (2016) and this work.
Revised: (2016) and this work.

Original: There, the low-level component is trained along with a provisional high-level component to solve a set of high-level tasks designed for the agent and transfer tasks, whereas in this work, the low-level controller is trained with only the objective “change the sensory responses” in each setting.
Revised: There, the low-level component is trained along with a provisional high-level component to solve a set of high-level tasks designed for the agent and transfer tasks, whereas in this work, the low-level controller is trained with only the objective “change the sensory responses” in each setting.

Original: Furthermore, our underlying architecture is different.
Revised: Furthermore, our underlying architecture is different.

Original: Whereas in Heess et al.
Revised: Whereas in Heess et al.

Original: the high-level agent interacts with the low-level component via continuous control vectors, in this work, we sample one of N low-level policies, as in Frans et al.
Revised: the high-level agent interacts with the low-level component via continuous control vectors, in this work, we sample one of N low-level policies, as in Frans et al.

Original: In addition, whereas they train their low-level actor with only the current state, our low-level actor is shown to beneﬁt from using a pattern generator.
Revised: In addition, whereas they train their low-level actor with only the current state, our low-level actor is shown to beneﬁt from using a pattern generator.

Original: Finally, because of the structure of our lowlevel task, we need to ensure that the low-level policies are diverse and controllable (which in Heess et al.
Revised: Finally, because of the structure of our lowlevel task, we need to ensure that the low-level policies are diverse and controllable (which in Heess et al.

Original: (2016) is given by the design of the pre-training task), and we do so by using multiple random instantiations of the “change the sensory response” policies that maximize equation 3.2.
Revised: (2016) is given by the design of the pre-training task), and we do so by using multiple random instantiations of the “change the sensory response” policies that maximize equation 3.2.

Original: Our work can thus be considered in some ways a simpliﬁcation and generalization of Heess et al.
Revised: Our work can thus be considered in some ways a simpliﬁcation and generalization of Heess et al.

Original: The use of phase information in the low-level actor recalls Holden et al.
Revised: The use of phase information in the low-level actor recalls Holden et al.

Original: (2017); Sharma & Kitani (2018).
Revised: (2017); Sharma & Kitani (2018).

Original: In the former, a neural network is used to control the gait of a video game character.
Revised: In the former, a neural network is used to control the gait of a video game character.

Original: The weights of the neural network are a function of a phase variable whose cycle corresponds to a cycle of the gait, and are trained as regressors on motion-capture data.
Revised: The weights of the neural network are a function of a phase variable whose cycle corresponds to a cycle of the gait, and are trained as regressors on motion-capture data.

Original: The latter work takes this approach to RL problems that have an explicit cyclic structure, in particular the walker and hopper environments from MuJoCo.
Revised: The latter work takes this approach to RL problems that have an explicit cyclic structure, in particular the walker and hopper environments from MuJoCo.

Original: Phase has also been used in similar way in Peng et al.
Revised: Phase has also been used in similar way in Peng et al.

Original: While we share a similar motivation with these, our low-level actor processes the phases in a different way: instead of a phase variable directly modulating the weight of the network, we concatenate an embedded phase index to the input of the network.
Revised: While we share a similar motivation with these, our low-level actor processes the phases in a different way: instead of a phase variable directly modulating the weight of the network, we concatenate an embedded phase index to the input of the network.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Section 3.1 describes the division of the state space into the proprioceptive or internal state and the external state.
Revised: Section 3.1 describes the division of the state space into the proprioceptive or internal state and the external state.

Original: In section 3.2 we describe the reward function which we use to train the low-level policies.
Revised: In section 3.2 we describe the reward function which we use to train the low-level policies.

Original: These policies take in the proprioceptive state and the reward maximizes displacement in the external state.
Revised: These policies take in the proprioceptive state and the reward maximizes displacement in the external state.

Original: Section 3.3 describes how we augment the low-level policy by adding an additional input that tells the agent where it is in a “phase cycle” and constraining the motion of the agent to match that phase cycle.
Revised: Section 3.3 describes how we augment the low-level policy by adding an additional input that tells the agent where it is in a “phase cycle” and constraining the motion of the agent to match that phase cycle.

Original: Finally, in section 3.4 we describe how we learn a high-level controller by learning to choose when and which of the trained low-level policies should be used.
Revised: Finally, in section 3.4 we describe how we learn a high-level controller by learning to choose when and which of the trained low-level policies should be used.

Original: 3. P ROPRIOCEPTIVE AND E XTERNAL S TATES
Revised: 3. P ROPRIOCEPTIVE AND E XTERNAL S TATES

Original: We divide the agent’s sensory output into two sets of variables.
Revised: We divide the agent’s sensory output into two sets of variables.

Original: The proprioceptive states, s p , are the measurements of the agents own body that can be directly affected by control.
Revised: The proprioceptive states, s p , are the measurements of the agents own body that can be directly affected by control.

Original: The external states, s e are measurements of the environment outside of the agents direct control.
Revised: The external states, s e are measurements of the environment outside of the agents direct control.

Original: The proprioceptive part of the state is fed into the low-level policy network.
Revised: The proprioceptive part of the state is fed into the low-level policy network.

Original: This is the part of the state that gives the agent the information about the conﬁguration of its own body and actuators.
Revised: This is the part of the state that gives the agent the information about the conﬁguration of its own body and actuators.

Original: The low-level policy is independent of the external state as the low-level policies are intended to be generally usefully in moving in the external state independently of the exact location.
Revised: The low-level policy is independent of the external state as the low-level policies are intended to be generally usefully in moving in the external state independently of the exact location.

Original: The external part of the state is fed into the high-level policy network.
Revised: The external part of the state is fed into the high-level policy network.

Original: The high-level policy is responsible for things like moving to particular coordinates in the world, and so the external state is necessary.
Revised: The high-level policy is responsible for things like moving to particular coordinates in the world, and so the external state is necessary.

Original: The high-level policy does not need the low-level states such as joint locations as that is handled by the low-level policy.
Revised: The high-level policy does not need the low-level states such as joint locations as that is handled by the low-level policy.

Original: 3. L OW - LEVEL POLICY TRAINING
Revised: 3. L OW - LEVEL POLICY TRAINING

Original: We would like a set of low-level policies that are diverse, effective, and controllable.
Revised: We would like a set of low-level policies that are diverse, effective, and controllable.

Original: Here “effective” means that they can make signiﬁcant changes to the environment.
Revised: Here “effective” means that they can make signiﬁcant changes to the environment.

Original: “Diverse” means they make different changes, hopefully covering all of the things the agent will need to be able to do in its test task.
Revised: “Diverse” means they make different changes, hopefully covering all of the things the agent will need to be able to do in its test task.

Original: Finally “controllable” means that the low-level policies are easy for the high-level policy to use, for example because they are predictable and their effect is independent of the external state.
Revised: Finally “controllable” means that the low-level policies are easy for the high-level policy to use, for example because they are predictable and their effect is independent of the external state.

Original: Empirically, we have found that simply instantiating multiple randomly initialized neural networks trained to change the external perception sufﬁces.
Revised: Empirically, we have found that simply instantiating multiple randomly initialized neural networks trained to change the external perception sufﬁces.

Original: Formally, our basic low-level reward at each time step is
Revised: Formally, our basic low-level reward at each time step is

Original: [Equation]
Revised: [Equation]

Original: Using this reward scheme, we train a number of agents (16 in the experiments below).
Revised: Using this reward scheme, we train a number of agents (16 in the experiments below).

Original: We use the same reward for each agent, and across environments (except regularized with the standard environment-speciﬁc control and survival rewards).
Revised: We use the same reward for each agent, and across environments (except regularized with the standard environment-speciﬁc control and survival rewards).

Original: As we show in Section 4.1, changing only the random seed gives us a wide variety of policies that maximize the displacement of the agent.
Revised: As we show in Section 4.1, changing only the random seed gives us a wide variety of policies that maximize the displacement of the agent.

Original: 3. P HASE - CONDITIONED P OLICIES
Revised: 3. P HASE - CONDITIONED P OLICIES

Original: One important aspect of our training procedure for our low-level policies is time index or phase inputs.
Revised: One important aspect of our training procedure for our low-level policies is time index or phase inputs.

Original: We expect good movement policies to be at least roughly periodic, and phase inputs allow the model to more easily achieve periodicity.
Revised: We expect good movement policies to be at least roughly periodic, and phase inputs allow the model to more easily achieve periodicity.

Original: Formally, we denote our phase-conditioned policy as π ( s p , φ ) , where s p is our proprioceptive observation and φ ∈ { 0 , 1 , . . .
Revised: Formally, we denote our phase-conditioned policy as π ( s p , φ ) , where s p is our proprioceptive observation and φ ∈ { 0 , 1 , . . .

Original: K − 1 } is the phase index, where K is the length of the period (in all our experiments, we choose K = 10 ).
Revised: K − 1 } is the phase index, where K is the length of the period (in all our experiments, we choose K = 10 ).

Original: At each time-step t , during training and evaluation, we receive our new observation s p , and phase index φ = t (mod K ) .
Revised: At each time-step t , during training and evaluation, we receive our new observation s p , and phase index φ = t (mod K ) .

Original: In particular, in our phase-conditioned networks, we take as input s p and a vector b φ ∈ R d , where b φ is a learned parameter, like a bias term, for each possible phase indices φ .
Revised: In particular, in our phase-conditioned networks, we take as input s p and a vector b φ ∈ R d , where b φ is a learned parameter, like a bias term, for each possible phase indices φ .

Original: d = 16 in our experiments.
Revised: d = 16 in our experiments.

Original: For our phase-conditioned policies, we also want to encourage the agents actions and internal states to be cyclic during training; at any given time step, the current action and proprioceptive state should match the one from the last cycle.
Revised: For our phase-conditioned policies, we also want to encourage the agents actions and internal states to be cyclic during training; at any given time step, the current action and proprioceptive state should match the one from the last cycle.

Original: Just as after one phase period, a pendulum is in the same state as it was before, during any given cycle, the behavior of the agent should be identical.
Revised: Just as after one phase period, a pendulum is in the same state as it was before, during any given cycle, the behavior of the agent should be identical.

Original: Formally, the reward maximization with a cyclic constraint can be written as:
Revised: Formally, the reward maximization with a cyclic constraint can be written as:

Original: [Equation]
Revised: [Equation]

Original: This can be changed to be a cyclic loss as
Revised: This can be changed to be a cyclic loss as

Original: [Equation]
Revised: [Equation]

Original: For the phase conditioned policies, we include these losses in the reward function during training.
Revised: For the phase conditioned policies, we include these losses in the reward function during training.

Original: In contrast to Holden et al.
Revised: In contrast to Holden et al.

Original: (2017); Sharma & Kitani (2018), the model need not be exactly periodic.
Revised: (2017); Sharma & Kitani (2018), the model need not be exactly periodic.

Original: 3. H IERARCHICAL TRAINING
Revised: 3. H IERARCHICAL TRAINING

Original: Once we have learned a variety of low-level policies, we want to learn how and when to use these low-level policies in a way that lets us solve much more complex task with sparse rewards.
Revised: Once we have learned a variety of low-level policies, we want to learn how and when to use these low-level policies in a way that lets us solve much more complex task with sparse rewards.

Original: The way we compose our low-level policies is quite simple.
Revised: The way we compose our low-level policies is quite simple.

Original: Similar to Eysenbach et al.
Revised: Similar to Eysenbach et al.

Original: and many other works, at each time step, our high-level policy chooses one of our low-level policies to run.
Revised: and many other works, at each time step, our high-level policy chooses one of our low-level policies to run.

Original: Given N trained low-level policies π ll ( s pt , φ ) , our high-level policy is
Revised: Given N trained low-level policies π ll ( s pt , φ ) , our high-level policy is

Original: [Equation]
Revised: [Equation]

Original: This is actually just a choice from among our low-level policies.
Revised: This is actually just a choice from among our low-level policies.

Original: We go through the complete formulation for our high-level policy in Appendix A.
Revised: We go through the complete formulation for our high-level policy in Appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments, we test on a variety of difﬁcult sparse reward problems simulated through Mujoco (Todorov et al., 2012).
Revised: In our experiments, we test on a variety of difﬁcult sparse reward problems simulated through Mujoco (Todorov et al., 2012).

Original: We use two popular and challenging agents: Ant and Humanoid.
Revised: We use two popular and challenging agents: Ant and Humanoid.

Original: In section 4.1 we show some results from our low-level training on Ant and Humanoid.
Revised: In section 4.1 we show some results from our low-level training on Ant and Humanoid.

Original: In section 4.
Revised: In section 4.

Original: we show the results of our high-level policies on several different mazes and a navigation task.
Revised: we show the results of our high-level policies on several different mazes and a navigation task.

Original: 4. L OW - LEVEL CONTROL
Revised: 4. L OW - LEVEL CONTROL

Original: As described in Section 3.2 and Section 3.3, we train a number of low-level policies using our simple reward function.
Revised: As described in Section 3.2 and Section 3.3, we train a number of low-level policies using our simple reward function.

Original: We train Ant and Humanoid in the default ﬂat plane environments during this stage.
Revised: We train Ant and Humanoid in the default ﬂat plane environments during this stage.

Original: Figure 1 shows the reward curves of our phase-conditioned low-level policies compared to standard networks.
Revised: Figure 1 shows the reward curves of our phase-conditioned low-level policies compared to standard networks.

Original: All networks are trained using PPO (Schulman et al., 2017).
Revised: All networks are trained using PPO (Schulman et al., 2017).

Original: See Appendix D for more training and network details.
Revised: See Appendix D for more training and network details.

Original: For Ant, phase-conditioning gives a respectable boost, but both achieve reasonable rewards and easily move.
Revised: For Ant, phase-conditioning gives a respectable boost, but both achieve reasonable rewards and easily move.

Original: For Humanoid, vanilla PPO (using a widely used implementation (Kostrikov, 2018)) same hyperparameters performs poorly, even though it performs well for all other standard Open AI Gym Mujoco tasks.
Revised: For Humanoid, vanilla PPO (using a widely used implementation (Kostrikov, 2018)) same hyperparameters performs poorly, even though it performs well for all other standard Open AI Gym Mujoco tasks.

Original: With the phase conditioning, however, using PPO, we are able to achieve good results with Humanoid surviving and moving forward.
Revised: With the phase conditioning, however, using PPO, we are able to achieve good results with Humanoid surviving and moving forward.

Original: In Figure 2 we show the traces of our low-level policies learned on Ant and Humanoid for 100 time steps.
Revised: In Figure 2 we show the traces of our low-level policies learned on Ant and Humanoid for 100 time steps.

Original: Each trajectory represents a different low-level policy.
Revised: Each trajectory represents a different low-level policy.

Original: We can see that we learn a great variety of movements, more than sufﬁcient for the high-level policy.
Revised: We can see that we learn a great variety of movements, more than sufﬁcient for the high-level policy.

Original: We see that Humanoid moves more slowly, both because Humanoid is much more difﬁcult to control, and because we use the standard
Revised: We see that Humanoid moves more slowly, both because Humanoid is much more difﬁcult to control, and because we use the standard

Original: OpenAI Gym (Brockman et al., 2016) simulator timesteps, which is shorter for Humanoid.
Revised: OpenAI Gym (Brockman et al., 2016) simulator timesteps, which is shorter for Humanoid.

Original: 4. M AZES AND NAVIGATION
Revised: 4. M AZES AND NAVIGATION

Original: The high-level policy network is trained according to Section 4.2, also using PPO.
Revised: The high-level policy network is trained according to Section 4.2, also using PPO.

Original: See Appendix E for more training and network details.
Revised: See Appendix E for more training and network details.

Original: The ﬁrst high-level environment that we evaluate on is the Cross Maze environment with ﬁxed goals from Haarnoja et al.
Revised: The ﬁrst high-level environment that we evaluate on is the Cross Maze environment with ﬁxed goals from Haarnoja et al.

Original: (2018a).
Revised: (2018a).

Original: As described in that work, the Cross Maze has three different goals along three different paths.
Revised: As described in that work, the Cross Maze has three different goals along three different paths.

Original: In their version of this environment, three different environments are used for each of the three different goal locations and thus a different model is learned for each version of the maze.
Revised: In their version of this environment, three different environments are used for each of the three different goal locations and thus a different model is learned for each version of the maze.

Original: In Figure 3 we compare our high-level policy to the published numbers from their method (SAC-LSP) and their baselines (see Haarnoja et al.
Revised: In Figure 3 we compare our high-level policy to the published numbers from their method (SAC-LSP) and their baselines (see Haarnoja et al.

Original: (2018a)
Revised: (2018a)

Original: Figure 5).
Revised: Figure 5).

Original: We used a different value for the length of the timestep for Ant in the simulator, so to make a fair comparison, we multiply our number of frames by 2.5 so that our x-axis compares the same amount of simulator time for high-level training.
Revised: We used a different value for the length of the timestep for Ant in the simulator, so to make a fair comparison, we multiply our number of frames by 2.5 so that our x-axis compares the same amount of simulator time for high-level training.

Original: This comparison favors their method as we receive 2.5x fewer observations.
Revised: This comparison favors their method as we receive 2.5x fewer observations.

Original: We can see that our method with a phase-input low-level policy converges faster than SAC-LSP and converges to a smaller ﬁnal distance to goal, meaning that our method is both more efﬁcient and more consistent on the task.
Revised: We can see that our method with a phase-input low-level policy converges faster than SAC-LSP and converges to a smaller ﬁnal distance to goal, meaning that our method is both more efﬁcient and more consistent on the task.

Original: Our method also has much smaller standard deviation across trials.
Revised: Our method also has much smaller standard deviation across trials.

Original: One major limitation of this environment is that from the perspective of each model, there is just one ﬁxed goal.
Revised: One major limitation of this environment is that from the perspective of each model, there is just one ﬁxed goal.

Original: So to solve it, the policy need only ﬁnd the one ﬁxed goal for that version of the environment and learn a policy to consistently move to it.
Revised: So to solve it, the policy need only ﬁnd the one ﬁxed goal for that version of the environment and learn a policy to consistently move to it.

Original: A much more difﬁcult environment would be one where the goal is randomly chosen from a set of available options each episode.
Revised: A much more difﬁcult environment would be one where the goal is randomly chosen from a set of available options each episode.

Original: To solve this kind of maze, the policy would have to explore enough to ﬁnd each goal during the fraction of episodes where it appears, and then consistent learn and keep separate paths of actions in its memory depending on the random goal location for the episode.
Revised: To solve this kind of maze, the policy would have to explore enough to ﬁnd each goal during the fraction of episodes where it appears, and then consistent learn and keep separate paths of actions in its memory depending on the random goal location for the episode.

Original: We refer to these kinds of mazes as “random goal” mazes.
Revised: We refer to these kinds of mazes as “random goal” mazes.

Original: All of the maze results except for Figureuse random goals.
Revised: All of the maze results except for Figureuse random goals.

Original: Figure 4 shows the training curves for two random goal mazes: the cross maze from before and the “skull maze.”
Revised: Figure 4 shows the training curves for two random goal mazes: the cross maze from before and the “skull maze.”

Original: See Figure 5 for the layout of these mazes.
Revised: See Figure 5 for the layout of these mazes.

Original: The skull maze has four possible goals instead of three and also forces the agent to make a decision immediately about which way to move, either to the top, left, or right corridor.
Revised: The skull maze has four possible goals instead of three and also forces the agent to make a decision immediately about which way to move, either to the top, left, or right corridor.

Original: We compare our method to baselines similar to those used in Haarnoja et al.
Revised: We compare our method to baselines similar to those used in Haarnoja et al.

Original: (2018a), all trained with
Revised: (2018a), all trained with

Original: PPO as is our method.
Revised: PPO as is our method.

Original: The baseline models are either trained with or without the phase conditioning, and either from scratch, or ﬁnetuned (meaning that we initialize the network using a network trained on our low-level objective).
Revised: The baseline models are either trained with or without the phase conditioning, and either from scratch, or ﬁnetuned (meaning that we initialize the network using a network trained on our low-level objective).

Original: We also give some of the baselines more information by also giving them a velocity reward during high-level training (meaning they are rewarded for movement of the agent).
Revised: We also give some of the baselines more information by also giving them a velocity reward during high-level training (meaning they are rewarded for movement of the agent).

Original: Even with velocity rewards and pre-training on the low-level objective (both of which could be considered exploration bonuses), all of the baselines fail to get close to the goal locations.
Revised: Even with velocity rewards and pre-training on the low-level objective (both of which could be considered exploration bonuses), all of the baselines fail to get close to the goal locations.

Original: The problem of exploration and consistent navigation to different random goals is too difﬁcult to learn from scratch.
Revised: The problem of exploration and consistent navigation to different random goals is too difﬁcult to learn from scratch.

Original: Figure 5 shows traces of Ant navigating the cross and skull mazes.
Revised: Figure 5 shows traces of Ant navigating the cross and skull mazes.

Original: Most traces reach the goal, although some terminate early.
Revised: Most traces reach the goal, although some terminate early.

Original: All move fairly decisively towards the correct goal.
Revised: All move fairly decisively towards the correct goal.

Original: This maze is smaller than the Ant version, so all curves start closer to the goal

Original: In Figure 6, we show the results using Humanoid on a smaller version of the cross maze.
Revised: In Figure 6, we show the results using Humanoid on a smaller version of the cross maze.

Original: Because the PPO without phase fails at the low-level policy, we omit all non-phase results.
Revised: Because the PPO without phase fails at the low-level policy, we omit all non-phase results.

Original: The low-level
Revised: The low-level

Original: Humanoid control problem is much harder, and moves less quickly (as we saw in Figure 2), so our method does not do as well as it did on Ant.
Revised: Humanoid control problem is much harder, and moves less quickly (as we saw in Figure 2), so our method does not do as well as it did on Ant.

Original: But it is still able to successfully reach the goal much of the time, and again the baselines again fail to learn the maze.
Revised: But it is still able to successfully reach the goal much of the time, and again the baselines again fail to learn the maze.

Original: To our knowledge, ours is the only work that shows results on a Humanoid maze task.
Revised: To our knowledge, ours is the only work that shows results on a Humanoid maze task.

Original: In Appendix F Table 4, we also show the rates of success of these methods at reaching the random goals.
Revised: In Appendix F Table 4, we also show the rates of success of these methods at reaching the random goals.

Original: Here it is even more obvious that na¨ıve RL methods fail completely in this task.
Revised: Here it is even more obvious that na¨ıve RL methods fail completely in this task.

Original: In Figure 7a, we compare to Eysenbach et al.
Revised: In Figure 7a, we compare to Eysenbach et al.

Original: (2018) which is similar to our method in the way it composes low-level policies, but generates these using an entropy method.
Revised: (2018) which is similar to our method in the way it composes low-level policies, but generates these using an entropy method.

Original: We compare to their
Revised: We compare to their

Original: [Figure]
Revised: [Figure]

Original: results on the Ant Navigation task (see section 5.2 of their paper) in which Ant has to reach four different waypoints and then travel back to the original waypoint.
Revised: results on the Ant Navigation task (see section 5.2 of their paper) in which Ant has to reach four different waypoints and then travel back to the original waypoint.

Original: The agent receives reward after reaching each waypoint, so it is another sparse reward task.
Revised: The agent receives reward after reaching each waypoint, so it is another sparse reward task.

Original: Because the plot in their paper measures by walltime instead of timesteps, we show their maximum result (DIAYN) as a dotted horizontal line.
Revised: Because the plot in their paper measures by walltime instead of timesteps, we show their maximum result (DIAYN) as a dotted horizontal line.

Original: While their method is only able to achieve an average of about 2 .
Revised: While their method is only able to achieve an average of about 2 .

Original: 5 waypoints after 15 hours on their hardware, ours can consistently get to all 5 in less time than that, running serially on CPU.
Revised: 5 waypoints after 15 hours on their hardware, ours can consistently get to all 5 in less time than that, running serially on CPU.

Original: In Figure 7b, we look at the effect of the choice of algorithm on our high-level training.
Revised: In Figure 7b, we look at the effect of the choice of algorithm on our high-level training.

Original: We compare
Revised: We compare

Original: PPO with A2C (Mnih et al., 2016) using the implementation from Kostrikov (2018), and our own implementation of DQN (Mnih et al., 2013; Watkins, 1989).
Revised: PPO with A2C (Mnih et al., 2016) using the implementation from Kostrikov (2018), and our own implementation of DQN (Mnih et al., 2013; Watkins, 1989).

Original: See Appendix C for more implimentation and hyperparameter details.
Revised: See Appendix C for more implimentation and hyperparameter details.

Original: We end DQN early as it took much longer in walltime, but we can see that the choice of algorithm is not critical for successfully training.
Revised: We end DQN early as it took much longer in walltime, but we can see that the choice of algorithm is not critical for successfully training.

Original: This suggests that our method is fairly robust to the choice of RL algorithm.
Revised: This suggests that our method is fairly robust to the choice of RL algorithm.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we present a simple, yet effective way of solving difﬁcult sparse reward RL problems.
Revised: In this work we present a simple, yet effective way of solving difﬁcult sparse reward RL problems.

Original: We ﬁrst train a set of proprioceptive low-level agents on an intuitive reward and then combine them by training a high-level policy to select the appropriate sequence of low-level policies.
Revised: We ﬁrst train a set of proprioceptive low-level agents on an intuitive reward and then combine them by training a high-level policy to select the appropriate sequence of low-level policies.

Original: With this method we solve difﬁcult Mujoco navigation tasks with sparse rewards that appear impossible to solve using standard RL algorithms.
Revised: With this method we solve difﬁcult Mujoco navigation tasks with sparse rewards that appear impossible to solve using standard RL algorithms.

Original: We show that our method performs well in difﬁcult sparse reward problems in Mujoco locomotion tasks.
Revised: We show that our method performs well in difﬁcult sparse reward problems in Mujoco locomotion tasks.

Original: However, it can be applied to any environment where we have a strong notion of internal and external observation.
Revised: However, it can be applied to any environment where we have a strong notion of internal and external observation.

Original: Our reward function encourages behaviors of the agent that change the robot’s state in the environment (i.e. “keep moving”) or changes something else external to the agent.
Revised: Our reward function encourages behaviors of the agent that change the robot’s state in the environment (i.e. “keep moving”) or changes something else external to the agent.

Original: This notion exists in environments beyond our Mujoco locomotion problems.
Revised: This notion exists in environments beyond our Mujoco locomotion problems.

Original: Some possible applications for our method are Atari and video game environments with sparse rewards (e.g. Montezuma’s
Revised: Some possible applications for our method are Atari and video game environments with sparse rewards (e.g. Montezuma’s

Original: Revenge) where to receive rewards the agent must complete a long sequence of actions such as ﬁnding a key in one area and bringing it to a lock in another.
Revised: Revenge) where to receive rewards the agent must complete a long sequence of actions such as ﬁnding a key in one area and bringing it to a lock in another.

Original: It could also have potential applications for robotics such as in navigation and manipulation where there are not dense rewards or there is a high labeling cost to those rewards.
Revised: It could also have potential applications for robotics such as in navigation and manipulation where there are not dense rewards or there is a high labeling cost to those rewards.

Original: In this work, we do not show any results where we ﬁne-tune the low-level policies during highlevel training.
Revised: In this work, we do not show any results where we ﬁne-tune the low-level policies during highlevel training.

Original: This is relatively straightforward, and we have implemented this.
Revised: This is relatively straightforward, and we have implemented this.

Original: We omitted these results as they did not substantially improve the results.
Revised: We omitted these results as they did not substantially improve the results.

Original: For other problems, however, there may be cases where this is useful.
Revised: For other problems, however, there may be cases where this is useful.

Original: For example, if the friction of the ground or some other property of the environment changed between the high-level and low-level environment ﬁne-tuning might be useful.
Revised: For example, if the friction of the ground or some other property of the environment changed between the high-level and low-level environment ﬁne-tuning might be useful.

Original: A H IGH L EVEL RL F ORMULATION D ETAILS
Revised: A H IGH L EVEL RL F ORMULATION D ETAILS

Original: Our high-level action space is
Revised: Our high-level action space is

Original: [Equation]
Revised: [Equation]

Original: As is standard, we add a slowness parameter to the high-level network.
Revised: As is standard, we add a slowness parameter to the high-level network.

Original: That is, the high-level policy makes a choice of which low-level policy to run for the next T time-steps.
Revised: That is, the high-level policy makes a choice of which low-level policy to run for the next T time-steps.

Original: In our experiments, we choose T = 10 .
Revised: In our experiments, we choose T = 10 .

Original: So our optimization for our sparse reward task is
Revised: So our optimization for our sparse reward task is

Original: [Equation]
Revised: [Equation]

Original: B B ASIC ALGORITHM 
Revised: B B ASIC ALGORITHM Algorithm 1 Our method

Original: In Algorithm 1 above, we give the high-level algorithm we use.
Revised: In Algorithm 1 above, we give the high-level algorithm we use.

Original: We ﬁrst train each of our N low-level policies π lli for T ll steps, using our RL algorithm algo , on the low-level objective and environment.
Revised: We ﬁrst train each of our N low-level policies π lli for T ll steps, using our RL algorithm algo , on the low-level objective and environment.

Original: We then train our high-level policiy π hl for T hl using RL algorithm algo on the high-level objective and using the high-level environment.
Revised: We then train our high-level policiy π hl for T hl using RL algorithm algo on the high-level objective and using the high-level environment.

Original: Remember that our high-level policy takes our N low-level policies as input and trains to choose which one to run.
Revised: Remember that our high-level policy takes our N low-level policies as input and trains to choose which one to run.

Original: C I MPLEMENTATION D ETAILS AND H YPERPARAMETERS
Revised: C I MPLEMENTATION D ETAILS AND H YPERPARAMETERS

Original: For all RL algorithms in the paper (except for DQN in Figure 7b and the values from other works in Figure 3 and Figure 7a), we used the implementation of Kostrikov (2018).
Revised: For all RL algorithms in the paper (except for DQN in Figure 7b and the values from other works in Figure 3 and Figure 7a), we used the implementation of Kostrikov (2018).

Original: For DQN, we wrote our own simple implementation.
Revised: For DQN, we wrote our own simple implementation.

Original: The hyperparameters for these three algorithms are shown in Tables 1, 2 and 3 We use the ADAM (Kingma & Ba, 2014) optimizer.
Revised: The hyperparameters for these three algorithms are shown in Tables 1, 2 and 3 We use the ADAM (Kingma & Ba, 2014) optimizer.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D L OW - LEVEL TRAINING AND NETWORK DETAILS
Revised: D L OW - LEVEL TRAINING AND NETWORK DETAILS

Original: During low-level training we train 80 policies using different random seeds.
Revised: During low-level training we train 80 policies using different random seeds.

Original: This is to give uslow-level policies for 5 variance runs of our high-level policy.
Revised: This is to give uslow-level policies for 5 variance runs of our high-level policy.

Original: The variance plots in Figure 1 use all 80 runs.
Revised: The variance plots in Figure 1 use all 80 runs.

Original: For our Ant models, we use a 3-layer MLP with tanh activation functions and a hidden size of 32 .
Revised: For our Ant models, we use a 3-layer MLP with tanh activation functions and a hidden size of 32 .

Original: For Humanoid we add skip connections between layers and decrease the hidden size to 16 .
Revised: For Humanoid we add skip connections between layers and decrease the hidden size to 16 .

Original: For all models, we use a vector of size 16 for our phase learned parameters b φ .
Revised: For all models, we use a vector of size 16 for our phase learned parameters b φ .

Original: We choose the cyclic constraint multipliers for state ( λ s ) and action ( λ a ) to be 0 . 05 and 0 .
Revised: We choose the cyclic constraint multipliers for state ( λ s ) and action ( λ a ) to be 0 . 05 and 0 .

Original: 01 respectively.
Revised: 01 respectively.

Original: E H IGH - LEVEL TRAINING
Revised: E H IGH - LEVEL TRAINING

Original: For all high-level result plots, we run 5 independent runs with different random seeds for each result.
Revised: For all high-level result plots, we run 5 independent runs with different random seeds for each result.

Original: For any model using pre-trained low-level policies, we use different random low-level policies for each run.
Revised: For any model using pre-trained low-level policies, we use different random low-level policies for each run.

Original: For our method, we use N = 16 low-level policies for our high-level training.
Revised: For our method, we use N = 16 low-level policies for our high-level training.

Original: We give the methods the goal index as a one-hot vector to the high-level policies and baselines.
Revised: We give the methods the goal index as a one-hot vector to the high-level policies and baselines.

Original: F S UCCESS RATES ON MAZE TASKS
Revised: F S UCCESS RATES ON MAZE TASKS

Original: To give another way of looking at the results on the maze tasks, we show below the average success rate on our three random goal maze environments averaged across runs across the last 100 episodes of training.
Revised: To give another way of looking at the results on the maze tasks, we show below the average success rate on our three random goal maze environments averaged across runs across the last 100 episodes of training.

Original: [Table]
Revised: [Table]