C OMPOSING T ASK -A GNOSTIC P OLICIES WITH D EEP R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines.
To date, there has been plenty of work on learning task-speciﬁc policies or skills but almost no focus on composing necessary, task-agnostic skills to ﬁnd a solution to new problems.
In this paper, we propose a novel deep reinforcement learning-based skill transfer and composition method that takes the agent’s primitive policies to solve unseen tasks.
We evaluate our method in difﬁcult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible or exhibits high sample complexity.
We show that our method not only transfers skills to new problem settings but also solves the challenging environments requiring both task planning and motion control with high data efﬁciency.
I NTRODUCTION
Compositionality is the integration of primitive functions into new complex functions that can further be composed into even more complex functions to solve novel problems (Kaelbling & LozanoP ´ erez, 2017).
Evidence from neuroscience and behavioral biology research shows that humans and animals have the innate ability to transfer their basic skills to new domains and compose them hierarchically into complex behaviors (Rizzolatti et al., 2001).
In robotics, the primary focus is on acquiring new behaviors rather than composing and re-using the already acquired skills to solve novel, unseen tasks (Lake et al., 2017).
In this paper, we propose a novel policy ensemble composition method 1 that takes the basic, taskagnostic robot policies, transfers them to new complex problems, and efﬁciently learns a composite model through standard- or hierarchical-RL (Lillicrap et al., 2015; Schulman et al., 2015; 2017;
Haarnoja et al., 2018b; Dayan & Hinton, 1993; Vezhnevets et al., 2017; Florensa et al., 2017;
Nachum et al., 2018).
Our model has an encoder-decoder architecture.
The encoder is a bidirectional recurrent neural network that embeds the given skill set into latent states.
The decoder is a feed-forward neural network that takes the given task information and latent encodings of the skills to output the mixture weights for skill set composition.
We show that our composition framework can combine the given skills both concurrently ( and -operation) and sequentially ( or -operation) as per the need of the given task.
We evaluate our method in challenging scenarios including problems with sparse rewards and benchmark it against the state-of-the-art standard- and hierarchical- RL methods.
Our results show that the proposed composition framework is able to solve extremely hard
RL-problems where standard- and hierarchical-RL methods are sample inefﬁcient and either fail or yield unsatisfactory results.
R ELATED W ORK
In the past, robotics research has been primarily focused on acquiring new skills such as Dynamic
Movement Primitives (DMPs) (Schaal et al., 2005) or standard reinforcement learning policies.
A lot of research in DMPs revolves around learning compact, parameterized, and modular representations of robot skills (Schaal et al., 2005; Ijspeert et al., 2013;
Paraschos et al., 2013; Matsubara et al., 2011).
However, there have been quite a few approaches that address the challenge of composing
DMPs in an efﬁcient, scalable manner.
To date, DMPs are usually combined through human-deﬁned heuristics, imitation learning or planning (Konidaris et al., 2012; Muelling et al., 2010; Arie et al., 2012; Veeraraghavan & Veloso, 2008; Zoliner et al., 2005).
Likewise, RL (Sutton & Barto, 2018) research is also centralized around learning new policies (Lillicrap et al., 2015; Schulman et al., 2015; 2017; Haarnoja et al., 2018b) for complex decision-making tasks by maximizing humandeﬁned rewards or intrinsic motivations (Silver et al., 2016; Qureshi et al., 2017; 2018; Levine et al., 2016).
To the best of the authors’ knowledge, there hardly exists approaches that simultaneously combine and transfer past skills into new skills for solving new complicated problems.
For instance, Todorov (2009), Haarnoja et al.
(2018a)
and Sahni et al.
(2017) require humans to decompose high-level tasks into intermediate objectives for which either Q-functions or policies are obtained via learning.
The high-level task is then solved by merely maximizing the average intermediate Q-functions or combining intermediate policies through temporal-logic.
Note that these approaches do not combine task-agnostic skills thus lack generalizability and the ability to transfer skills to the new domains.
A recent and similar work to ours is a multiplicative composition policies (MCP) framework (Peng et al., 2019).
MCP comprises i) a set of trainable Gaussian primitive policies that take the given state and proposes the corresponding set of action distributions and ii) a gating function that takes the extra goal information together with the state and outputs the mixture weights for composition.
The primitive policies and a gating function trained concurrently using reinforcement learning. In their transfer learning tasks(Peng et al., 2019), the primitive polices parameters are kept ﬁxed, and the gating function is trained to output the mixture weights according to the new goal information. In our ablation studies, we show that training an MCP like-gating function that directly outputs the mixture weights without conditioning on the latent encoding of primitive actions gives inferior performance compared to our method. Our method utilizes all information (states, goals, and primitive skills) in a structured way through attention framework, and therefore, leads to better performance.
Recent advancements lead to Hierarchical RL (HRL) that automatically decomposes the complex tasks into subtasks and sequentially solves them by optimizing the given objective function (Dayan & Hinton, 1993;
Vezhnevets et al., 2017; Nachum et al., 2018).
In a similar vein, the options framework (Sutton et al., 1999; Precup, 2000) is proposed that solves the given task through temporal abstraction.
Recent methods such as option-critic algorithm (Bacon et al., 2017) simultaneously learns a set of sub-level policies (options), their termination functions, and a high-level policy over options to solve the given problem.
Despite being an exciting step, the option-critic algorithm is hard to train and requires regularization (Vezhnevets et al., 2016;
Harb et al., 2018), or else it ends up discovering options for every time step or a single option for the entire task.
In practice, the sub-level options or objectives obtained via HRL are inherently task-speciﬁc and therefore cannot be transferred to new domains.
B ACKGROUND
We consider a standard RL formulation based on Markov Decision Process (MDP) deﬁned by a tuple {S , A , P , R} , where S and A represent the state and action space, P is the set of transition probabilities, and R denotes the reward function.
At time t ≥ 0 , the agent observes a state s t ∈ S and performs an action a t ∈ A .
The agent’s action a t transitions the environment state from s t ∈ S to s t +1 ∈ S with respect to the transition probability P ( s t
+1 | s t , a t ) and leads to a reward r t ∈ R .
For compositionality, we extend the standard RL framework by assuming that the agent has access to the ﬁnite set of primitive policies
Π = { π i } Ni =
that could correspond to agent’s skills, controller, or motor-primitives.
Our composition model is agnostic to the structure of primitive policy functions, but for the sake of this work, we assume that each of the sub-policies { π i } Ni = solves the MDP deﬁned by a tuple { ˆ S , A , ˆ P , ˆ R i } .
Therefore, ˆ S , ˆ P and ˆ R
i are the state-space, transition probabilities and rewards of the primitive policy π i , respectively.
Each of the primitive policies π
i : ˆ S ×
A → [0 , 1] , ∀ i ∈ [0 , 1 , · · · , N ] , takes a state ˆ s ∈ ˆ S and outputs a distribution over the agent’s action space A .
We deﬁne our composition model as a composite policy π cθ : S
×
A N +1 × A → [0 , 1] , parameterize by θ , that outputs a distribution over the action space conditioned on the environment’s current state s ∈ S and the primitive policies { ˆ a
i ∈ A} Ni =0 ∼ Π .
The state space of the composite model is S =
[ ˆ S , G ] .
The space G could include any task spe-
𝑎 𝑡 ∼ ෍ 𝑖= 𝑁
𝑤 𝑖 𝒩(𝜇 𝑖 ,𝜎 𝑖 )
Decoder
Encoder
Attention Ƹ𝑠 𝑡 Ƹ𝑠 𝑡 ℎ 𝑓 ℎ 𝑏 𝑎𝑎ℎ 𝑁−1𝑏 ℎ 𝑁 𝑏 ℎ 1 𝑓 ℎ 0𝑓 𝜋𝜋
Primitive Policies 𝑎 𝑁 Ƹ𝑠 𝑡 ℎ 0𝑏 ℎ 𝑓 ℎ 𝑁𝑓 𝜋 𝑁 𝑠 𝑡 :[ Ƹ𝑠 𝑡 𝑔 𝑡 ] ℎ 𝑑 … … …
[𝑤 0 ,𝑤 1 ,…,𝑤 𝑁 ] ciﬁc information such as target locations.
Hence, in our framework, the state inputs to the primitive policies Π and composite policy π cθ need not to be the same.
In remainder of this section, we show that our composition model solves an MDP problem. To avoid clutter, we assume that both primitive policy ensemble and composite policy have the same state space S , i.e., G = ∅ . The composition model samples an action from a distribution parameterized by the actions of sub-level policies and the state s ∈ S of the environment. Therefore, we can augment the given state space S as S c : A N +1 × S , where A N +1 : {A i } Ni =0 are the outputs of sub-level policies.
Hence, compositional MDP is deﬁned as {S c , A , P c , R} where S c =
A N × S is the new composite state-space, A is the action-space, P c :
S c
×
S
c
×
A → [0 , 1] is the transition probability function, and R is the reward function for the given task.
P OLICY E NSEMBLE C OMPOSITION
In this section, we present our policy ensemble composition framework, shown in Fig.
Our composition model consists of i) the encoder network that takes the outputs of primitive policies and embeds them into latent spaces; ii) the decoder network that takes current state s t of the environment and the latent embeddings from the encoder network to parameterize the attention network; iii) the attention network that outputs the probability distribution over the primitive low-level policies representing their mixture weights.
The remainder of the section explains the individual models of our composition framework and the overall training procedure.
4. E NCODER N ETWORK
Our encoder is a bidirectional recurrent neural network (BRNN) that consists of Long Short-Term
Memory units (Hochreiter & Schmidhuber, 1997).
The encoder takes the outputs of the policy ensemble { ˆ a i } Ni =0 and transform them into latent states of forward and backward RNN, denoted as { h fi } N +1 i =
and { h bi } N +1 i =0 , respectively, where h fi , h bi ∈ R d ; ∀ i ∈ [0 , 1 , · · · , N + 1] .
The N +states of forward and backward RNN corresponds to their last hidden states denoted as h f and h b , respectively, in Fig.
4. D ECODER N ETWORK
Our decoder is a simple feed-forward neural network that takes the last hidden states of the forward and backward encoder network, i.e., { h f , h b } , and the current state of the environment s to map them into a latent space h ∈ R d .
The state input to the decoder network is deﬁned as s : [ˆ s, g ] , where ˆ s ∈ ˆ S is the state input to the low-level policy ensemble and g ∈ G could be any additional information related to the given task, e.g., goal position of the target to be reached by the agent.
4. A TTENTION N ETWORK
The composition weights (see Fig. 1) { w i ∈ [0 , 1] } Ni =0 are determined by the attention network as follows:
[Equation]
where W f , W b , W d ∈ R d × d and W ∈ R d .
The weights { w i } Ni =0 for the composite policy are computed using gumbel-softmax denoted as softmax(q / T) , where T is the temperature term (Jang et al., 2016).
4. C OMPOSITE POLICY
Given the primitive policy ensemble Π =
{ π i } Ni =0 , the composite action is the weighted sum of all primitive policies outputs, i.e., π cθ = (cid:80)
Ni w i π i .
Since, we consider the primitive policies to be
Gaussian distributions, the output of each primitive policy is parameterized by mean µ and variance σ , i.e., { ˆ a i ∼ N ( µ i , σ i ) } Ni =
← { π i } Ni =0 .
Hence, the composite policy can be represented as π cθ = (cid:80)
Ni w i N ( µ i , σ i ) , where N ( · ) denotes Gaussian distribution, and (cid:80) i w i = 1 .
Given the mixture weights, other types of primitive policies, such as DMPs (Schaal et al., 2005), can also be composed together by the weighted combination of their normalized outputs.
4. C OMPOSITE MODEL TRAINING OBJECTIVE
The general objective of RL methods is to maximize the cumulative expected reward, i.e., J ( π cθ ) =
E π cθ [ (cid:80) ∞ t =0 γ t r t ] , where γ : (0 , 1] is a discount factor.
We consider the policy gradient methods to update the parameters θ of our composite model, i.e., θ ← θ + η (cid:53) θ J ( π cθ ) , where η is the learning rate.
We show that our composite policy can be trained through standard RL and HRL methods, described as follow.
4.5. S TANDARD R EINFORCEMENT LEARNING
In standard RL, the policy gradients are determined by either on-policy or off-policy updates (Lillicrap et al., 2015; Schulman et al., 2015; 2017; Haarnoja et al., 2018b) and any of them could be used to train our composite model.
However, in this paper, we consider off-policy soft-actor critic (SAC) method (Haarnoja et al., 2018b) for the training of our policy function.
SAC maximizes the expected entropy H ( · ) in addition to the expected reward, i.e.,
[Equation]
where λ is a hyperparameter.
We use SAC as it motivates exploration and has been shown to capture the underlying multiple modes of an optimal behavior.
Since there is no direct method to estimate a low-variance gradient of Eq (2), we use off-policy value function-based optimization algorithm (for details refer to Appendix A.1 of supplementary material).
4.5. H IERARCHICAL R EINFORCEMENT L EARNING
In HRL, there are currently two streams - task decomposition through sub-goals (Nachum et al., 2018) and option framework (Bacon et al., 2017) that learns temporal abstractions.
In the options framework, the options can be composite policies that are acquired with their termination functions.
In task decomposition methods that generate sub-goal through high-level policy, the low-level policy can be replaced with our composite policy.
In our work, we use the latter approach (Nachum et al., 2018), known as HIRO algorithm, to train our policy function.
Like, standard HIRO, we use two level policy structure.
At each time step t , the high-level policy π hi θ (cid:48) , with parameters θ (cid:48) , observes a state s t and takes an action by generating a goal g t ∈ S in the state-space S for the composite low-level policy π c : low θ to achieve.
The π c : low θ takes the state s t , the goal g t , and the primitive actions { ˆ a i } N 0 to predict a composite action a t through which an agent interacts with the environment.
The high-level policy is trained to maximize the expected task rewards given by the environment whereas the composite low-level policy is trained to maximize the expected intrinsic reward deﬁned as the negative of distance between current and goal states, i.e., [Figure] (cid:107) s t + g t − s t +1 (cid:107) 2 .
To conform with HIRO settings, we perform off-policy correction of the highlevel policy experiences and we train both high- and low-level policies via TD3 algorithm (Fujimoto et al., 2018)
(for details refer to Appendix A.2 of supplementary material).
E XPERIMENTS AND R ESULTS
We evaluate and compare our method against standard RL, and HRL approaches in challenging environments (shown in Fig. 2) that requires complex task planning and motion control.
The implementation details of all presented methods and environment settings are provided in Appendix C of supplementary material.
We also do an ablative study in which we take away different components of our composite model to highlight their importance.
Furthermore, we depict attention weights of our model in a navigation task to highlight its ability of concurrent and sequential composition.
We consider the following seven environments for our analysis: (1) Pusher: A simple manipulator has to push an object to a given target location.
(2) Ant Random Goal: In this environment, a quadruped-Ant is trained to reach the randomly sampled goal location in the conﬁned circular region.
(3) Ant Cross Maze: The cross-maze contains three target locations.
The task for a quadruped Ant is to reach any of the three given targets by navigating through a 3D maze without collision.
HalfCheetah Hurdle:
In this problem, the task for a halfcheetah is to run and jump over the three barriers to reach the given target location.
(5) Ant Maze: A ⊃ -shaped maze poses a challenging navigation task for a quadruped-Ant.
In this task, the agent is given random targets all along the maze to reach while training.
However, during the evaluation, we test the agent for reaching
In all tasks, we also acquire primitive skills of the agent for our composite policy.
For Ant, we use four basic policies for moving left, right, up, and down.
The pusher uses two primitive policies that are to push an object to the left and down.
In HalfCheetah hurdle environment, the low-level policies include jumping and running forward.
Finally fot the point-mass robot, the composition model takes four policies for moving in the up, down, left and right directions.
Furthermore, in all environments, except pusher, the primitive policies were agnostic of high-level tasks ( such as target locations) that were therefore provided separately to our composite model via decoder network.
This highlights the ability of our model to transfer basic robot skills to novel problems.
[Table]
Our
[Table] [Table]
D i s t a n c e f r o m g
o a l
Ant Random Goal 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Steps (Million) 0.
D i s t a n c e f r o m g
o a l
Ant Cross Maze
PPO
TRPO
Composition-SAC
SAC
5. C OMPARATIVE STUDY
In our comparative studies, we divide our test environments into two groups.
The ﬁrst group includes
Pusher, Random Goal Ant, Ant Cross Maze, and HalfCheetah-Hurdle environments, whereas the second group comprises the remaining environments that require task and motion planning under weak reward signals.
In the ﬁrst group of settings, we compare our composite model trained with SAC (Haarnoja et al., 2018b) against the standard Gaussian policies obtained using SAC (Haarnoja et al., 2018b), PPO (Schulman et al., 2017), and TRPO (Schulman et al., 2015).
We exclude HRL methods in these cases as the environment rewards sufﬁciently represent the underlying task, whereas HRL approaches are applicable in cases that have a weak reward signal or require task and motion planning.
Tablepresents the mean and standard deviation of the agent’s ﬁnal distance from the given targets after the end of an evaluation rollout over the ten trials.
Fig.
3 shows the mean learning performance over all trials during the three million training steps.
In these set of problems, TRPO and PPO entirely fail to reach the goal, and SAC performs reasonably well but only in simple Ant Random Goal and Pusher environments as it fails in other cases.
Our composite policy obtained using SAC successfully solves all tasks and exhibit high data-efﬁciency by learning in merely a few thousand training steps.
In our second group of environments, we use distance-based rewards that are weak signals as greedily following them does not lead to solving the problem.
Furthermore, in these environments, policies trained with standard RL, including our composite policy, failed to solve the problem even after 20 million training steps.
Therefore, we trained our composite policy with HIRO (Nachum et al., 2018) and compared its performance against standard HIRO formulation (Nachum et al., 2018).
We also tried to include option-critic framework (Bacon et al., 2017), but we were unable to get any considerable results with their online implementation despite several attempts with the parameter tuning.
One of the reasons option-critic fails is because it relies purely on task rewards to learn, which makes them inapplicable for cases with weak reward signals (Nachum et al., 2018). Nachum et al.
(2018) use a modiﬁed Ant in their paper that has 30 units joint torque limit (lowtorque-Ant). For our composition method, we use Mujoco standard Ant that has a torque limit
[Figure]
[Figure]
[Figure]
[Figure]
D i s t a n c e f r o m g o a l [Figure] of 150 units which makes the learning even harder as the Ant is now more prone to instability than a low-torque-Ant.
For standard HIRO formulation, we trained three variants i)
HIRO trained on standard Mujoco Ant, ii) HIRO trained on low-torque Ant (as in (Nachum et al., 2018)); iii)
HIRO pretrained for the equal number of training steps as used to train the primitive policies of our composition method on a low-torque-Ant (For more details, refer to Appendix C.1).
Fig.
4 shows the learning performance, averaged over ten trials, during 10 million steps.
It can be seen that the composite policy with HIRO outperforms standard HIRO (Nachum et al., 2018) by a signiﬁcant margin.
It also certiﬁes the utility of solving RL tasks using composition by leveraging basic pre-acquired skills.
Furthermore, HIRO performs poorly with standard Ant, even pretraining did not improve the performance, as it imposes a harder control problem.
5. A BLATIVE STUDY
We remove bidirectional RNN (BRNN) (similar to (Peng et al., 2019)), attention-network, and both attention-network and BRNN (AttBRNN) from our composition model to highlight their importance in the proposed architecture in solving complex problems.
We train all models with SAC (Haarnoja et al., 2018b).
The ﬁrst model is our composite policy without attention in which the decoder network takes the state information and last hidden states of the encoder (BRNN) to directly output actions rather than mixture weights.
The second model is without attention network and BRNN, it is a feed-forward neural network that takes the state information and the primitive actions and predicts the action to interact with the environment.
The third model is without BRNN, it is a feedforward neural network that takes the state and goal information, and output the mixture weights.
The mixture weights are then used to combine the actions from primitive policies.
This setting is same as (Peng et al., 2019) for their transfer learning problems.
Fig.
5 shows the mean performance comparison, over ten trials, of our composite model against its ablated versions on a Ant Random Goal, Cross Maze Ant, and Pusher environment.
We exclude remaining test environments in this study as ablated models completely failed to perform or show any progress.
Note that the better performance of our method compared to ablated versions highlight the merits of our architecture design.
Intuitively, BRNN allows the dynamic encoding of a skill set that could be of variable lengths 2 .
And our decoder network uses the encoded skills together with given state and goal information (states, goals, and primitive skills) in a structured way using attention framework and provides signiﬁcantly better performance than other ablated models, including (Peng et al., 2019).Furthermore, another merit of using the attention network is that it bypasses the complex transformation of action embeddings (composition-without-attention) or actions and state-information (composition-without-AttBRNN) directly to action space.
Hence, the proposed architecture is crucial for the composition of task-agnostic sub-level policies to solve new problems.
5. D EPICTION OF ATTENTION WEIGHTS
In order to further assess the merit of utilizing an attention network, we apply our model to a simple 2D multi-goal point-mass environment as shown in Fig.
The point-mass is initialized around the origin (with variance σ 2 = 0 . 1 ) and must randomly choose one of four goals to reach.
For this experiment we use dense rewards with both a positional and actuation cost.
Primitive policies of up ( + y ), down ( − y ), left ( − x ), and right ( + x ) were trained and composed to reach goals, represented here as red dots, in the “diagonal” directions where a combination of two or more primitive policies are required to reach each goal.
The four mappings in the ﬁgure give us insight into how the attention network is utilizing the given primitives to achieve the desired task.
At each step in a given path, the weights { w i } Ni = for each low-level policy are assigned and composed together to move the point-mass in the desired direction.
We see here that even with some noise and short-term error, the attention weights are strongest for primitive policies that move the point-mass to its chosen goal.
We also see that multiple policies are activated at once to achieve more direct movements toward the goal, as opposed to “stair-stepping” where only one primitive is activated at a time.
Both of these observations point to the concurrent and sequential nature of this composition model.
Please refer to Appendix D for depiction of attention weights in other complicated environments.
C ONCLUSIONS AND F UTURE WORK
We present a novel policy ensemble composition method that combines a set of independent and task-agnostic primitive policies through reinforcement learning to solve the given tasks.
We show that our method can transfer the given skills to novel problems and can compose them both sequentially ( or -operation) and concurrently ( and -operation) to ﬁnd a solution for the task in hand.
Our experiments highlight that composition is vital for solving problems requiring complex motion skills and decision-making where standard reinforcement learning and hierarchical reinforcement learning methods either fail or need a massive number of interactive experiences to achieve the desired results.
In our future work, we plan to extend our method to automatically acquire the missing skills in the given skillset that are necessary to solve the speciﬁc problems (refer to Appendix B for preliminary results).
We also aim towards a system that learns the hierarchies of composition models by combining primitive policies into complex policies that would further be composed together for a combinatorial outburst in the agent’s skillset.
A C OMPOSITE M ODEL T RAINING A LGORITHMS A. T RAINING WITH S OFT A CTOR -C RITIC
In this section, we brieﬂy describe the procedure to train our composition model using SAC (Haarnoja et al., 2018b).
Although any RL method can be used to optimize our model, we use SAC as it is reported to perform better than other training methods.
Our composite policy is a tractable function π cθ ( a t | s t , { π i } Ni =0 ) parameterized by θ .
The composite policy update through SAC requires the approximation of Q- and value-functions.
The parametrized value- and Q-function are denoted as V φ ( s t ) with parameters φ , and Q ξ ( s t , a t ) with parameters ξ , respectively.
Since, SAC algorithm build on the soft-policy iteration, the soft value-function V φ ( s t ) and soft Q-function Q ξ ( s t , a t ) are learned by minimizing the squared residual error J V ( φ ) and squared Bellman error J Q ( ξ ) , respectively, i.e.,
[Equation]
where M is a replay buffer, ˆ V ( s t ) =
E a t ∼ π cθ [ Q ξ ( s t , a t ) − log π cθ ( a t
| s t )] and ˆ Q is the Bellman target computed as follows:
[Equation]
The function V φ (cid:48)
( s t ) is the target value function with parameters φ (cid:48) .
The parameters φ (cid:48) are the moving average of the parameters φ computed as τφ + (1 − τ ) φ (cid:48) , where τ is the smoothing coefﬁcient.
Finally the policy parameters are updated by minimizing the following expected KL-divergence.
[Equation]
where Z ξ is a partition function that normalizes the distribution.
Since, just-like SAC, our Q-function is differentiable, the above cost function can be determined through a simple reparametization trick, see Haarnoja et al.
(2018b)
for details.
Like SAC, we also maintain two Q-functions that are trained independently, and we use the minimum of two Q-functions to compute Eqn. 3 and Eqn. 6.
This way of using two Q-function has been shown to alleviate the positive biasness problem in the policy improvement step.
The overall training procedure is summarized in Algorithm 1.
A. T RAINING WITH HIRO
In this section, we outline the algorithm to train composite policy through HIRO that employs the two level policy structure.
The high-level policy generates the sub-goals for the low-level composite
Initialize parameter vectors φ , φ (cid:48) , θ , ξ
Input: Primitive policies Π
=
{ π i } Ni = for each iteration do for each environment step do
Compute primitive policies state ˆ s t ← s t \ g t
Sample primitive actions { ˆ a i,t } Ni =
∼ Π(ˆ s t )
Sample high-level action g t ∼ π hi (ˆ s t )
Sample composite action a t ∼ π cθ ( a t | s t , g t , { ˆ a i,t } Ni =0 )
Sample next state s t +1 ∼ p ( s t +1 | s t , a t )
M ← M ∪ { ( s t , g t , a t , { ˆ a i,t } Ni =0 , r t , s t +1 ) } for each gradient step do
Sample mini-batch with c-step transitions { ( g k , s kt : t + c , a kt : t + c − 1 , { ˆ a kj,t : t
+ c − 1 } Nj =0 , r kt : t + c − 1 ) }
Bk =1 ∼ M
Compute rewards for low-level policy { r loi } Bi =
← { r lo ( s i , g i , s i +1 ) }
Bi =
Update π c : lo w.r.t Q lo using { ( s k , g k , a k , { ˆ a i,k } N i =0 , r lok , s k +1 ) } B − 1 k =
(Nachum et al., 2018)
Update π hi w.r.t Q hi using { ( s k , ˆ g k , (cid:80) c − 1 i =
k r k , s k + c )
} B − 1 k =
(Nachum et al., 2018) policy to achieve.
The low-level composite policy also have access to the primitive policy actions.
Like HIRO, we use TD3 algorithm (Fujimoto et al., 2018) to train both high-level and low-level policies with their corresponding Q-functions, Q hi and Q lo , respectively.
The low-level policy π c : low θ , with parameters θ , is trained to maximize the Q-values from the low-level Q-function Q lo for the given state-goal pairs.
The Q-function ( Q lo ) parameters ξ are optimized by minimizing temporal-difference error for the given transitions, i.e.,
[Equation]
) where r lo ( s t , g t , s t +1 )
= −(cid:107) s t
+ g t − s t +1 (cid:107) and g t +1 ∼ π hiθ (cid:48)
( s t +1 ) .
The high-level policy π hiθ (cid:48) , with parameters θ (cid:48) , is trained to maximize the values of Q hi .
The Qfunction ( Q hi ) parameters ξ (cid:48) are trained through minimizing the following loss for the given transitions.
[Equation]
During training, the continuous adaptation of low-level policy poses a non-stationery problem for the high-level policy.
To mitigate the changing behavior of low-level policy, Nachum et al.
(2018) introduced off-policy correction of the high-level actions.
During correction, the high-level policy action g is usually re-labeled with ˆ g that would induce the same low-level policy behavior as was previously induced by the original high-level action g (for details, refer to (Nachum et al., 2018)).
Algorithm 2 presents the procedure to train our composite policy with HIRO.
B F UTURE R ESEARCH D IRECTIONS & D ISCUSSION
In this section, we present preliminary results to highlight the potential research avenues to further extend our method.
Apart from compositionality, another research problem in the way of building intelligent machines is the autonomous acquisition of new skills that were lacking in the system and therefore, hindered it from the solving the given task (Lake et al., 2017).
In this section, we demonstrate that our composition model holds the potential for building such a system that can simultaneously learn new missing skills and compose them together with the existing skill set to solve the given problem.
Fig.
7 shows a simple ant-maze environment in which the 3D quadruped-Ant has to reach the target, indicated as a green region.
In this problem, we equip our composition model with a single primitive
3.0 Steps (Million) 0.
D i s t a n c e f r o m g o a l Composition-SACSAC policy for walking in the right direction and a trainable, parametric policy function.
The trainable policy function takes the state without information and outputs the parameters for Gaussian distribution to model the robot’s action space.
Note that the composite model requires a skill of walking upward in addition to moving in the right direction to reach the given goal.
To determine if our composition model can teach the new parametric policy function to move upward, we trained our composition model along with the new policy using the shared Q- and value-functions.
Once trained, we observed that our composition model learned the parametric policy function for walking in an upward direction with slight right-turning behavior.
The plot in Fig.
7 shows the performance of our composition model and standard RL policy obtained with SAC in this environment.
The vertical axis indicates the distance from the target location averaged over ﬁve randomly seeded trials.
The horizontal axis shows the number of environment steps in millions.
It can be observed that our method converged faster than standard RL policy.
It shows that our approach is also utilizing the existing skills and therefore learns faster than a standard policy function that solves the entire task from scratch.
Furthermore, in current settings, we do not impose any constraint on the new policy function that would make it learn only the missing skills rather than replicating existing skills or solving the given task entirely by itself.
We leave the formulation of such constraint for autonomous acquisition of new, missing skills to our future works.
C I MPLEMENTATION DETAILS C. P RETRAINING B ENCHMARK M ETHOD
In our results, we also consider the pretraining of benchmark methods for an equivalent amount of time as used to train the primitive policies for our composition model for a better analysis. The total number of steps needed to train primitive skills of Ant (up, down, left, and right), halfcheetah (run and jump), and pusher (down and left) were 1, 0.5, and 0.1 million, respectively. In Fig. 4, HIRO was pretrained for both standard (30 units) and low-torque-ant (150 units).
The results of HIRO standardant are excluded in Fig.4 to avoid clutter as they led to no improvement in the performance.
For the non-hierarchical tasks presented in Fig.
3, the benchmark methods are not pretrained.
However, to account for pretraining, the performance of other methods (TRPO, PPO, SAC) can be accessed after 1, 0.5, and 0.1 million for the ant, halfcheetah, and pusher environments, respectively.
Also, notice that in Fig. 3, the pretraining will have no signiﬁcant effect on the performance of TRPO and PPO in all environments and SAC in half-cheetah hurdle and cross-maze-ant environments.
Furthermore, since pusher and random-goal-ant are relatively simple environments (due to no obstacles or maze around), pretrained SAC can perform similar to our composition method.
C. E NVIRONMENT D ETAILS
In this section, we present the environment details including reward functions, primitive policies, and state space information.
The reward functions are presented in the Table 3 together with the overall reward scaling values.
For Ant and halfcheetah-hurdle environments, the primitive policies were trained in a free-space that had no mazes, hurdle or ﬁxed targets in the surrounding.
C.2. A NT ENVIRONMENTS
In these environments, we use 8 DOF four-legged Ant with 150 units torque limit.
The primitive policies of moving left, right, down and up were shared across all these tasks.
These sub-levelpolicies were obtained using SAC and each policy was trained for about 0.25 million steps.
In these environments, the information g in the state s : [ˆ s, g ] corresponds to the target location.
Let us introduce the notation to deﬁned reward function.
Let r xy , g xy , u , and f c denote xy-position of the robot’s torso, xy-position of the goal, joint torques, and contact-cost, respectively.
The scaling factors are deﬁned as λ .
The reward function for the following environments is deﬁned as with reward scaling of 5 units:
[Equation]
Ant Random Goal:
In this environment, the ant has to navigate to any randomly sampled target within the conﬁned circular region of radius 5 units.
The goal radius is deﬁned to be 0.25 units.
The reward function coefﬁcients λ g , λ v , λ s , λ ct , and λ c are 0 .
01 , and 0 .
001 , respectively.
Ant Cross Maze: In this environment, the ant has to navigate through the 3D maze to reach any of the target sampled from the three targets.
The goal radius is deﬁned to be 1.0 units.
The reward function parameters are same as for the random-goal ant environment.
For the remaining environment (Ant Maze, Ant Push and Ant Fall), we use the following reward function with no reward scaling: [Equation]
[Equation]
where coefﬁcients λ g , λ ct , and λ c are set to be 1 . 0 , 0 . 05 , and 0 .
5 × 10 − 4 .
Ant Maze: In this environment, we place the Ant in a ⊃ -shaped maze for a navigation task between given start and goal conﬁgurations.
The goal radius is deﬁned to be 5 units.
During training, the goal is uniformly sampled from [ − 4 , 20] ×
[ − 4 , 20] space, and the Ant initial location is always ﬁxed at (0 , 0) .
During testing, the agent is evaluated to reach the farthest end of the maze located at (0 , 19) within L2 distance of 5.
Ant Push: In this environment, the Ant is initially located at (0 , 0) coordinate, the moveable block is at (0 , 8) , and the goal is at (0 , 19) .
The agent is trained to reach randomly sampled targets whereas during testing, we evaluate the agent to reach the goal at (0 , 19) within L2 distance of 5.
Ant Fall:
In this environment, the Ant has to navigate in a 3D maze.
The initial agent location is (0 , 0) , and a movable block is at (8 , 8) at the same elevation as Ant.
Their is a rift in the region [ − 4 , 12] ×
To reach the target on the other side of the rift, the Ant must push the block down into the rift, and then step on it to get to the goal position.
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
C.2. P USHER
In pusher environment, a simple manipulator has to move an object to the target location.
The primitive policies were to push the object to the bottom and left.
These low-level policies were obtained via SAC after 0.1 million training steps.
In this environment, the state information for both primitive policies and the composite policy include the goal location.
Therefore, G , in this case, is null.
The reward function is given as:
[Equation]
[Table]
where o xy , g xy , r xy , and u are xy-position of object, xy-position of goal, xy-position of arm, and joint-torques.
The coefﬁcients λ g , λ o , and λ ct are 1 . 0 , 0 . 1 , and 0 . 1 , respectively.
C.2. H ALFCHEETAH - HURDLE
In halfcheetah-hurdle environment, a 2D cheetah has to jump over the three hurdles to reach the target.
The primitive policies include run forward and jump where each poilcy was trained with
SAC for 0.25 million steps.
Furthermore, in this environment, the information g in the state s : [ˆ s, g ] corresponds to the x-position of the next nearest hurdle in front of the agent as well as the distance from that hurdle.
The reward function is deﬁned as:
[Equation]
(12) where r xy , g xy , v z , and v x are xy-position of robot torso, xy-position of goal, velocity along z-axis, and velocity along x-axis, respectively.
The function hc ( · ) returns a count indicating the number of hurdles in front of the robot.
The indicator function I (goal) returns 1 if the agent has reached the target otherwise 0.
The function cc ( · ) is a collision checker which returns 1 if the agent collides with the hurdle otherwise 0.
The reward function coefﬁcients λ g , λ hc , λ rg , λ z , λ v , and λ c are 0 .
1 , 1 . 0 , 1000 , 0 . 3 , 1 . 0 and 2 , respectively.
C. H YPERPARAMETERS AND N ETWORK A RCHITECTURES
Table 2 summarizes the hyperparameters used to train policies with SAC (Haarnoja et al., 2018b),
TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), and HIRO (Nachum et al., 2018).
Table 3 summarizes the network architectures.
D D EPICTION OF A TTENTION W EIGHTS
In this section, we illustrate the attention weights of our composite policy in halfcheetah-hurdle (Fig.
9), pusher (Fig. 10), and cross-maze-ant (Fig. 11) environments.
Our results highlight the ability of our model to compose a given set of primitive polices, both sequentially and concurrently, to solve the given problem.
Fig.
8 shows the color-coding for the scale of attention weight value given by the composite policy.