H IERARCHICAL AND I NTERPRETABLE S KILL A CQUISITION IN M ULTI - TASK R EINFORCEMENT L EARNING
Anonymous
authors
Paper under double-blind review
A BSTRACT
Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL).
It is also a requirement for its deployment in real-world scenarios.
This paper proposes a novel framework for efﬁcient multi-task reinforcement learning.
Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill.
This enables agents to continually acquire new skills during different stages of training.
Each learned task corresponds to a human language description.
Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices.
In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills.
We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.
I NTRODUCTION
Deep reinforcement learning has demonstrated success in policy search for tasks in domains like game playing (Mnih et al., 2015; Silver et al., 2016; 2017; Kempka et al., 2016;
Mirowski et al., 2017) and robotic control (Levine et al., 2016a;b; Pinto & Gupta, 2016).
However, it is very difﬁcult to accumulate multiple skills using just one policy network Teh et al.
Knowledge transfer techniques like distillation (Bengio, 2012; Rusu et al., 2016; Parisotto et al., 2016;
Teh et al., 2017) have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task policy.
Existing approaches usually treat all tasks independently.
This often prevents full exploration of the underlying relations between different tasks.
They also typically assume that all policies share the same state space and action space.
This precludes transfer of previously learned simple skills to a new policy deﬁned over a space with differing states or actions.
When humans learn new skills, we often take advantage of our existing skills and build new capacities by composing or combining simpler ones.
For instance, learning multi-digit multiplication relies on the knowledge of single-digit multiplication; learning how to properly prepare individual ingredients facilitates cooking dishes based on complex recipes.
Inspired by this observation, we propose a hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills.
It achieves this by discovering the underlying relations between skills.
To represent the skills and their relations in an interpretable way, we also encode all tasks using human instructions such as “put down.”
This allows the agent to communicate its policy and generate plans using human language.
Figure 1 illustrates an example: given the instruction “Stack blue,” our hierarchical policy learns to compose instructions and take multiple actions through a multi-level hierarchy in order to stack two blue blocks together.
Steps from the top-level policy π 3 (i.e., the red branches) outline a learned high-level plan – “Get blue → Find blue → Put blue.”
In addition, from lower level policies, we may also clearly see composed plans for other tasks.
Based on policy π 2 , for instance, the task “Get blue” has two steps – “Find blue → action: turn left,” whereas “Put blue” can be executed by a single action “put down” according to π 3 .
Through this hierarchical model, we
[Figure]
t
[Figure]
None None
[Figure]
P o li c y un f o l d i n g
[Figure]
may i) accumulate tasks progressively from a terminal policy to a top-level policy and ii) unfold the global policy from top-level to basic actions.
In order to better track temporal relationships between tasks, we train a stochastic temporal grammar (STG) model on the sequence of policy selections (previously learned skill or new skill) for positive episodes.
The STG focuses on modeling priorities of tasks: for example, it is necessary to obtain an object before putting it down.
Integrating the STG into the hierarchical policy boosts efﬁciency and accuracy by explicitly modeling such commonsense world knowledge.
We validated our approach by testing it on object manipulation tasks implemented in a Minecraft world.
Our experimental results demonstrate that this framework can (i) efﬁciently learn hierarchical policies and representations for multi-task RL; (ii) learn to utter human instructions to deploy pretrained policies, improve their explainability and reuse skills; and (iii) learn a stochastic temporal grammar via self-supervision to predict future actions.
R ELATED W ORK
Multi-task Reinforcement Learning .
Previous work on multi-task reinforcement learning mainly falls into two families: knowledge transfer through distillation (Rusu et al., 2016; Parisotto et al., 2016;
Teh et al., 2017; Tessler et al., 2017) or modular policy design through 2-layer hierarchical policy (Andreas et al., 2017).
Our multi-level policy is more similar to the latter approach.
The main differences between our model and the one in Andreas et al.
(2017) are two-fold: i) we do not assume that a global task can be executed by only performing predeﬁned sub-tasks; ii) in our multi-level policy, global tasks at a lower-level layer may also be used as sub-tasks by global tasks carried out at higher-levels.
Hierarchical Reinforcement Learning .
Complex policies often require the modeling of longer temporal dependencies than what standard Markov decision processes (MDPs) can capture.
To combat this, hierarchical reinforcement learning was introduced to extend MDPs to semi-MDPs (Sutton et al., 1999), where options (or macro actions) are introduced on top of primitive actions to decompose the goal of a task into multiple subgoals.
In hierarchical RL, two sets of policies are trained: local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the ﬁnal goal of a task (Bacon &
Precup, 2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).
This two-layer hierarchical policy design signiﬁcantly improves the ability of discovering complex policies which can not be learned by ﬂat policies.
However, it also often makes some strict assumptions that limit its ﬂexibility: i) a task’s global policy cannot use a simpler task’s policy as part of its base policies; ii) a global policy is assumed to be executable by only using local policies over speciﬁc options, e.g., (Kulkarni et al., 2016; Andreas et al., 2017).
In this work, we aim to learn a multi-level global policy which does not have these two assumptions.
In addition, previous work usually uses a latent variable to represent a task.
In our work, we encode a task by a human instruction to learn a task-oriented language grounding as well as to improve the interpretability of plans composed by our hierarchical policies.
Language grounding via reinforcement learning .
Recently, there has been work on grounding human language in 3D game environments (Hermann et al., 2017; Chaplot et al., 2017) or in text-based games (Narasimhan et al., 2015) via reinforcement learning.
In these games agents are instructed to pick up an item described by a sentence.
Besides visual grounding, Andreas et al.
(2017) grounded instructions (not necessarily using human language) to local policies in hierarchical reinforcement learning.
Our approach not only learns the language grounding for both visual knowledge and policies, but is also trained to utter human instructions as an explicit explanation of its decisions to humans.
To our knowledge, this is the ﬁrst model that learns to compose plans for complex tasks based on simpler ones which have human descriptions.
M ODEL
In this section, we discuss our multi-task RL setting, hierarchical policy, stochastic temporal grammar, and how interaction of these components can achieve plan composition.
3. M ULTITASK RL S ETTING
Let G be a task set, where each task g is uniquely described by a human instruction.
For simplicity, we assume a two-word tuple template consisting of a skill and an item for such a phrase, i.e., (cid:104) u skill , u item (cid:105) .
Each tuple describes an object manipulation task.
In this paper, we deﬁne g = (cid:104) u skill , u item (cid:105) by default, thus tasks and instructions are treated as interchangeable concepts.
For each task, we deﬁne a Markov decision process (MDP) represented by states s ∈ S and primitive actions a ∈ A .
Rewards are speciﬁed for goals of different tasks, thus we use a function R ( s, g ) to signal the reward when performing any given task g .
We assume that as a starting point, we have a terminal policy π 0 (as shown in Figure 2a) trained for a set of basic tasks (i.e., a terminal task set G 0 ).
The task set is then progressively increased as the agent is instructed to do more tasks by humans at multiple stages, such that G 0 ⊂ G 1 ⊂ · · · ⊂ G K , which results in life-long learning of polices from π 0 for G 0 to π K for G K as illustrated by the “task accumulation” direction in Figure 1.
At stage k > 0 , G k − 1 is deﬁned as the base task set of G k .
The tasks in G k − 1 are named as base tasks at this stage and π k − 1 becomes the base policy of π k .
Here, we utilize weak supervision from humans to deﬁne what tasks shall be augmented to the previous task set at each new stage.
3. H IERARCHICAL P OLICY
One of our key ideas is that a new task in current task set G k may be decomposed into several simpler subtasks, some of which can be base tasks in G k − 1 executable by base policy π k − 1 .
Therefore, instead of using a ﬂat policy (Figure 2a) as π 0 that directly maps state and human instruction to a primitive action, we propose a hierarchical design (Figure 2b) with the ability to reuse the base policy (i.e., π k − 1 ) for performing base tasks as subtasks.
Namely, at stage k , the global policy π k is deﬁned by a hierarchical policy.
This hierarchy consists of four sub-policies: a base policy for executing previously learned tasks, an instruction policy that manages communication between the global policy and the base policy, an augmented ﬂat policy which allows the global policy to directly execute actions, and a switch policy that decides whether the global policy will primarily rely on the base policy or the augmented ﬂat policy.
The base policy is deﬁned to be the global policy at the previous stage k − 1 .
The instruction policy maps state s and task g ∈ G k to a base task g (cid:
) ∈
G k − 1 .
The purpose of this policy is to inform base policy π k − 1 which base tasks it needs to execute.
Since an instruction is represented by two words, we deﬁne the instruction policy using two conditionally independent distributions, i.e., π inst k ( g (cid:48) =
(cid:104) u skill , u item (cid:105)| s, g ) =
p skill k ( u skill | s, g ) p item k
( u item | s, g ) .
An augmented ﬂat policy, π aug k ( a | s, g ) , maps state s and task g to a primitive action a for ensuring that the global policy is able to perform novel
Policy ⇡ k ⇡ k
VisualEncoder
InstructionEncoder
Fusion
LSTM action “Get white”
Actions
V ( s,g ) (a) Flat policy.
SwitchPolicy
InstructionPolicy
Augmented Policy
VisualEncoder
InstructionEncoder
Fusion
LSTM
Base Policy ⇡ k  1 ⇡ k
Selector action
Global Policy ⇡ k action “Get white”
Actions
V ( s,g )
V sw ( s,e,g )
[Figure]
tasks in G k that can not be achieved by only reusing the base policy.
To determine whether to perform a base task or directly perform a primitive action at each step, the global policy further includes a switch policy, π sw k
( e | s, g ) , where e is a binary variable indicating the selection of the branches, π inst k ( e = 0 ) or π aug k ( e = 1 ).
Note that the above description of the hierarchical policy does not account for an STG.
The instruction policy and switch policy introduced here are simpliﬁed from the ones in the full model (see
Section 3.3).
At each time step, we ﬁrst sample e t from our switch policy π sw k to decide whether the global policy π k will rely on the base policy π k − 1 or the augmented ﬂat policy π aug k .
We also sample a new instruction g (cid:48) t from our instruction policy π inst k in order to sample actions from the base policy.
This can be summarized as:
[Equation]
[Equation]
and ﬁnally
[Equation]
where π k and π k − 1 are the global policies at stage k and k − 1 respectively.
After each step, we will also obtain a reward r t =
R ( s t , g ) .
3. S TOCHASTIC T EMPORAL G RAMMAR
Different tasks may have temporal relations.
For instance, to move an object, one needs to ﬁrst ﬁnd and pick up that object.
There has been previous research (Si et al., 2011; Pirsiavash & Ramanan, 2014) using stochastic grammar models to capture such temporal relations.
Inspired by this, we summarize temporal transitions between various tasks with a stochastic temporal grammar (STG).
In our full model, the STG interacts with the hierarchical policy described above through the modiﬁed switch policy and instruction policy by using the STG as a prior.
This amounts to treating the past history of switches and instructions in positive episodes as a guidance on whether the hierarchical policy should defer to the base policy to execute a speciﬁc base task or employ its own augmented ﬂat policy to take a primitive action.
In an episode, the temporal sequence of e t and g (cid:48) t , i.e., {(cid:104) e t , g (cid:48) t (cid:105) ; t ≥ 0 } , can be seen as a ﬁnite state Markov chain (Baum & Petrie, 1966).
Note that the state here is referred to the tuple (cid:104) e t , g (cid:48) t (cid:105) , which is not the state of the game s t ∈ S deﬁned in Section 3.1.
Consequently, at each level k > 0 , we may deﬁne an STG of a task g by i) transition probabilities, ρ k ( e t , g (cid:48) t | e t − 1 , g (cid:48)
t − 1 , g ) , and ii) the distribution of (cid:104) e 0 , g (cid:48)
(cid:105) , q k ( e 0 , g (cid:48)
| g ) , all of which follow categorical distributions.
With the estimated probabilities, we sample e t and g (cid:48) t in an episode at level k > 0 w [Equation] r
[Equation]
t. to reshaped policies π sw (cid:48) k and π inst k (cid:48) respectively:
If
[Equation]
• Otherwise,
[Equation]
g (cid:48) t | e t − 1 , g (cid:48) t − 1 , g ) .
Note that primitive action sampling is not affected by the STG.
3. P LAN C OMPOSITION
Combined with our hierarchical policy and STG deﬁned above, we are able to run an episode to compose a plan for a task speciﬁed by a human instruction.
Algorithm 1 in Appendix A summarized this procedure with respect to the policy and STG at level k .
Note that to fully utilize the base policy, we assume that once triggered, a base policy will play to the end before the global policy considers the next move.
L EARNING
The learning algorithm is outlined in Algorithm 2 in Appendix A.
We learn our ﬁnal hierarchical policy through k stages of skill acquisition.
Each of these stages is broken down into a base skill acquisition phase and a novel skill acquisition phase in a 2-phase curriculum learning.
In the base skill acquisition phase, we only sample tasks from the base task set G k − 1 .
This ensures that the global policy learns how to use previously learned skills by issuing instructions to the base policy.
In other words, this phase teaches the agent how to connect its instruction policy to its base policy.
Once the average reward for all base tasks exceeds a certain threshold, we proceed to the next phase.
In the novel skill acquisition phase, we sample tasks from the full task set, G k , for the k -th stage of skill acquisition.
It is in this phase that the agent can learn when to rely on the base policy and when to rely on the augmented ﬂat policy for executing novel tasks.
In each of these phases, all policies are trained with advantage actor-critic (A2C) (Section 4.1) and distributions in the STG are estimated based on accumulated positive episodes (Section 4.2).
4. P OLICY O PTIMIZATION BY A DVANTAGE A CTOR -C RITIC
We use advantage actor-critic (A2C) for policy optimization with off-policy learning (Su et al., 2017).
Here, we only consider the gradient for global policies (i.e., k > 0 ) as we assume the terminal policy has been trained as initial condition.
Let V k ( s t , g ) be a value function indicating the expected return given state s t and task g .
To reﬂect the nature of the branch switching in our model, we introduce another value function V sw k
( s t , e t , g ) to represent the expected return given state s t , task g and current branch selection e t .
Thus, given a trajectory Γ = {(cid:104) s t , e t , g (cid:48) t , a t , r t , µ sw k ( ·| s t ) , µ inst k ( ·| s t , g ) , µ aug k ( ·| s t , g ) , g (cid:105) : t = 0 , 1 , · · · , T } generated by old policies µ sw k ( ·| s t ) , µ inst k ( ·| s t , g ) , and µ aug k ( ·| s t , g ) , the policy gradi- ent reweighted by importance sampling can be formulated as
[Equation]
where ω sw t =
[Equation]
and [Equation] are importance sampling weights for the three terms respectively; A ( s t , g, e t ) , A ( s t , g, e t , g (cid:48) t ) , and A ( s t , g, e t , a t ) are estimates of advantage functions, which have multiple possible deﬁnitions.
In this paper, we deﬁne them by the difference between empirical return and value function estimation: A ( s t , g, e t ) = (cid:80) [Equation] = (cid:80) ∞ [Equation]
γ τ R
[Equation]
s
[Equation]
+
[Equation]
g )
[Equation]
V sw k
( s t , g, e t ) , where γ is the discounted coefﬁcient.
Finally, the value functions can be updated using the following gradient:
[Equation]
To increase the episode efﬁciency, after running an episode, we conduct n mini-batch updates where n is sampled from a Poisson distribution with λ = 4 , similar to Wang et al.
Note that one can also apply other common policy optimization methods, e.g., A3C (Mnih et al., 2016), to our model.
We leave this as future work to evaluate the efﬁciency of different methods when using our model.
Optimizing all three sub-policies together leads to unstable learning.
To avoid this, we apply a simple alternating update procedure.
For each set of M iterations, we keep two of the sub-policies ﬁxed and train only the single policy that remains.
When we reach M iterations, we switch the policy that is trained.
For all experiments in this paper, we use M = 500 .
This alternating update procedure is used within both phases of curriculum learning.
4. L EARNING AN STG
If at any point in the aforementioned training process the agent receives a positive reward after an episode, we update the stochastic temporal grammar.
ρ k and q k of the STG are both initialized to be uniform distributions.
Since the STG is a ﬁnite state Markov chain over tuples (cid:104) e t , g (cid:48) t (cid:105) , we use maximum likelihood estimation (MLE) to update the distributions (Baum & Petrie, 1966).
As the training progresses, the STG starts to guide the exploration.
To avoid falling into local minima in the early stages of training, it is important to encourage random exploration in early episodes.
Based on our experiments, we ﬁnd that using (cid:15) -greedy sufﬁces.
E XPERIMENTS 5. G AME E NVIRONMENT AND T ASK S PECIFICATIONS
Figure 3 (left) shows the two room environment in Minecraft that we created using the Malmo platform (Johnson et al., 2016).
In each episode, an arbitrary number of blocks with different colors (totaling 6 colors in our experiments) are randomly placed in one of the two rooms.
The agent is initially placed in the same room with the items.
We consider ﬁve sets of tasks: i) G (0) = { “Find x” } , walking to the front of a block with color x, ii) G (1) =
{ “Get x” } , picking up a block with color x, iii) G (2)
= { “Put x” } , putting down a block with color x, iv) G (3) =
{ “Stack x” } , stacking two blocks with color x together, and v) G (4) = { ’Put x on y’ } , putting a block with color x on top of a block with a different color y.
In total, there are 54 tasks.
An agent can perform the following actions: “move forward,” “move backward,” “move left,” “move right,” “turn left,” “turn right,” “pick up,” “put down.”
[Figure]
Without loss of generality, we assume the following skill acquisition order: G k = ∪ kκ
=1 G ( κ ) , ∀ k = 0 , 1 , 2 , 3 , 4 , which is a natural way to increase skill sets.
One may also alter the order, and the main conclusions shall still hold.
This results in policies { π k : k = 0 , 1 , 2 , 3 , 4 } for these four task sets.
For the last task set, we hold out 6 tasks out of all 30 tasks (i.e., 3 pairs of colors out of 15 color combinations) for testing and the agent will not be trained on these 6 tasks.
We adopt a sparse reward function: when reaching the goal of a task, the agent gets a +1 reward; when generating an instruction g (cid:48) that is not executable in current game (e.g., trying to ﬁnd an object that does not exist in the environment), we give a − 0 .
5 reward; otherwise, no reward will be given.
Whenever a non-zero reward is given, the game terminates.
Note that the negative reward is only given during training.
5. I MPLEMENTATION D ETAILS
We specify the architecture of the modules in our model in Appendix B, where the visual and instruction encoding modules have the same architectures as the ones in Hermann et al.
We train the network with RMSProp (Tieleman & Hinton, 2012) with a learning rate of 0.0001.
We set the batch size to be 36 and clip the gradient to a unit norm.
For all tasks, the discounted coefﬁcient is γ = 0 .
For the 2-phase curriculum learning, we set the average reward threshold to be 0.
(average rewards are estimated from the most recent 200 episodes of each task).
To encourage random exploration, we apply (cid:15) -greedy to the decision sampling for the global policy (i.e., only at the top level k at each stage k > 0 ), where (cid:15) gradually decreases from 0 .
1 to 0 .
[Figure]
[Figure]
[Table]
[Table]
L EARNING E FFICIENCY
To evaluate the learning efﬁciency, we compare our full model with 1) a ﬂat policy (Figure 2a) as in Hermann et al.
(2017) ﬁne-tuned on the terminal policy π 0 , 2) H-DRLN (Tessler et al., 2017) and variants of our approach: 3) ours without STG, 4) ours without alternating policy optimization, and 5) ours without V sw k
( s, e, g ) (replaced by V k ( s, g ) instead).
Note that all the rewards have been converted to the same range, i.e., [0 , 1] for the sake of fair comparison.
In Figure 4a, we use various methods to train policy π 1 for the task set G 1 based on the same base policy π 0 .
The large dip in the reward indicates that the curriculum learning switches from phase 1 to phase 2.
From Figure 4a, we may clearly see that our full model and variants can all converge within 22,000 episodes, whereas the average reward of the ﬂat policy is still below 0.8 given the same amount of episodes.
In addition, our full model ﬁnishes phase 1 signiﬁcantly faster than other methods and its curve of average reward maintains notably higher than the remaining ones.
To further examine the learning efﬁciency during phase 2 when new tasks are added into the training process, we ﬁrst pretrain π 3 using our full model following our deﬁnition of phase 1 in the curriculum learning.
We then proceed to learning phase 2 using different approaches all based on this pretrained policy.
As shown in Figure 4b, our full model has the fastest convergence and the highest average reward upon convergence.
By comparing Figure 4a and Figure 4b, we further show that our full model has a bigger advantage when learning more complex tasks.
Since we have a large number of previously learned tasks, H-DRLN is clearly not able to learn a descent policy according the results.
Note that an H-DRLN can only learn one task at a time, each of its curves in Figure 4 is for a single task (i.e., “Get white” and “Stack white” respectively).
To demonstrate the effects of our 2-phase curriculum learning and the − 0 . 5 penalty on the training efﬁciency, we visualize the learning curves of our model trained without the curriculum learning or without the penalty along with the one trained with the full protocol in Figure 5.
According to the results, the curriculum learning indeed helps accelerate the convergence, which empirically proves the importance of encouraging a global policy to reuse relevant skills learned by its base policy.
It also appears that adding the penalty is an insigniﬁcant factor on learning efﬁciency except that it helps shorten the episode lengths as an episode ends whenever a penalty is given.
5. P OLICY G ENERALIZATION
Finally, we evaluate how the hierarchical design and encoding tasks by human instructions beneﬁt the generalization of learned policies in the following three ways.
First, we train π 1 in a simpler setting where in each episode, only one item (i.e, the target item of the given task) is present.
We then test the policy π 1 for “Get x” tasks in a room where there will be multiple items serving as distraction and the agent must interact with the correct one.
Both the ﬂat policy and the hierarchical policy can achieve near perfect testing success rate in the simple setting.
However, in the more complex setting, ﬂat policy can not differentiate the target item from other items that are also placed in the room (the success rate drops to 29%), whereas our hierarchical policy still maintains a high success rate (94%).
This ﬁnding suggests that the hierarchical policy not only picks up the concept of “ﬁnd” and “get” skills as the ﬂat policy does, but also inherits the concept of items from the base policy by learning to utter correct instructions to deploy “ﬁnd” skill in the base policy.
Second, we reconﬁgure the room layout in Figure 3 (left) and test the ﬂat policy and our full model in the new rooms shown in Figure 3 (right) for various tasks.
Both policies are trained in the same environment.
There are multiple items in a room for both training and testing cases.
The success rates are summarized in Table 1.
Using the ﬂat policy results in a much bigger drop in the testing success rate compared to using out full model.
This is mainly because that our global policy will repeatedly call its base policy to execute the same task until the agent ﬁnally achieves the goal even though the trained agent is unable to reach the goal by just one shot due to the simplicity of the training environment.
Third, we evaluate the learned policy on the 6 unseen tasks for the “Put x on y” tasks as a zeroshort evaluation.
The success rate reported in Table 1 suggests that our model is able to learn the decomposition of human instructions and generate correct hierarchical plans to perform unseen tasks.
5. P OLICY I NTERPRETABILITY
We visualize typical hierarchical plans of several tasks generated by global policies learned by our full model in Appendix C (Figure 6).
It can been seen from the examples that our global policies adjust the composed plans in different scenarios.
For instance, in the second plan on the ﬁrst row, πdid not deploy base policy π 0 as the agent was already in front of the target item at the beginning of the episode, whereas in the plan on the second row, π 1 deployed π 0 for the “Find x” base task twice consecutively, as it did not ﬁnish the base task in the ﬁrst call.
C ONCLUSION
In this work, we have proposed a hierarchal policy modulated by a stochastic temporal grammar as a novel framework for efﬁcient multi-task reinforcement learning through multiple training stages.
Each task in our settings is described by a human instruction.
The resulting global policy is able to reuse previously learned skills for new tasks by generating corresponding human instructions to inform base policies to execute relevant base tasks.
We evaluate this framework in Minecraft games and have shown that our full model i) has a signiﬁcantly higher learning efﬁciency than a ﬂat policy does, ii) generalizes well in unseen environments, and iii) is capable of composing hierarchical plans in an interpretable manner.
Currently, we rely on weak supervision from humans to deﬁne what skills to be learned in each training stage.
In the future, we plan to automatically discover the optimal training procedures to increase the task set.
A P SEUDO C ODE OF O UR A LGORITHMS
[Table]
Sample
[Table] [Table]
Get
[Table] [Table]
←
[Table] [Table]
[Table]
Add (cid:104)
[Table] [Table]
(cid:105) to Γ
[Table]
else
[Table]
if
[Table]
t
[Table]
then
[Table] [Table]
[Table]
else
[Table]
if
[Table]
Get
[Table] [Table]
[Table]
22: end if 23:
[Table] [Table]
[Table]
B A RCHITECTURES OF M ODULES
The architecture designs of all modules in our model shown in Figure 2 are as follows:
Visual Encoder extracts feature maps from an input RGB frame with the size of 84 × 84 through three convolutional layers: i) the ﬁrst layer has 32 ﬁlters with kernel size of 8 × 8 and stride of 4 ; ii) the second layer has 64 ﬁlters with kernel size of 4 × 4 and stride of 2; iii) the last layer includesﬁlters with kernel size of 3 × 3 and stride of 1.
The feature maps are ﬂatten into a 3136-dim vector.
We reduce the dimension of this vector to 256 by a fully connected (FC) layer resulting a 256-dim visual feature as the ﬁnal output of this module.
Instruction Encoder ﬁrst embeds each word into a 128-dim vector and combines them into a single vector by bag-of-words (BOW).
Thus the output of this module is a 128-dim vector.
For more complex instructions such as “Put x on y”, we replace BOW by a GRU with 128 hidden units.
Fusion layer simply concatenates the encoded visual and language representations together and outputs 384-dim fused representation.
We then feed this 384-dim vector into an LSTM withhidden units.
The hidden layer output of the LSTM is served as the input of all policy modules and value function modules.
Switch Policy module has a FC layer with output dimension of 2 and a softmax activation to get π sw k
( e | s, g ) .
Instruction Policy module has two separate FC layers, both of which are activated by softmax to output the distribution of skill , p skill k
( u skill | s, g ) , and the distribution of item , p item k
( u item | s, g ) , respectively.
Augmented Policy module outputs π aug ( a | s, g ) also through a FC layer and softmax activation.
The two Value Function modules, V ( s, g ) and V sw ( s, e, g ) , all have a scalar output through a FC layer.
Finally, the Selector module selects the action sampled from Augmented Policy module or Base
Policy module based on the switching decision sampled from the Switch Policy module.
C C OMPOSED H IERARCHICAL P LANS
Figure 6 shows several plans for different tasks composed by executing our hierarchical policies.