Learning Parameterized Task Structure for Generalization to Unseen Entities
Abstract
Real world tasks are hierarchical and compositional.
Tasks can be composed of multiple subtasks (or sub-goals) that are dependent on each other.
These subtasks are deﬁned in terms of entities (e.g., apple, pear ) that can be recombined to form new subtasks (e.g., pickup apple , and pickup pear ).
To solve these tasks efﬁciently, an agent must infer subtask dependencies (e.g. an agent must execute pickup apple before place apple in pot ), and generalize the inferred dependencies to new subtasks (e.g. place apple in pot is similar to place apple in pan ).
Moreover, an agent may also need to solve unseen tasks, which can involve unseen entities.
To this end, we formulate factored subtask graph inference (FSGI), a method for modeling subtask dependencies using ﬁrst-order logic with factored entities.
To facilitate this, we learn factor attributes in a zero-shot manner, which are used as quantiﬁers (e.g. is_pickable(X) ) for the factored subtask graph.
We show this approach accurately learns the latent structure on hierarchical and compositional tasks more efﬁciently than prior work, and show FSGI can generalize by modelling structure on subtasks unseen during adaptation.
Introduction
Real world tasks are compositional and hierarchical [Carvalho et al., 2020, Loula et al., 2018,
Andreas et al., 2017, Oh et al., 2017].
These tasks are composed of multiple subtasks (or sub-goals) that must be completed in certain order.
For example, the cooking task shown in Figure 1 requires an agent to boil some food object (e.g. Cooked egg ).
An agent must place the food object x in a cookware object y , place the cookware object on the stove, before boiling this food object x .
Parts of this task can be decomposed into sub-goals, or subtasks (e.g. Pickup egg , Put egg on pot ).
Solving these tasks requires long horizon planning and reasoning ability [Erol, 1996, Xu et al., 2018,
Ghazanfari and Taylor, 2017, Sohn et al., 2018].
This problem is made more difﬁcult of rewards are sparse , if only few of the subtasks in the environment provide reward to the agent.
Real world tasks are also parameterized .
These tasks may contain multiple objects and interactions between these objects, which we refer to as entities .
These entities can be recombined to form new subtasks, e.g. (place, apple, pot) .
These components can be numerous, leading to a combinatorial number of subtasks.
For example, the cooking task shown in Figure 1 contains subtasks that follow a action-objects structure.
The action Pickup admits many subtasks, where any object x composes into a new subtask (e.g. Pickup egg , Pickup pot ).
Solving parameterized tasks entities also requires reasoning [Andreas et al., 2017, Oh et al., 2017].
Without reasoning on the relations between components between tasks, exploring the space of a combinatorial number of subtasks is extremely inefﬁcient.
In this work we propose to tackle the problem of hierarchical and parameterized tasks.
Prior work has tackled learning hierarchical task structures by modelling dependencies between subtasks in a graph structure [Sohn et al., 2018, 2020, Xu et al., 2018, Huang et al., 2019].
In these settings, during training, the agent tries to efﬁciently adapt to a task by inferring the latent graph structure, then uses the inferred graph to maximize reward during test.
However, this approach does not scale for parameterized tasks.
Prior work tries to infer the structure of subtasks individiually – they do not consider the relations between entities in parameterized tasks.
E.g. they do not infer that Place egg on pot and Pickup egg on pan have similar hierarchical structure.
We propose the parameterized subtask graph inference (PSGI) approach for tackling hierarchical and compositional tasks.
We present an overview of our approach in Figure 1.
This approach extends the problem introduced by Sohn et al.
Similar to Sohn et al.
[2020], we assume options [Sutton et al., 1999] (low level policies) for completing subtasks have been trained or are given as subroutines for the agent.
These options are imperfect, and require certain conditions on the state to be meet before they can be successfully executed.
We model the problem as a transfer RL problem.
During training, an exploration policy gathers trajectories.
These trajectories are then used to infer the latent parameterized subtask graph , (cid:98) G .
(cid:98) G models the hierarchies between compositional tasks and options in symbolic graph structure (shown in 1).
In PSGI, we infer the preconditions of options, subtasks that must be completed before an option can be successfully executed, and the effects of options, subtasks that are completed after they are executed.
The parameterized subtask graph is then used to maximize reward in the test environment by using GRProp, a method introduced by Sohn et al.
[2018] which propagates a gradient through (cid:98) G to learn the test policy.
In PSGI, we model using parameterized options and subtasks.
This allows PSGI to infer the latent task structure in a ﬁrst-order logic manner.
For example, in the cooking task in Figure 1 we represent all Pickup -object options using a parameratized option ,
Pickup x .
Representing options and subtasks in parameterized form serves two roles: 1.
The resulting graph is more compact .
There is less redundancy when representing compositional tasks that share common structure.
Hence a parameterized subtask graph requires less samples to infer (e.g. relations for Pickup apple ,
Pickup pan , etc. are inferred at once with a parameterized option Pickup x ).
The resulting graph can generalize to unseen subtasks, where unseen subtasks may share similar structure but are not encountered during adaptation (e.g. Pickup cabbage in Figure 1).
To enable parameterized representation, we also learn the attributes of the components in the compositional tasks.
These attributes are used to indicate differences in the structures of parameterized options and subtasks.
For example, in the cooking task in Figure 1, not every object can be picked up with Pickup , so the inferred attribute (cid:98) f pickupable ( x ) is a precondition to Pickup ( x ) .
Similarly, in a more complex cooking task, some object x may need to be sliced, before it can be boiled (e.g. cabbage), but some do not (e.g. egg).
We model these structures using parameter attributes , (cid:98) A att (in the cooking task case objects are parameters).
We present a simple scheme to infer attributes in a zero-shot manner, where we infer attributes that are useful to infer relations between parameters without supervision.
These attributes are then used to generalize to other parameters (or entities), that may be unseen during adaptation.
We summarize our contributions as follows:
• We propose the approach of parameterized subtask graph inference (PSGI) to efﬁciently infer the subtask structure of hierarchical and parameterized tasks in a ﬁrst order logic manner .
• We propose a simple zero-shot learning scheme to infer entity attributes , which are used to relate the structures of compositional subtasks.
• We demonstrate PSGI on a symbolic cooking environment that has complex hierarchical and parameterized task structure.
We show PSGI can accurately infer this structure more efﬁciently than prior work and generalize this structure to unseen tasks.
Problem Deﬁnition 2.1 Background: Transfer Reinforcement Learning
A task is characterized by an MDP M G =
(cid:104)A , S , T G , R G (cid:105) , which is parameterized by a task-speciﬁc
G , with an action space A , state space S , transition dynamics T G , and reward function R G .
In the transfer RL formulation Duan et al. [2016], Finn et al. [2017], an agent is given a ﬁxed distribution of training tasks M train , and must learn to efﬁciently solve a distribution of unseen test tasks M test .
Although these distributions are disjoint, we assume there is some similarity between tasks such that some learned behavior in training tasks may be useful for learning test tasks.
In each task, the agent is given k timesteps to interact with the environment (the adaptation phase), in order to adapt to the given task.
After, the agent is evaluated on its adaptation (the test phase).
The agent’s performance is measured in terms of the expected return:
[Equation]
where π K is the policy after k timesteps of the adaptation phase, H is the horizon in the test phase, and r t is the reward at time t of the test phase.
2.2 Background: The Subtask Graph Problem
The subtask graph inference problem is a transfer RL problem where tasks are parameterized by hierarchies of subtasks Sohn et al. [2020], Φ .
A task is composed of N subtasks, { φ 1 , . .
φ N } ⊂ Φ , where each subtask φ ∈ Φ is parameterized by the tuple (cid:104)S comp , G r (cid:105) , a completion set S comp ⊂ S , and a subtask reward G r : S → R .
The completion set S comp denotes whether the subtask φ is complete , and the subtask reward G r is the reward given to the agent when it completes the subtask.
Following Sohn et al.
[2020], we assume the agent learns a set of options O =
{O 1 , O 2 , . . .
} that completes the corresponding subtasks Sutton et al.
These options can be learned by conditioning on subtask goal reaching reward: r t =
I ( s t ∈ S
i comp ) .
Each option
O ∈
O is parameterized by the tuple (cid:104) π, G prec , G effect (cid:105) .
There is a trained policy π corresponding to each O .
These options may be eligible at different precondition states G prec ⊂ S , where the agent must be in certain states when executing the option, or the policy π fails to execute (also the initial set of
O following Sutton et al.
However, unlike Sohn et al.
[2020], these options may complete an unknown number of subtasks (and even remove subtask completion).
This is parameterized by
G effect ⊂ S (also the termination set of O following Sutton et al.
Environment: We assume that the subtask completion and option eligibility is known to the agent.
(But the precondition, effect, and reward is hidden and must be inferred).
In each timestep t the agent is the state s t = { x t , e t , step t , step phase ,t , obs t } .
• Completion: x t ∈ { 0 , 1 } N denotes which subtasks are complete.
• Eligibility: e t ∈ { 0 , 1 } M denotes which options are eligible.
• Time Budget: step
t ∈
Z > 0 is the number steps remaining in the episode.
• Adaptation Budget: step phase ,t ∈ Z > 0 is the number steps remaining in the adaptation phase.
• Observation: obs t ∈ R d is a low level observation of the environment at time t.
2.3 The Parameterized Subtask Graph Problem
[-Precondition of 𝑫 𝑿 𝑫 𝑿 = 𝒇(𝑿)𝑨 𝑿 +𝒇 𝑿 𝑨 𝑿 𝒈(𝑿) 𝑫 𝑿 =𝑫 𝑿 = 𝟏 𝑫 𝑿 = 1 ⋮           1 ⋮   
-]
Subtasks and Option Entities In the real world, compositional subtasks can be described in terms of a set of entities , E .
(e.g. pickup , apple , pear , · · · ∈ E ) that can be recombined to form new subtasks (e.g. ( pickup , apple ), and ( pickup , pear )).
We assume that these entities are given to the agent.
Similarly, the learned options that execute these subtasks can also be parameterized by the same entities (e.g. [ pickup , apple ], and [ pickup , pear ]).
In real world tasks, we expect learned options with entities that share “attributes” to have similar policy, precondition, and effect, as they are used to execute subtasks with similar entities .
For example, options [ cook , egg , pot ] and [ cook , cabbage , pot ] share similar preconditions (the target ingredient must be placed in the pot ), but also different ( cabbage must be sliced, but the egg does not).
In this example, egg and cabbage are both boilable entities, but egg is not sliceable .
To model these similarities, we assume in each task, there exist boolean latent attribute functions which indicate shared attributes in entities.
E.g. f pickupable : E → { 0 , 1 } , where f pickupable ( apple ) = 1 .
We will later try to infer the values of these latent entities, so we additionally assume there exist some weak supervision, where a low-level embedding of entities is provided to the agent, f entityembed : E → R D .
The Parameterized Subtask Graph Our goal is to infer the underlying task structure between subtasks and options so that the agent may complete subtasks in an optimal order.
As deﬁned in the previous sections, this task structure can be completely determined by the option preconditions, option effects, and subtask rewards.
As such we deﬁne the parameterized subtask graph to be the tuple of the parameterized preconditions, effects, and rewards for all subtasks and options:
[Equation]
where [Equation]
The parameterized precondition, G prec , is a function from an option with N entities and a subtask completion set to { 0 , 1 } , which speciﬁes whether the option is eligible under a completion set.
E.g. If G prec ([ X 1 , X 2 ] , s ) = 1 , then the option [ X 1 , X 2 ] is eligible if the agent is in state s .
The parameterized effect, G eff , is a function from an option with N entities and subtask completion set to a different completion set.
Finally, the parameterized reward, G r , is a function from a subtask with N entities to the reward given to the agent from executing that subtask.
Our previous assumption that options with similar entities and attributes share preconditions and effects manifests in G prec and G eff where these functions tend to be smooth .
Similar inputs to the function (similar option entities) tend to yield similar output (similar eligibility and effect values).
This smoothness gives two beneﬁts.
We can share experience between similar options for inferring preconditions and effect.
This enables generalization to preconditions and effects of unseen entities.
Note that this smoothness does not apply to the reward G r .
We assume reward given for subtask completion is independent across tasks.
𝑡 𝒙 𝒆 𝐴 ! 𝐴 " 𝐵 ! 𝐵 " 𝐶 ! 𝐶 " 𝐷 !
𝐷 " 𝐸 !
⋮ 𝐻
Option 𝑫 𝑿
Agent Trajectory 𝒙 𝑎𝑢𝑔 𝒆 𝐴 # 𝐵 # 𝑓 𝑥 𝑔(𝑥)
𝐷 # ⋮
Augmented Predicate Trajectory
F T 𝒇 𝑿 >𝑫 𝑿 =𝑨 𝑿 > 𝟎 F 𝒈 𝑿 > 𝟎 F
T
Decision tree of 𝑫 𝑿
T
CART train
Logic expression
{+Precondition of 𝑫 𝑿 𝑫 𝑿 = 𝒇(𝑿)𝑨 𝑿 +𝒇 𝑿 𝑨 𝑿 𝒈(𝑿) 𝑫 𝑿 =𝑫 𝑿 = 𝟏 𝑫 𝑿 = 1 ⋮           1 ⋮   
+}
⋮
We translate the decision tree into an equivalent predicate boolean expression, which is one part of the inferred parameratized subtask graph (cid:98) G .
Method
We propose the Parameterized Subtask Graph Inference (PSGI) method to efﬁciently infer the latent parameterized subtask graph G =
(cid:104)G prec , G eff , G r (cid:105) .
Figure 2 gives an overview of our approach.
At a high level, we use the adaptation phase to gather adaptation trajectories from the environment using an adaptation policy π adapt θ .
Then, we use the adaptation trajectories to infer the latent subtask graph (cid:98) G .
In the test phase, a test policy π test (cid:98) G is conditioned on the inferred subtask graph (cid:98) G and maximizes the reward.
As the performance of the test policy is dependent on the inferred subtask graph (cid:98) G , it is important to accurately infer this graph. Note that the test task may contain subtasks that are unseen in the training task.
We learn a predicate subtask graph (cid:98) G that can generalize to these unseen subtasks and options.
3.1 Zero-shot Learning Entity Attributes
In the Parameterized Subtask Graph Problem deﬁnition, we assume there exist latent attributes that indicate shared structure between options and subtasks with the same attributes.
E.g. One attribute may be f pickupable : E → { 0 , 1 } , where f pickupable ( apple ) = 1 , etc.
Our goal is to infer a set of candidate attribute functions,
(cid:98) A att
= { (cid:98) f 1 , (cid:98) f 2 , . . .
} , such that options with the same attributes indicates the same preconditions.
As there is no supervision involved, we formulate this inference as a zero shot learning problem Palatucci et al.
Note the inferred attributes that are preconditions for options should not only construct an accurate predicate subtask graph for options seen in the adaptation phase, but also unseen options.
During the adaptation phase, the agent will encounter a set of seen entities E ⊂ E .
We construct candidate attributes from E using our smoothness assumption, where similar entities result in similar preconditions.
We generate candidate attributes based on similarity using the given entity embedding, f entityembed : E → R D .
Let C
= { C 1 , C 2 , . . .
} be an exhaustive set of clusters generated from E using f entityembed .
Then, we deﬁne a candidate attribute function from each cluster.
[Equation]
[Equation]
To infer the attribute of an unseen entity X (cid:54)∈ E , we use a 1-Nearest Neighbor classiﬁer that uses the attributes of the nearest seen entity Fix [1985].
[Equation]
where X ∗ = argmin X (cid:48) ∈
E dist ( f entityembed ( X ) , f entityembed ( X (cid:48) )) .
3.2 Parameterized Subtask Graph Inference
Let τ H
= { s 1 , o 1 , r 1 , d 1 , . . .
, s H } be the adaptation trajectory of the adaptation policy π adapt θ after H time steps.
Our goal is to infer the maximum likelihood parameterized subtask graph G given this trajectory τ H .
[Equation]
By expanding this likelihood term, we show that to maximize (cid:98) G , it sufﬁces to maximize (cid:98) G prec , (cid:98) G eff , and (cid:98) G r individually. [Equation]
[Equation]
We show details of this derivation in the appendix.
Next, we explain how to compute (cid:98) G prec , (cid:98) G eff , and (cid:98) G r .
Parameterized Precondition Inference via Predicate Logic Induction We give an overview of how we infer the option preconditions (cid:98) G prec in Figure 2.
Note from the deﬁnition, we can view the precondition G prec as a deterministic function, f G prec : ( E, x ) (cid:55)→ { 0 , 1 } , where E is the option entities, and x is the completion set vector.
Hence, the probability term in Eq.(27) can be written as p ( e t | x t , G prec ) =
(cid:81)
Ni =
I [ e ( i ) t = f G prec ( E ( i ) , x t )] where I is the indicator function, and E ( i ) is the entity set of the i th option in the given task.
Thus, we have
[Equation]
Following Sohn et al.
[2020], this can be maximized by ﬁnding a boolean function (cid:98) f G prec over only subtask completions x t that satisﬁes all the indicator functions in Eq.(13).
However this yields multiple possible solutions — particularly the preconditions of unseen option entities in the trajectory τ H .
If we infer a (cid:98) f G prec separately over all seen options (without considering the option parameters), this solution is identical to the solution proposed by Sohn et al.
We want to additionally generalize our solution over multiple unseen subtasks and options using the entities, E .
We leverage our smoothness assumption — that (cid:98)
f G prec is smooth with respect to the input entities and attributes.
E.g. If the inferred precondition for the option [ pickup , X ] is the candidate attribute (cid:98) f ( X ) , any entity X where (cid:98) f ( X ) = 1 has the same precondition.
I.e. For some unseen entity set E ∗ we want the following property to hold:
[Equation]
To do this, we infer a boolean function (cid:100)
f G prec over both subtask completions x t and entity variables
X ∈ E .
We use (previously inferred) candidate attributes over entities, (cid:98)
f i ( X ) ∀ X ∈ E in the boolean function to serve as quantiﬁers .
Inferring in this manner insures that the precondition function (cid:98) f G prec is smooth with respect to the input entities and attributes.
Note that some but not all attributes may be shared in entities.
E.g. [ cook , cabbage ] has similar but not the same preconditions as [ cook , egg ].
So, we cannot directly reuse the same preconditions for similar entities.
We want to generalize between different combinations of attributes.
We translate this problem as an inductive logic programming (ILP) problem Muggleton and De Raedt [1994].
We infer the eligibility (boolean output) of some option O with some entities(s) E
= { X 1 , X 2 , . . .
, from boolean input formed by all possible completion values { x it } Ht =1 , and all attribute values { (cid:98) f i ( X ) } i =1 ... X ∈ E .
We use the classiﬁcation and regression tree (CART) with Gini impurity to infer the the precondition functions (cid:98)
f G prec for each parameter E Breiman et al.
Finally, the inferred decision tree is converted into an equivalent symbolic logic expression and used to build the parameterized subtask graph.
Parameterized Effect Inference We include an visualization of how we infer the option effects (cid:98) G eff in the appendix in the interest of space.
From the deﬁnitions of the parameterized subtask graph problem, we can write the predicate option effect G eff as a deterministic function f
G eff : ( E, x t ) (cid:55)→ x t +1 , where if there is subtask completion x t , executing option O (with entities E ) successfully results in subtask completion x t +1 .
Similar to precondition inference, we have
[Equation]
As this is deterministic, we can calculate the element-wise difference between x t (before option) and x t +1 (after option) to infer f G eff .
[Equation]
Similar to precondition inference, we also want to infer the effect of options with unseen parameters.
We leverage the same smoothness assumption:
[Equation]
Unlike preconditions, we expect the effect function to be relatively constant across attributes, i.e., the effect of executing option [ cook , X ] is always completing the subtask ( cooked , X ), no matter the attributes of X .
So we directly set the effect of unseen entities, (cid:100)
f G eff ( E ∗ , x t ) , by similarity according to Equation 17.
Reward Inference We model the subtask reward as a Gaussian distribution G r ( E ) ∼ N ( (cid:98) µ E , (cid:98) σ E ) .
The MLE estimate of the subtask reward becomes the empirical mean of the rewards received during the adaptation phase when subtask with parameter T becomes complete.
For the i th subtask in the task with entities E i ,
[Equation]
Note we do not use the smoothness assumption for (cid:98) G r ( E ) , as we assume reward is independently distributed across tasks.
We initialize (cid:98) G r ( E ∗ )
=
for unseen subtasks with entities E ∗ and update these estimates with further observation.
3.3 Task Transfer and Adaptation
In the test phase, we instantiate a test policy π test (cid:98) G prior using the parameterized subtask graph (cid:98) G prior , inferred from the training task samples.
The goal of the test policy is to maximize reward in the test environment using (cid:98) G prior .
As we assume the reward is independent across tasks, we re-estimate the reward of the test task according to Equation 18, without task transfer.
With the reward inferred, this yields the same problem setting given in Sohn et al.
Sohn et al.
[2018] tackle this problem using GRProp, which models the subtask graph as differentiable function over reward, so that the test policy has a dense signal on which options to execute are likely to maximally increase the reward.
However, the inferred parameterized subtask graph may be imperfect, the inferred precondition and effects may not transfer to the test task.
To adapt to possibly new preconditions and effects, we use samples gathered in the adaptation phase of the test task to infer a new parameterized subtask graph (cid:98) G test , which we use to similarly instantiate another test policy π test (cid:98) G test using GRProp.
We expect (cid:98) G test to eventually be more accurate than (cid:98) G prior as more timesteps are gathered in the test environment.
To maximize performance on test, we thus choose to instantiate a posterior test policy π testposterior , which is an ensemble policy over π test (cid:98) G prior and π test (cid:98) G test .
We heuristically set the weights of π testposterior to favor π test (cid:98) G prior early in the test phase, and π test (cid:98) G test later in the test phase.
Related Work
Subtask Graph Inference.
The subtask graph inference (SGI) framework Sohn et al. [2018, 2020] assumes that a task consists of multiple base subtasks, such that the entire task can be solved by completing a set of subtasks in the right order.
Then, it has been shown that SGI can efﬁciently solve the complex task by explicitly inferring the precondition relationship between subtasks in the form of a graph using an inductive logic programming (ILP) method.
The inferred subtask graph is in turn fed to an execution policy that can predict the optimal sequence of subtasks to be completed to solve the given task.
However, the proposed SGI framework is limited to a single task; the knowledge learned in one task cannot be transferred to another.
This limits the SGI framework such that does not scale well to compositional tasks, and cannot generalize to unseen tasks.
We extend the SGI framework by modeling parameterized subtasks and options, which encode relations between tasks to allow efﬁcient and general learning.
In addition, we generalize the SGI framework by learning an effect model – In the SGI framework it was assumed that for each subtask there is a corresponding option, that completes that subtask (and does not effect any other subtask).
Compositional Task Generalization.
Prior work has also tackled compositional generalization in a symbolic manner Loula et al.
[2018], Andreas et al.
[2017], Oh et al.
Loula et al.
[2018] test compositional generalization of natural language sentences in recurrent neural networks.
Andreas et al.
[2017], Oh et al.
[2017] tackle compositional task generalization in an instruction following context, where an agent is given a natural language instruction describing the task the agent must complete (e.g. “pickup apple").
These works use analogy making to learn policies that can execute instructions by analogy (e.g. “pickup X ").
However, these works construct policies on the option level – they construct policies that can execute “pickup X " on different X values.
They also do not consider hierarchical structure for the order which options should be executed (as the option order is given in instruction).
Our work aims to learn these analogy-like relations at a between-options level, where certain subtasks must be completed before another option can be executed.
Classical Planning.
At a high level, a parameterized subtask graph G is equivalent to a STRIPS planning domain Fikes and Nilsson [1971] with an attribute model add-on Frank and Jónsson [2003].
Prior work in classical planning has proposed to learn STRIPS domain speciﬁcations (action schemas) through given trajectories (action traces) Suárez-Hernández et al. [2020], Mehta et al.
[2011], Walsh and Littman [2008], Zhuo et al.
Our work differs from these in 3 major ways: 1.
PSGI learns an attribute model, which is crucial to generalizing compositional tasks with components of different behaviors.
We evaluate PSGI on more hierarchical domains, where prior work has evaluated on pickup-place/travelling classical planning problems, which admit ﬂat structure.
We evaluate PSGI on generalization, where there may exist subtasks and options that are not seen during adaptation.
Experiments
We aim to answer the following questions:
Can PSGI generalize to unseen evaluation tasks in zero-shot manner by transferring the inferred task structure?
Does PSGI efﬁciently infers the latent task structure compared to prior work (MSGI Sohn et al. [Figure] [Figure]
5.1 Environments
We evaluate PSGI in a novel symbolic environment, Cooking and Mining .
An example of the simpliﬁed Cooking task is shown in Figure 1.
For each task, we manually deﬁne the parameterized subtask graph in terms parameterized subtasks, options, and attributes in ﬁrst-order logic form.
Then, the subtask graph can be built by replacing each predicate with a set of task-speciﬁc entities.
Each task is randomized by randomly sampling the entities from the entity pool.
The entity pool for training and evaluation tasks are different such that the agent should be able to generalize to unseen entities to solve the evaluation tasks in zero-shot setting.
Tasks.
Cooking has a pool of 22 entities and 10 entities are chosen at random for each task.
Then, the subtasks and options are populated by replacing the parameters in parameterized subtasks and options by the sampled entities; e.g ., we replace X and Y in the parameterized subtask (pickup, X,
Y) by { apple, cabbage, table } to populate nine subtasks.
This results in around 320 options and 100 subtasks.
The ground-truth attributes are also predeﬁned in the templates, but is not available to the agent.
Similarly for Mining , we randomly sample 12 entities from a pool of 18 entities and populate around 180 subtasks and 180 options for each task.
The reward is assigned at random to one of the subtasks that have the largest critical path length, where the critical path length is the minimum number of options to be executed to complete each subtask.
See the appendix for more details on the tasks.
Observations.
At each time step, the agent observes the completion and eligibility vectors (see Parameterized Subtask Graph Problem section for deﬁnitions) and the corresponding embeddings.
The subtask and option embeddings are the concatenated vector of the embeddings of its entities; e.g ., for pickup, apple, table the embedding is [ f ( pickup ) , f ( apple ) , f ( table )] where f ( · ) can be an image or language embeddings.
In experiment, we used 50 dimensional GloVE word embeddings [Pennington et al., 2014] as the embedding function f ( · ) .
5.2 Baselines
• MSGI
+ is the MSGI Sohn et al. [2020] agent modiﬁed to be capable of solving our Cooking and Mining tasks.
We augmented MSGI with an effect model, separate subtasks and options in the ILP algorithm, and replace the GRProp with cyclic GRProp, a modiﬁed version of GRProp that can run with cycles in the subtask graph.
• HRL Andreas et al. [2017] 2 is the option-based hierarchical reinforcement learning agent.
It is an actor-critic model over the pre-learned options.
• Random agent randomly executes any eligible option.
We meta-train PSGI on training tasks and meta-eval on evaluation tasks to test its adaptation efﬁciency and generalization ability.
We train HRL on evaluation tasks to test its adaptation ( i.e ., learning) efﬁciency.
We evaluate Random baseline on evaluation tasks to get a reference performance.
We use the same recurrent neural network with self-attention-mechanism so that the agent can handle varying number of (unseen) parameterized subtasks and options depending on the tasks.
See the appendix for more details on the baselines.
5.3 Zero-/Few-shot Transfer Learning Performance
Zero-shot learning performance.
Figure 4 compares the zero-shot and few-shot transfer learning performance on Mining and Cooking domains.
First, PSGI achieves over 50 and 30% success rate on Cooking and Mining domain without observing any samples ( i.e ., x-axis value = 0) in unseen evaluation tasks.
This indicates that the parameterized subtask graph effectively captures the shared task structure, and the inferred attributes generalizes well to unseen entities in zero-shot manner.
Note that MSGI + , HRL , and Random baselines have no ability to transfer its policy from training tasks to unseen evaluation tasks.
We note that MSGI + has no mechanism for generalizing to the tasks with unseen entities.
Few-shot learning performance.
In Figure 4, PSGI achieves over 90 and 80% success rate on
Cooking and Mining domains respectively after only 1000 steps of adaptation, while other baselines do not learn any meaningful policy except MSGI + in Cooking .
This demonstrates that the parameterized subtask graph enables PSGI to share the experience of similar subtasks and options ( e.g ., pickup X on Y for all possible pairs of X and Y ) such that the sample efﬁciency is increased by roughly the factor of number of entities compared to using subtask graph in MSGI + .
5.4 Comparison on Task Structure Inference
We ran PSGI and MSGI + in the Cooking and Mining , inferring the latent subtask graphs fortimesteps.
The inferred graphs at 2000 timesteps are shown in Figure 3.
In the interest of space, we have shown the inferred graph by MSGI + in the appendix in Figure 8.
PSGI infers the parameterized graph using ﬁrst-order logic, and thus it is more compact.
On the other hand, MSGI + infers the subtask graph without parameterizing out the shared structure, resulting in non-compact graph with hundreds of subtasks and options.
Moreover, graph inferred by PSGI has 0% error in precondition and effect model inference.
The graph inferred by MSGI + has 38% error in the preconditions (the six options that MSGI + completely failed to infer any precondition are not shown in the ﬁgure for readability).
The result shows that the parameterized graph can infer the task structure much more efﬁciently than the subtask graph by sharing the experiences of different options using the parameterized options and subtasks.
Conclusion
In this work we presented parameterized subtask graph inference (PSGI), a method for efﬁciently inferring the latent structure of hierarchical and compositional tasks.
PSGI also facilitates inference of unseen subtasks during adaptation, by inferring relations using predicates.
PSGI additionally learns parameter attributes in a zero-shot manner, which differentiate the structures of different predicate subtasks.
Our experimental results showed that PSGI is more efﬁcient and more general than prior work.
In future work, we aim to to tackle noisy settings, where options and subtasks exhibit possible failures, and settings where the option policies must also be learned. A Appendix [Figure]
B Details on the Method B.1 Parameterized Subtask Graph MLE Derivation
Recall in the Parameterized Subtask Graph Inference section our goal is to infer the maximum likelihood factored subtask graph G given a trajectory τ H .
[Equation]
(19) where τ H = s 1 , o 1 , r 1 , d 1 , . .
, s H } is the adaptation trajectory of the adaptation policy π adapt θ after
H time steps.
We can expand the likelihood term as:
[Equation]
where we dropped terms independent of G .
From our deﬁnitions of the Parameterized Subtask
Graph problem, the predicate precondition G prec determines the mapping from completion x to option eligibility e , x (cid:55)→ e , the predicate effect G eff determines the mapping from completion and option to completion, ( x t , o )
(cid:55)→ x t +1 , and ﬁnally, the predicate reward G r determines the reward given when a subtask is completed at time t .
Then, we can rewrite the MLE as:
[Equation]
+1 , G r ) (cid:19) (28)
The rest of PSGI follows in maximizing (cid:98) G prec , (cid:98) G eff , and (cid:98) G r individually.
C Details on the Tasks
Cooking The Cooking domain is modelled after cooking interactions from the AI2Thor simulated home environment Kolve et al.
The agent has 4 main interactions in the environment: pickup , put , slice , and cook .
The agent can execute options to move objects with pickup and put , and execute options to change object states with slice and cook .
Figure 1 shows a simpliﬁed version of the preconditions and effects of interactions.
The full parameterized subtask graph is shown in Figure 3, which was correctly inferred by the PSGI agent.
Mining The Mining domain is modelled after the open world video game Minecraft and the domain introduced by Sohn et al.
Similar to the mining from Sohn et al.
[2018], the agent hasmain interactions in the environment: get , light , smelt , and craft .
The agent may retrieve/mine materials with get , use light and smelt to prepare materials in order to craft them into usable tools.
However, Mining has additional added complexity from Sohn et al.
Mining has one more “tier” of mining difﬁculty — a stone pickaxe must be used to mine iron, and a iron pickaxe must be used to mine diamond.
This makes Mining signiﬁcantly more difﬁcult for the agent, and more closely matches the gameplay in the Minecraft video game.
Mining also has many more materials added than Sohn et al.
E.g. in some tasks the agent will encounter stone , iron , copper , gold , diamond , etc, each material with similar or different latent attributes.
Again, this makes the task more difﬁcult, but more similar to Minecraft.
The latent parameterized subtask graph of Mining is shown in Figure 6.
D Details on the Baselines and Hyperparameters
HRL The baseline HRL is an actor-critic model over pre-learned options Andreas et al.
As our compared approach PSGI utilizes the entity embeddings (used for zero-shot learning entity attributes), we use an architecture for HRL that uses attention over the entities as well as the observations, following Luong et al.
We brieﬂy describe this architecture.
Let x ∈ { 0 , 1 } N , e ∈ { 0 , 1 } M be the completion and eligibility vector respectively, where there are
N subtasks and M options.
Let E x ∈ R N,D , E e ∈ R N,D be the entity embeddings for each subtask and option entities concatenated.
Let D (cid:48) be the embedding dimension.
We apply attention over the observations by:
[Equation]
Similarly, we also use attention to calculate the option logits:
[Equation]
We searched through the following hyperparameters for HRL in a simple grid search.
HRL hyperparameters
Learning Rate {1e-4, 2.5e-4}
Entropy Cost {0.01, 0.03}
Baseline Cost 0.
N -step horizon
Discount 0.
MSGI+ We implement MSGI + following work from Sohn et al.
[2020], however, we adjust prior work to additionally infer option effects (where previous options were assumed to only complete singular subtasks), by using the PSGI effect model, but without leveraging the smoothness assumptions.
I.e. we directly infer effect from Equation (16), or, skipping step 2. of Figure 5.
We use the following hyperparameters for MSGI + .
MSGI + hyperparameters
Exploration count-based
GRProp Temperature 200.
GRProp w a 3.
GRProp β a 8.
GRProp (cid:15) or 0.
GRProp t or 2.
PSGI
We use the following hyperparameters for PSGI .
We use mostly similar parameters to
MSGI + .
PSGI hyperparameters
Exploration count-based
[Figure]
E Additional Experiments
We additionally wanted to ask the following research questions:
Can PSGI generalize to unseen subtasks with unseen entities using the inferred attributes?
How does the quality of the PSGI’s prior affect transfer learning performance?
E.1 Zero Shot Learning Attributes
In our experiments
[Figure]
we assume the ﬁrst entity of every subtask and option serves as a “verb” entity (e.g. pickup , cook , etc.).
We assume there is there is no shared structure across subtasks and options with different verbs.
As described in section 3.1, (when attributes are not provided), we infer attributes from an exhaustive powerset of all possible features on seen parameters.
The attributes that are used for the graph are then likely to be semantically meaningful, as the decision tree selects the most efﬁcient features.
Hence, to test whether PSGI is generalizable, we can evaluate whether attributes are accurately inferred for unseen parameters when only given the ground truth attributes on seen parameters (given that PSGI will infer the ground truth for the seen parameters).
We measure the generalization error of PSGI if some “weak" signal is provided through parameters.
We suppose the word labels for options and subtasks are provided in Cooking .
I.e. the words for parameters “pickup", “apple", etc. are known.
Then, we can infer low level (but semantically meaningful) features from these words by using word embeddings to encode the parameters Pennington et al.
We choose to use 50 dimensional GloVE word embeddings from Pennington et al.
We then evaluate by measuring the accuracy of attributes for 20 additional unseen test parameters, all words related to kitchens and cooking.
We show the results in Table 1.
From these results, we can extrapolate that at least 70% of edges (on unseen entities) in the predicate subtask graph using these attributes are accurate.
E.2 Effect of the Prior
As described in the Task Transfer and Adaptation section, in PSGI, the training phase is used to learn a prior parameterized subtask graph, (cid:98) G prior .
(cid:98) G prior is then used in an ensemble with the the test policy’s learned graph (cid:98) G test .
Then, we can infer that the more accurate (cid:98) G prior is, the better the transfer policy will perform.
To study this, we varied the number of timesteps used to learn the priors in the Cooking [Table] and Mining domains.
We trained priors T prior = 100 , 500 , and 2000 timesteps respectively, shown in
Figure 7.
In the Cooking domain, we can see that when T prior is higher, the average return and success rate increases.
However, in the Mining domain, we see no obvious correlation between T prior and performance.
We reason that this may be from a number of factors — the prior graphs may not have signiﬁcant difference between T prior = 100 , 500 , and 2000 timesteps, or that (cid:98) G prior is signiﬁcantly different from the latent parameterized subtask graph during testing, rendering the prior less useful.
[Table]
[Figure]