G ENERALISATION IN L IFELONG R EINFORCEMENT L EARNING THROUGH L OGICAL C OMPOSITION
A BSTRACT
We leverage logical composition in reinforcement learning to create a framework that enables an agent to autonomously determine whether a new task can be immediately solved using its existing abilities, or whether a task-speciﬁc skill should be learned.
In the latter case, the proposed algorithm also enables the agent to learn the new task faster by generating an estimate of the optimal policy.
Importantly, we provide two main theoretical results: we bound the performance of the transferred policy on a new task, and we give bounds on the necessary and sufﬁcient number of tasks that need to be learned throughout an agent’s lifetime to generalise over a distribution.
We verify our approach in a series of experiments, where we perform transfer learning both after learning a set of base tasks, and after learning an arbitrary set of tasks.
We also demonstrate that, as a side effect of our transfer learning approach, an agent can produce an interpretable Boolean expression of its understanding of the current task.
Finally, we demonstrate our approach in the full lifelong setting where an agent receives tasks from an unknown distribution.
Starting from scratch, an agent is able to quickly generalise over the task distribution after learning only a few tasks, which are sub-logarithmic in the size of the task space.
I NTRODUCTION
Reinforcement learning (RL) is a framework that enables agents to learn desired behaviours by maximising the rewards received through interaction with an environment (Sutton et al., 1998).
While
RL has achieved recent success in several difﬁcult, high-dimensional domains (Mnih et al., 2015;
Levine et al., 2016; Lillicrap et al., 2016; Silver et al., 2017), these methods require millions of samples from the environment to learn optimal behaviours.
This is ultimately a fatal ﬂaw, since learning to solve complex, real-world tasks from scratch for every task of interest is typically infeasible.
Hence a major challenge in RL is building general-purpose agents that are able to use existing knowledge to quickly solve new tasks in the environment.
The question of interest is then: after learning n tasks sampled from some distribution, how can an agent transfer or leverage the skills learned from those n tasks to improve its starting performance or learning speed in task n + 1 ?
This problem setting is formalised by lifelong RL (Thrun, 1996; Abel et al., 2018).
One approach to transfer in lifelong RL is composition (Todorov, 2009), which allows an agent to leverage its existing skills to build complex, novel behaviours that can then be used to solve or speed up learning of a new task (Todorov, 2009; Saxe et al., 2017; Haarnoja et al., 2018; van Niekerk et al., 2019; Hunt et al., 2019; Peng et al., 2019).
Recently, Nangue Tasse et al.
(2020) proposed a framework for deﬁning a
Boolean algebra over the space of tasks and their optimal value functions.
This allowed for tasks and value functions to be composed using the union, intersection and negation operators in a principled manner to yield optimal skills zero-shot.
In this work, we propose a framework for lifelong RL that focuses not only on transfer between tasks for faster RL, but also provides guarantees on the generalisation of an agent’s skills over an unknown task distribution.
We ﬁrst extend the logical composition framework of Nangue Tasse et al.
(2020) to discounted and stochastic tasks.
We provide theoretical bounds for our approach in stochastic settings, and also compare them to previous work in the discounted setting.
We then show how our framework leverages logical composition to tackle the lifelong RL problem.
The framework enables agents to iteratively solve tasks as they are given, while at the same time constructing a library of skills that can be composed to obtain behaviours for solving future tasks faster, or even without further learning.
We empirically verify our framework in a series of experiments, where an agent is i) pretrained on a set of base tasks provided by the Boolean algebra framework, and ii) when the pretrained tasks are not base tasks.
We show that agents here are able to achieve signiﬁcant jumpstarts on new tasks.
Finally, we demonstrate our framework in the lifelong RL setting where an agent receives tasks from an unknown (possibly non-stationary) distribution and must determine what skills to learn and add to its library, and how to combine its current skills to solve new tasks.
Results demonstrate that this framework enables agents to quickly learn a set of skills, resulting in a combinatorial explosion in their abilities.
Consequently, even when tasks are sampled randomly from an unknown distribution, an agent can leverage its existing skills to solve new tasks without further learning, thereby generalising over task distributions.
B ACKGROUND
We consider tasks modelled by Markov Decision Processes (MDPs).
An MDP is deﬁned by the tuple ( S , A , p, r, γ ) , where (i) S is the state space, (ii) A is the action space, (iii) p ( s (cid:48) | s, a ) is a
Markov transition probability, (iv) r is the real-valued reward function bounded by [ r MIN , r MAX ] , and (v) γ ∈ [0 , 1) is the discount factor.
In this work, we focus on tasks where an agent is required to reach a set of desirable goals in a goal space G ⊆ S (a set of boundary states).
Here, termination in
G is modelled similarly to van Niekerk et al.
by augmenting the state space with a virtual state, ω , such that p ( ω | s, a ) = 1 ∀ ( s, a ) ∈ ( G × A ) and the rewards are zero after reaching ω .
We hence consider the set of tasks
M such that the tasks are in the same environment—described by a background MDP ( S , A , p, γ, r 0 ) —and each task can be uniquely speciﬁed by a set of desirable and undesirable goals:
[Equation]
The goal of the agent is to compute a Markov policy π from S to A that optimally solves a given task.
A given policy π is characterised by a value function V π ( s ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t )] , specifying the expected return obtained under π starting from state s .
The optimal policy π ∗ is the policy that obtains the greatest expected return at each state:
V π ∗ ( s ) =
V ∗ ( s ) = max π V π ( s ) for all s in S .
A related quantity is the Q -value function, Q π ( s, a ) , which deﬁnes the expected return obtained by executing a from s , and thereafter following π .
Similarly, the optimal Q -value function is given by
Q ∗ ( s, a ) = max π Q π ( s, a ) for all s in S and a in A .
2. L OGICAL COMPOSITION
Nangue Tasse et al.
(2020) recently proposed the notion of a Boolean task algebra, which allows an agent to perform logical operations—conjunction ( ∧ ), disjunction ( ∨ ) and negation ( ¬ )—over the space of tasks and value functions.
While they only considered deterministic shortest path tasks ( γ = 1 with deterministic dynamics), we summarise their approach here and later extend it to discounted stochastic tasks (Section 3.1).
To achieve zero-shot logical composition, Nangue Tasse et al.
(2020) extend the standard rewards and value functions used by an agent to deﬁne goal-oriented versions as follows:
Deﬁnition 1.
The extended reward function ¯ r : S × G ×
A → R is given by the mapping
[Equation]
where ¯ r MIN ≤ min { r MIN , ( r MIN − r MAX ) D } , and D is the diameter of the MDP (Jaksch et al., 2010).
Deﬁnition 2.
The extended Q-value function ¯ Q : S × G ×
A → R is given by the mapping
[Equation]
where ¯ V ¯ π ( s, g )
=
E ¯ π [ (cid:80) ∞ t =0 ¯ r
( s t , g, a t )] .
By penalising the agent for achieving goals different from those it wanted to reach ( ¯ r MIN if g (cid:54) = s and s ∈ G )
, the extended reward function has the effect of driving the agent to learn how to separately achieve all desirable goals.
Importantly, the standard reward and value functions can be recovered from their extended versions by simply maximising over goals.
As such, the agent can also recover the task policy by maximising over both goals and actions:
[Equation]
The logic operators over tasks and extended action-value functions are then deﬁned as follows:
Deﬁnition 3.
Let M be a set of tasks with bounds M MIN ,
M MAX ∈ M such that,
[Equation]
Deﬁne the ¬ , ∨ , and ∧ operators over M as ¬ ( M ) :
= ( S , A , p, r ¬ M ) , where r ¬ M ( s, a ) :=
( r M MAX ( s, a ) + r M MIN ( s, a )) − r M ( s, a ) ∨ ( M 1 , M 2 ) :
= ( S , A , p, r M 1 ∨ M 2 ) , where r M 1 ∨ M 2 ( s, a ) :
= max { r M 1 ( s, a ) , r M 2 ( s, a ) } ∧ ( M 1 , M 2 ) :
= ( S , A , p, r M 1 ∧ M 2 ) , where r M 1 ∧ M 2 ( s, a ) :
=
min { r M 1 ( s, a ) , r M 2 ( s, a ) }
Deﬁnition 4.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M , with bounds ¯ Q ∗ MIN , ¯ Q ∗ MAX ∈ ¯ Q ∗ which are respectively the optimal ¯ Q -functions for the tasks
M MIN , M MAX ∈ M .
Deﬁne the ¬ , ∨ , and ∧ operators over ¯ Q ∗ as,
[Equation]
Using the deﬁnitions for the logical operations over M and ¯ Q ∗ given above, Nangue Tasse et al.
(2020) construct a Boolean algebra over tasks and extended value functions.
Furthermore, by leveraging the goal-oriented deﬁnition of extended value functions, they also show that M and ¯ Q ∗ are homomorphic.
As a result, if a task can be expressed using the Boolean algebra, the optimal value function for the task can immediately be computed.
This enables agents to solve any new task that is given as the logical combination of learned ones.
L IFELONG TRANSFER THROUGH COMPOSITION
In lifelong RL, an agent is presented with a series of tasks sampled from some distribution D .
The agent then needs to not only transfer knowledge learned from previous tasks to solve new but related tasks quickly, but it also should not forget learned knowledge in the process.
We formalise this lifelong learning problem as follows:
Deﬁnition 5.
Let D be an unknown, possibly non-stationary, distribution over a set of tasks
M ( S , A , p, γ, r 0 ) .
The lifelong learning problem consists of the repetition of the following steps for t ∈ N :
The agent is given a task M t ∼ D ( t ) , 2.
The agent interacts with the MDP M t until it is (cid:15) -optimal in M 0 , ..., M t .
This formulation of lifelong RL is similar to that of Abel et al.
(2018); the main difference is that we do not assume that D is stationary, and we explicitly require an agent to retain learned skills.
As discussed in the introduction, one of the main goals in this setting is that of transfer (Taylor &
Stone, 2009).
We add an important question to this setting: how many tasks should an agent learn during its lifetime in order to generalise over the task distribution?
In other words, how many tasks should it learn to be able to solve any new task immediately?
While most approaches focus on the goal of transfer, the question of the number of tasks is often neglected by simply assuming the case where the agent has already learned n tasks (Abel et al., 2018; Barreto et al., 2018).
Consider, for example, a task space with only |G| = 40 goals.
Then, given the combination of all possible goals, the size of the task space is |M| = 2 |G| ≈ 10 12 .
If D is a uniform distribution over |M| , then for most transfer learning methods an agent will have to learn most of the tasks it is presented with, since the probability of observing the same task will be approximately zero.
This is clearly impractical for a setting like RL, where learning methods often have a high sample complexity even with transfer learning.
It is also extremely memory inefﬁcient, since the learned skills of most tasks must be stored.
3. E XTENDING THE B OOLEAN ALGEBRA FRAMEWORK
In this section, we show how logical composition can be leveraged to learn a subset of tasks that is sufﬁcient to generalise over the task distribution.
Since the logical composition results of
Nangue Tasse et al.
(2020) were only shown for deterministic shortest path tasks (where γ = 1 ), we extend the framework to discounted and stochastic tasks M (Equation 1).
To achieve this, we ﬁrst redeﬁne the extended reward function (Deﬁnition 1) to use the simpler penalty ¯ r MIN = r MIN .
We also redeﬁne ¬ over ¯ Q ∗ as follows:
[Equation]
The intuition behind this re-deﬁnition of the negation operator is as follows: since each goal is either desirable or not, the optimal extended value function ¯ Q ∗ ( s, g, a ) is either ¯ Q ∗ MAX ( s, g, a ) or ¯ Q ∗ MIN ( s, g, a ) .
Hence, if ¯ Q ∗ ( s, g, a ) is closer to ¯ Q ∗ MIN ( s, g, a ) , then its negation should be ¯
Q ∗ MAX ( s, g, a ) , and vice versa.
For tasks in M , this is equivalent to the previous deﬁnition of ¬ for optimal ¯ Q -value functions, but it will give us tight bounds when composing (cid:15) -optimal ¯ Q -value functions (see Theorem 1).
We now show that the Boolean algebra and zero-shot composition results of Nangue Tasse et al.
(2020) also hold for tasks in M .
Proposition 1.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Then,
(i) M and ¯ Q ∗ respectively form a Boolean task algebra ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) and a Boolean extended value functions algebra ( ¯
[Equation]
(ii) A is a homomorphism between M and ¯ Q ∗ .
We can now solve any new task in M zero-shot if we are given the correct Boolean expression that informs the agent how to compose its optimal skills.
This is essential for the following results.
3. T RANSFER BETWEEN TASKS
In this section, we leverage the logical composition results to address the following question of interest: given an arbitrary set of learned tasks, can we transfer their skills to solve new tasks faster?
As we will show in Theorem 1, we answer this question in the afﬁrmative.
To achieve this, we ﬁrst note that each task M ∈ M can be associated with a binary vector T ∈ { 0 , 1 } |G| which represents its set of desirable goals, as illustrated by the tasks in Table 1.
The approximation ˜ T of this task representation can be learned just from task rewards ( r M ( s, a ) ) by simply computing ˜ T ( s )
= 1 r M ( s,a )=
r MAX at each terminal state s that the agent reaches.
We can then use any generic method, such as the sum-ofproducts ( SOP ), to determine a candidate Boolean expression ( B EXP ) in terms of the learned binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } of a set of past tasks ˆ M = { M 1 , ..., M n } ⊆ M .
An estimate of the optimal ¯ Q -value function of M can then be obtained by composing the learned ¯ Q -value functions ˜¯
Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } according to B EXP .
Theorem 1 shows the optimality of this process.
Theorem 1.
Let M ∈ M be a task with reward function r , binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T F =
[Equation]
EXP (
[Equation]
and ¯ Q F =
[Equation] [Equation]
Q ∗ n ) , where B EXP is derived from ˜ T n and ˜ T using a generic method F .
[Equation]
F where Q F := max
[Equation]
¯ Q F ( s, g, a ) .
Then,
[Equation]
(ii) if the dynamics are deterministic,
[Equation]
where 1 is the indicator function, r g ( s, a ) :
=
¯ r
( s, g, a
[Equation]
, r ∆ := r MAX − r
[Equation]
, and (cid:107) f −
[Equation]
(cid:107) ∞
[Equation]
=
[Equation]
| .
Theorem 1(i) states that if ¯ Q F is close to optimal, then acting greedily with respect to it is also close to optimal.
Interestingly, this is similar to the bound obtained by Barreto et al.
(Proposition 1) for transfer learning using generalised policy improvement (GPI), but stronger.
This is unsurprising, since π ( s ) ∈ arg max a ∈A max g ∈G ¯ Q F ( s, g, a ) can be interpreted as generalised policy improvement on the set of goal policies of the extended value function ¯ Q F .
Importantly, if the environment is deterministic, then we obtain a strong bound on the composed value functions (Theorem 1(ii)).
This bound shows that transfer learning using logical composition is (cid:15) -optimal—that is, there is no loss in optimality—when the new task is expressible as a logical combination of past ones.
With the exponential nature of logical combination, this gives agents a strong generalisation ability over the task space—and hence over any task distribution—as we will show in Theorem 2.
3. G ENERALISATION OVER A TASK DISTRIBUTION
We leverage Theorem 1 to design an algorithm that combines the SOP approach with goal-oriented learning to achieve fast transfer in lifelong RL.
Given an off-policy RL algorithm A , the agent initializes its extended value function ˜¯ Q , the task binary vector ˜ T , and a goal buffer.
At the beginning of each episode, the agent computes T SOP and Q SOP for ˜ T using the SOP method and its library of learned task vectors and extended Q-functions.
It then acts using the behaviour policy ( (cid:15) -greedy for example) of A with ¯ Q SOP for the action-value function if T SOP =
˜ T , and ¯ Q SOP ∨ ˜¯ Q otherwise.
If T SOP (cid:54) =
˜ T , the agent also updates ˜¯ Q for each goal in the goal buffer using A .
Additionally, when the agent reaches a terminal state s , it adds it to the goal buffer and updates ˜ T ( s ) using the reward it receives ( ˜ T ( s ) = 1 r M ( s,a )=
r MAX ).
Training stops when the agent has reached the desired level of optimality (or after n episodes in practice), after which the agent adds the learned ˜ T and ˜¯ Q to its library if T SOP (cid:54) =
˜ T .
The full algorithm is included in Appendix B.
We refer to this algorithm as
SOPGOL ( Sum Of Products with Goal-Oriented Learning ).
When G is ﬁnite, we show in Theorem 2 that SOPGOL generalises over any unknown task distribution after learning only a number of tasks logarithmic in the size of the task space.
The lower bound is (cid:100) log |G|(cid:101) , since this is the minimum number of tasks that span the task space, as can be seen in Table 1 (top) for example.
The upper bound is |G| because that is the dimensionality of the task binary representations { 0 , 1 } |G| .
Since the number of tasks is |M| = 2 |G| , we have that the upper bound |G| = log |M| is logarithmic in the size of the task space.
Since ¯ Q SOP ∨ ˜¯ Q = max { ¯ Q SOP , ˜¯ Q } , it is equivalent to GPI and hence is guaranteed to be equal or more optimal than the value functions.
Hence using ¯ Q SOP ∨ ˜¯ Q in the behaviour policy gives a straightforward way of leveraging ¯ Q SOP to learn ˜¯ Q faster.
Theorem 2.
Let D be an unknown, possibly non-stationary, distribution over a set of tasks
M ( S , A , p, γ, r 0 ) with ﬁnite G .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗
M for all M in M .
Let ˜
T t +1 , ˜¯ Q ∗ t +1 =
SOPGOL ( A , M t , ˜ T t , ˜¯ Q ∗ t ) where M t ∼ D ( t ) and ˜ T
= ˜¯ Q ∗ 0 = ∅ ∀ t ∈ N .
Then,
[Equation]
where N t := | ˜
T t | = | ˜¯ Q ∗ t | .
Interestingly, Theorem 2 holds even in the case where a new task is expressible in terms of past tasks ( T SOP =
˜ T ), but we wish to solve it to a higher degree of optimality than past tasks.
In this case, we can pretend T SOP (cid:54) =
˜ T and learn a new ¯ Q -function to the desired degree of optimality.
We can then add it to our library, and remove any other skill from our library (the least optimal for example).
E XPERIMENTS
The red triangle represents the agent.
Goals
T a 1 0 1 0
T b 0 0 1 1
T c 0 1 1 0
T d 0 0 0 1
Goals
T0 0 0 0 01 T0 0 1 1 10 T
[Table]
tasks.
0 or 1 corresponds to a goal reward of r MIN or r MAX .
4. T RANSFER AFTER PRETRAINING ON A SET OF TASKS
We consider the P ICK U P O BJ domain from the M INI G RID environment (Chevalier-Boisvert et al., 2018), illustrated by Figure 1, where an agent must navigate in a 2D room to pick up objects of various shapes and colours from pixel observations.
This type of domain is prototypical in the literature (Nangue Tasse et al., 2020;
Barreto et al., 2020; van Niekerk et al., 2019; Abel et al., 2018), because it allows for easy demonstration of transfer learning in many-goal tasks.
In this domain, there are |G| = 15 goals each corresponding to picking up objects of 3 possible types—box, ball, key—and 5 possible colours—red, blue, green, purple, and yellow.
Hence a set of (cid:100) log 2 |G|(cid:101)
= 4 base tasks can be selected that can be used to solve all 2 |G| = 32768 possible tasks under a Boolean composition of goals.
The agent receives a reward of 2 when it picks up desired objects, and − 0 .otherwise.
For all of our experiments in this section, we use deep Q-learning (Mnih et al., 2015) as the RL method for SOPGOL and as the performance baseline.
We also compare SOPGOL to
SOPGOL-transfer, or to SOPGOL-continual.
SOPGOL-transfer refers to when no new skill is learned and SOPGOL-continual refers to when a new skill is always learned using the SOP Q estimate to speed up learning, even if the new task could be solved zero shot.
Since SOPGOL determines automatically which one to use, we compare whichever one it chooses with the other one in each of our experiments.
We ﬁrst demonstrate transfer learning after pretraining on a set of base tasks—a minimal set of tasks that span the task space.
This can be done if the set of goals is known upfront, by ﬁrst assigning a
Boolean label to each goal in a table and then using the rows of the table as base tasks.
These are illustrated in Table 1 (top).
Having learned the (cid:15) -optimal extended value functions for our base tasks,
3.0 Episodes 1e
R e t u r n s
SOPGOLSOPGOL continual DQN     5 Episodes 1e
G o a l s (a) M0.
2.5 Episodes 1e
R e t u r n s
SOPGOLSOPGOL continual DQN     5 Episodes 1e
G o a l s (b)
M       7 Episodes 1e
R e t u r n s
SOPGOLSOPGOL continual DQN     5 Episodes 1e
G o a l s (c) M
[Equation]
we can now leverage logical composition for transfer learning on test tasks.
We consider the three test tasks shown in Table 1 (bottom).
For each, we run SOPGOL, SOPGOL-continual
[Figure]
and a standard
DQN.
Figure 2 illustrates the results where, as predicted by our theoretical results in Section 3.2,
SOPGOL correctly determines that the current test tasks are solvable from the logical combinations of the learned base tasks.
Its performance from the start of training is hence the best.
Now that we have demonstrated how SOPGOL enables an agent to solve any new task in an environment after training on base tasks
[Figure]
we consider the more practical case where new tasks are not fully expressible as a Boolean expression of previously learned tasks.
The agent in this case
[Figure] [Figure]
, Mand M 3 after pretraining on the non-base set of tasks , , and .
The shaded regions on the episodic returns indicate one standard deviation over 4 runs.
The learned binary representations are similarly averaged over 4 runs, and reported for the ﬁrst 500 episodes.
The initial drop in DQN performance is a result of the initial exploration phase where the exploration constant decays from 0.5 to 0.05.
The
Boolean expressions generated by SOPGOL for the respective test tasks are:
[Equation]
is pretrained on a set of tasks that do not span the task space, { , , , } , corresponding to the tasks of picking up green objects, blue objects, yellow objects, and keys.
We then train the agent with SOPGOL, SOPGOL-transfer, and a standard DQN on the same set of test tasks considered previously (Table 1 (bottom)).
The results in Figure 3 demonstrate how SOPGOL now chooses to learn a task-speciﬁc skill after transfer, and hence outperforms SOPGOL-transfer since the test tasks are not entirely expressible in terms of the pretrained ones.
Consider Figure 3a, for example.
The test task is to pick up a yellow box, but the agent has only learned how to pick up red objects, blue objects, yellow objects, and keys.
It has not learned how to pick up boxes.
However, we note from the inferred Boolean expression ( (cid:102) M 1 ) that the agent correctly identiﬁes that the desired objects are, at the very least, yellow.
Without further improvements to this transferred policy (SOPGOL-transfer), we can see that this approach outperforms DQN from the start.
This is due to two main factors: (i) the transferred policy navigates to objects more reliably, so takes fewer random actions; and (ii) although the transferred policy does not have a complete understanding of which are the desirable objects, it at least navigates to yellow objects, which are sometimes yellow boxes.
Finally, since SOPGOL is able to determine that the current task is not entirely expressible in terms of its previous tasks (by checking whether T SOP =
˜ T ), it is able to learn a new ¯ Q -value function that improves on the transferred policy.
Additionally, its returns are strictly higher than those of
SOPGOL-transfer because SOPGOL learns the new ¯ Q -value function faster by using ¯ Q SOP ∨ ˜ ¯ Q in the behaviour policy.
4. L IFELONG TRANSFER
In this section, we consider the more general setting where the agent is not necessarily given pretrained skills upfront, but is rather presented with tasks sampled from some unknown distribution.
We revisit the example given in Section 3, but now more concretely by using a stochastic Four Rooms domain (Sutton et al., 1999), with a goal space of size |G| = 40 and a task space of size |M| = 2 |G| ≈ 10 12 .
Complete environment details are given in Appendix C.
We demonstrate the ability of SOPGOL to generalise over task distributions by evaluating the approach with the following distributions: (i) D sampled : the goals for each task are chosen uniformly at random over G ; (ii) D best : the ﬁrst (cid:100) log 2 |G|(cid:101) tasks are the base tasks, while the rest follow
D sampled .
This distribution gives the agent the minimum number of tasks to learn and store, since the agent learns the base tasks ﬁrst before being presented with any other task.
(iii) D worst : the ﬁrst |G| tasks are each deﬁned by a single goal that differs from the previous tasks, while the rest follow
D sampled .
This distribution forces the agent to learn and store the maximum number of tasks, since none of the |G| tasks can be expressed as a logical combination of the others.
We use Q-learning (Watkins, 1989) as the RL method for SOPGOL, and Q-learning with max
Q initialisation as a baseline.
This has been shown by previous work (Abel et al., 2018) to be a practical method of initialising value functions with a theoretically optimal optimism criterion that speeds-up convergence during training.
Our results (Figure 4) show that SOPGOL enables a lifelong agent to quickly generalise over an unknown task distribution.
Interestingly, both graphs show that the convergence
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
speed during a randomly sampled task distribution D sampled is very close to that of the best task distribution D best .
This suggests that there is room to make the bound in Theorem 2 even tighter by making some assumptions on the task distribution—an interesting avenue for future work.
R ELATED WORK
There have been several approaches in recent years for tackling the problem of transfer in lifelong
RL.
Most closely related is the line of work on concurrent skill composition (Todorov, 2009; Saxe et al., 2017; Haarnoja et al., 2018; van Niekerk et al., 2019; Hunt et al., 2019).
These methods usually focus on multi-goal tasks, where they address the combinatorial amount of desirable goals by composing learned skills to create new ones.
Given a reward function that is well approximated by a linear function, Barreto et al.
(2020) propose a scheme for few-shot transfer in RL by combining
GPI and successor features (SF) (Barreto et al., 2017).
In general, approaches based on GPI with SFs (Barreto et al., 2021) are suitable for tasks deﬁned by linear preferences over features (latent goal states).
Given the set of features for an environment, Alver & Precup (2022) shows that a base set of successor features can be learned, which is sufﬁcient to span the task space.
While these approaches also support tasks where goals are not terminal, the smallest number of successor features that must be learned to span the task space is |G| (the upper-bound in Theorem 2).
Our work is similar to these approaches in that it can be interpreted as performing GPI with the logical composition of extended value functions, which leads to stronger theoretical bounds than GPI with the linear composition of successor features (see Appendix A.4).
Finally, none of these works consider the lifelong RL setting where an agent starts with no skill and receives tasks sampled from an unknown distribution (without additional knowledge like base features or true task representations).
In contrast, SOPGOL is able to handle this setting with logarithmic bounds on the number of skills needed to generalise over the task distribution (Theorem 2).
Other approaches like options (Sutton et al., 1999) and hierarchical RL (Barto & Mahadevan, 2003) address the lifelong RL problem via temporal compositions.
These methods are usually focused on single-goal tasks, where they address the potentially long trajectories needed to reach a desired goal by composing sub-goal skills sequentially (Levy et al., 2017; Bagaria & Konidaris, 2019).
While they do not consider the multi-goal setting, they can be used in conjunction with concurrent composition to learn how to achieve a combinatorial amount of desirable long horizon goals.
Finally, there are also non-compositional approaches (Finn et al., 2017; Abel et al., 2018; Singh et al., 2021), which usually aim to learn the policy for a new task faster by initializing the networks with some pre-training procedure.
These can be used in combination with SOPGOL to learn new skills faster.
C ONCLUSION
In this work, we proposed an approach for efﬁcient transfer learning in RL.
Our framework, SOPGOL, leverages the Boolean algebra framework of Nangue Tasse et al.
(2020) to determine which skills should be reused in a new task.
We demonstrated that, if a new task is solvable using existing skills, an agent is able to solve it with no further learning.
However, even if this is not the case, an estimate of the optimal value function can still be obtained to speed up training.
This allows agents in a lifelong learning setting to quickly generalise over any unknown (possibly non-stationary) task distribution.
The main limitation of this work is that it only consider tasks with binary goal rewards—where goals are either desirable or not.
Although this covers a vast number of many-goal tasks, combining our framework with works on weighted composition (van Niekerk et al., 2019; Barreto et al., 2020) could enable a similar level of generalisation over tasks with arbitrary goal rewards.
Another exciting avenue for future work would be to extend our transfer learning and generalisation results to include temporal tasks by leveraging temporal composition approaches like options.
Finally, we note that just like previous work, we rely on the existence of an off-the-shelf RL method that is able to learn goal-reaching tasks in a given environment.
Since that is traditionally very sample inefﬁcient, our framework can be complemented with other transfer learning methods like M AX QI NIT (Abel et al., 2018) to speed up the learning of new skills (over and above the transfer learning and task space generalisation shown here).
Our approach is a step towards the goal of truly general, long-lived agents, which are able to generalise both within tasks, as well as over the distribution of possible tasks it may encounter.
A CKNOWLEDGEMENTS
GNT is supported by an IBM PhD Fellowship.
This research was supported, in part, by the National
Research Foundation (NRF) of South Africa under grant number 117808.
The content is solely the responsibility of the authors and does not necessarily represent the ofﬁcial views of the NRF.
The authors acknowledge the Centre for High Performance Computing (CHPC), South Africa, for providing computational resources to this research project.
Computations were also performed using
High Performance Computing Infrastructure provided by the Mathematical Sciences Support unit at the University of the Witwatersrand
A P ROOFS OF THEORETICAL RESULTS A. B OOLEAN A LGEBRA D EFINITION
Deﬁnition 6.
A Boolean algebra is a set B equipped with the binary operators ∨ (disjunction) and ∧ (conjunction), and the unary operator ¬ (negation), which satisﬁes the following Boolean algebra axioms for a, b, c in B :
(i) Idempotence: a ∧ a = a ∨ a = a .
(ii) Commutativity: a
∧ b = b ∧ a and a ∨ b =
b ∨ a .
(iii) Associativity: a ∧ ( b ∧ c ) =
( a ∧ b ) ∧ c and a ∧ ( b ∨ c )
= ( a ∨ b ) ∨ c .
(iv) Absorption: a ∧ ( a ∨ b ) =
a ∨ ( a ∧ b ) = a .
(v) Distributivity: a ∧ ( b ∨ c )
= ( a ∧ b ) ∨ ( a ∧ c ) and a ∨ ( b ∧ c ) =
( a ∨ b ) ∧ ( a ∨ c ) .
(vi) Identity: there exists 0 , 1 in B such that
[Equation]
(vii) Complements: for every a in B , there exists an element a (cid:48) in B such that a ∧ a (cid:48)
= 0 and a ∨ a (cid:48)
= 1 .
A. P ROOFS FOR P ROPOSITION
Lemma 1.
Let M be a set of tasks.
Then ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) is a Boolean algebra.
Proof.
Let M 1 , M 2 ∈ M .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i)–(v):
These easily follow from the fact that the min and max functions satisfy the idempotent, commutative, associative, absorption and distributive laws.
(vi): Let r M MAX ∧ M 1 and r M 1 be the reward functions for M MAX ∧ M 1 and M 1 respectively.
Then for all ( s, a ) in
S × A , r M MAX ∧ M 1 ( s, a ) = (cid:26) min { r MAX , r M 1 ( s, a ) } , if s ∈ G min { r 0 ( s, a ) , r 0 ( s, a ) } , otherwise.
= (cid:26) r M 1 ( s, a ) , if s ∈ G r 0 ( s, a ) , otherwise.
( r M 1 ( s, a ) ∈ { r MIN , r MAX } for s ∈ G ) =
r M 1 ( s, a ) .
Thus M MAX ∧ M 1 = M 1 .
Similarly M MAX ∨ M 1 = M MAX , M MIN ∧ M 1 = M MIN , and
M MIN ∨ M 1 = M 1 .
Hence M MIN and M MAX are the universal bounds of M .
(vii): Let r M 1 ∧¬ M 1 be the reward function for M 1 ∧ ¬ M 1 .
Then for all ( s, a ) in S × A , r M 1 ∧¬ M 1 ( s, a ) = (cid:26) min { r M 1 ( s, a ) , ( r MAX + r MIN ) − r M 1 ( s, a ) } , if s ∈ G min { r 0 ( s, a ) , ( r 0 ( s, a ) + r 0 ( s, a )) − r 0 ( s, a ) } , otherwise.
=   r MIN , if s ∈ G and r M 1 ( s, a ) =
r MAX r MAX , if s ∈ G and r M 1 ( s, a ) =
r MIN r 0 ( s, a ) , otherwise.
= r M MIN ( s, a ) .
Thus M 1 ∧ ¬ M 1 = M MIN , and similarly M 1 ∨ ¬ M 1 = M MAX .
Lemma 2.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Then ( ¯ Q ∗ , ∨ , ∧ , ¬ , ¯ Q ∗ MAX , ¯ Q ∗ MIN ) is a Boolean Algebra.
Proof.
Let ¯ Q ∗ M 1 , ¯ Q ∗ M 2 ∈ ¯ Q ∗ be the optimal ¯ Q -value functions for tasks M 1 , M 2 ∈ M with reward functions r M 1 and r M 2 .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i
)–(v): These follow directly from the properties of the min and max functions.
(vi): For all ( s, g, a ) in S × G × A ,
[Equation] [Equation]
Similarly,
[Equation]
(vii): For all ( . ) in S × G × A ,
[Equation]
min { ¯ Q ∗ M 1 ( . ) , ¬ ¯ Q ∗ M 1 ( . ) }
= (cid:26) min { ¯ Q ∗ MIN ( . ) , ¯ Q ∗ MAX
[Equation]
} if | ¯ Q ∗ ( . )
− ¯ Q
[Equation]
MIN ( .
[Equation]
| ≤ | ¯ Q ∗ ( .
[Equation]
− ¯ Q
[Equation]
MAX ( . )
| min { ¯ Q ∗ MAX ( . ) , ¯ Q ∗ MIN ( . ) } otherwise , = ¯ Q ∗ MIN ( . ) .
Similarly, ¯ Q ∗ M 1 ∨ ¬ ¯ Q ∗ M 1 = ¯ Q ∗ MAX .
Lemma 3.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M .
Then for all M 1 , M 2 ∈ M , we have (i) ¯ Q ∗¬ M 1 = ¬ ¯ Q ∗ M 1 , (ii) ¯ Q ∗ M 1 ∨ M 2 = ¯ Q ∗ M 1 ∨ ¯ Q ∗ M 2 , and (iii) ¯ Q ∗ M 1 ∧ M 2 = ¯ Q ∗ M 1 ∧ ¯ Q ∗ M 2 .
Proof.
Let M 1 , M 2 ∈ M .
Then for all ( s, g, a ) in S × G × A , (i): ¯ Q ∗¬ M 1 ( s, g, a ) = (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if r ¬ M 1 ( g, a (cid:48) )
=
r MAX ∀ a (cid:48)
∈
A ¯
Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if r M 1 ( g, a (cid:48) ) =
r MIN ∀ a (cid:48) ∈
A ¯
Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if ¯ Q ∗ M 1 ( s, g, a ) =
¯ Q ∗ MIN ( s, g, a ) ¯ Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if | ¯ Q ∗ M 1 ( s, g, a ) − ¯ Q
[Equation]
[Equation]
(iii): Follows similarly to (ii).
Proposition 2.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Then, (i
M
and
¯ Q ∗ respectively form a Boolean task algebra ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) and a Boolean extended value functions algebra ( ¯
[Equation]
(ii
A is a homomorphism between M and ¯ Q ∗ .
Proof.
(i): Follows from Lemma 1 and 2.
(ii): Follows from Lemma 3.
A. P ROOFS FOR T HEOREM
Lemma 4.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Denote ˜¯ Q ∗ M as the (cid:15) -optimal
¯ Q -value function for a task M ∈ M such that
[Equation]
Then for all M 1 , M 2 in M and ( s, g, a ) in S × G × A ,
[Equation]
i
[Equation]
ii)
[Equation]
(iii) (cid:12)(cid:12)(cid:12) ¬
¯
[Equation]
a )
[Equation] [Equation]
Proof.
(i):
[Equation]
[Table]
[Equation]
Lemma 5.
Let M ∈ M be a task with reward function r , binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T F = B EXP
[Equation]
˜
[Equation]
and ¯ Q F = B EXP ( ˜¯
[Equation]
n ) , where B EXP is derived from ˜ T n and ˜ T using a generic method F .
Deﬁne π ( s ) ∈ arg max a ∈A Q F where Q F := max g ∈G ¯ Q F ( s, g, a ) .
Then,
[Equation]
where 1 is the indicator function, r ∆ :=
r MAX − r MIN , and (cid:107) f − h (cid:107) ∞ := max s,g,a | f
( s, g, a ) − h
( s, g, a ) | .
Proof.
|
[Equation]
(Using Lemma 4)
If T = T F , then ¯ Q ∗ ( s, g, a ) = ¯ Q ∗F ( s, g, a ) , and we are done.
Let T (cid:54) =
T F .
Without loss of generality, let ¯ Q ∗ ( s, g, a ) =
¯ Q ∗ MAX ( s, g, a ) and ¯ Q ∗F ( s, g, a ) =
¯ Q ∗ MIN ( s, g, a ) .
Then,
[Equation]
Lemma 6.
Let Q ∗ and ¯ Q ∗ be the optimal Q -value function and optimal extended Q -value function respectively for a deterministic task in M .
Then for all ( s, a ) in S × A , we have
[Equation]
Proof.
We ﬁrst note that
[Equation]
Now deﬁne
[Equation]
Then it follows that
[Equation]
Since p is
[Equation]
Using Equation 4)
[Equation]
since ¯ r ( s, g, a ) =
r 0 ( s, a ) ∀ s / ∈ G and p ( s, a, ω )
= 1 with ¯ Q ∗ ( ω, g, a (cid:48) )
= 0 ∀ s ∈ G .
[Equation]
Hence ¯ Q ∗ max is a ﬁxed point of the Bellman optimality operator.
If s ∈ G , then
[Equation]
Since ¯ Q ∗ max = Q ∗ holds in G and ¯ Q ∗ max is a ﬁxed point of the Bellman operator, then ¯ Q ∗ max =
Q ∗ holds everywhere.
Theorem 1.
Let M ∈ M be a task with reward function r , binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T F = B EXP ( ˜ T n ) and ¯ Q F = B EXP ( ˜¯ Q ∗ n ) , where B EXP is derived from ˜ T n and ˜ T using a generic method F .
Deﬁne π ( s
) ∈
arg max
a
∈A Q F
where
Q
F := max g ∈G ¯
Q
F
s, g, a )
Then, (i) (cid:107) Q
[Equation]
Q π (cid:107)
[Equation]
γ (( 1 T (cid:54)
[Equation]
T F + 1 r/ ∈{ r g }
[Equation]
G| ) r ∆ + (cid:15) ) , (ii) if the dynamics are deterministic,
[Equation]
where 1 is the indicator function, r g ( s, a ) :
=
¯ r
( s, g, a ) , r ∆ := r MAX − r MIN , and (cid:107) f − h (cid:107) ∞ :
= max s,g,a | f ( s, g, a ) − h ( s, g, a ) | .
Proof.
(i): We ﬁrst note that each g in G can be thought of as deﬁning an MDP M g :=
( S , A , p, r g , γ ) with reward function r g ( s, a ) :
=
¯ r
( s, g, a ) , optimal policy π ∗ g ( s ) =
¯ π ∗ ( s, g ) and optimal Q-value function Q π ∗ g ( s, a ) =
¯ Q ∗ ( s, g, a ) .
Then this proof follows similarly to that of Barreto et al.
Theorem 2,
[Equation]
Theorem
[Equation]
( Since rewards only differ in G where r ( s, a ) , r
[Equation] [Equation]
MIN ,
[Equation]
MAX
[Equation]
∈ G )
[Equation]
Hence, (cid:107)
[Equation]
( Since min g 1 r (cid:54) =
r g = 0 only when r ∈ { r g } |G| ) .
[Equation]
A. C OMPARING THE BOUNDS OF T HEOREM 1 WITH THAT OF GPI IN B ARRETO ET AL . (2018)
We ﬁrst restate Proposition 1 (Barreto et al., 2018) here.
Proposition 3 ((Barreto et al., 2018)) .
Let M ∈ M and let Q π ∗
j
i be the action value function of an optimal policy of M j ∈ M when executed in M i ∈ M .
Given approximations { ˜ Q π 1 i , ..., ˜ Q π n i } such that | Q π j i − ˜ Q
π j i | ≤ (cid:15) for all s, a ∈ S × A , and j ∈ { 1 , ..., n } , let
[Equation]
∈ arg
[Equation]
i
[Equation]
s
[Equation]
a ) .
then,
[Equation]
where Q ∗ is the optimal value function of M , Q π
We can simplify the bound in Proposition 3 as follows:
[Equation]
( Since rewards only differ in G where r ( s, a )
[Equation]
r
[Equation]
s, a
[Equation]
∈ { r MIN
[Equation]
r MAX } for s ∈ G )
[Equation]
( Since min j 1 r i (cid:54)
[Equation]
=
[Equation]
only when r i ∈ { r j } n )
[Equation]
where 1 is the indicator function, and r ∆ :=
r MAX − r MIN .
We can see that this bound is similar to that of Theorem 1(i) but weaker.
This because:
(i)
The ﬁrst term of this bound ( 1 r (cid:54) =
r i ) requires that reward function of the current task ( r ) be identical to that of a reference task ( r i ).
In Barreto et al.
(2018), r i is taken as the best linear approximation of r .
In contrast, the ﬁrst term of Theorem 1(i) ( 1 T (cid:54) =
T F ) only requires the current task to be expressible as a Boolean composition of past tasks.
(ii)
The second term of this bound ( 1 r i /
∈{ r j } n ) requires that the reference task (the best linear approximation to the current task) is exactly one of the past tasks.
In contrast, the second term of Theorem 1(i) ( 1 r/ ∈{ r g } |G| ) only requires the current task to have a single desirable goal.
This suggests that we can can think of the logical composition approach as an efﬁcient way of doing
GPI, one which leads to tight performance bounds on the transferred policy (Theorem 1(ii)).
A. P ROOFS FOR T HEOREM
Theorem 2.
Let D be an unknown distribution, possibly non-stationary, over a set of tasks
M ( S , A , p, γ, r 0 ) .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Let
[Equation]
M t ∼ D ( t ) and ˜ T 0 = ˜¯ Q
[Equation]
= ∅ ∀ t ∈ N .
Then,
[Equation]
where N t := | ˜
T t | = | ˜ ¯ Q ∗ t | .
Proof.
Let ˜ T t be the approximate binary representation of task M t learned by SOPGOL.
We ﬁrst note that SOPGOL returns ˜ T t ∪ { ˜ T t } only if ˜ T t is not in the span of ˜ T t .
That is, ˜
T t +1 =
˜ T t ∪ { ˜ T t } iff ˜ T t (cid:54) =
B EXP ( ˜ T t ) where B EXP = SOP ( ˜ T t , ˜ T t ) .
Hence, it is sufﬁcient to show that the number, N , of linearly independent binary vectors, ˜ T ∈ { 0 , 1 } |G| , that span the Boolean vector space (Subrahmanyam, 1964), GF (2) |G| , 5 is bounded by
[Equation]
|
[Equation]
|
[Equation]
N ≤ |G| .
This follows from the fact that (cid:100) log |G|(cid:101) is the size of a minimal set of generators for GF (2) |G| (as can easily be seen with a Boolean table), and |G| is its dimensionality.
B S UM O F P RODUCTS WITH G OAL O RIENTED L EARNING
Algorithm 1 shows the full pseudo-code for SOPGOL.
Here, SOP ( ˜ T , ˜ T ) is the classical sum of products method in Boolean logic.
Given a list of binary vectors T =
[ ˜ T 1 , ..., ˜ T n ] , a Boolean expression for a new binary vector ˜ T is obtained as follows:
Identify all rows of ˜ T with a 1. 2.
For each such row: Make a product (conjunction) of all the input variables and make the negation of each variable with a 0 in this row.
Take the sum (disjunction) of all these product terms.
The output of SOP is a function B EXP that takes in | ˜ T | variables, and applies disjunctions, conjunctions and negations to them according to the Boolean expression obtained above.
Algorithm 1: SOPGOL
Input : off-policy RL algorithm A , / * e.g DQN * / task MDP M , set of (cid:15) -optimal task binary representations ˜ T , set of (cid:15) -optimal
¯ Q -value functions ˜¯ Q .
Initialise ˜ T : G → { 0 , 1 }
Initialise ˜¯ Q : S × G ×
A → R according to A
Initialise goal buffer ˜
G with terminal states observed from a random policy while ˜¯ Q is not converged do
Initialise state s from M
B EXP ← SOP ( ˜ T , ˜ T )
T SOP , ¯ Q SOP ← B EXP ( ˜ T ) , B EXP ( ˜¯ Q ∗ ) ¯ Q ← ¯ Q SOP if ˜ T
=
T SOP else ˜¯ Q ∨ ¯ Q SOP g ← arg max g (cid:
) ∈ ˜ G (cid:18) max a ∈A ¯ Q ( s, g (cid:48) , a ) (cid:19) while s is not terminal do
Select action a using the behaviour policy from A : a ← ¯ π ( s, g ) /* e.g (cid:15) -greedy */
Take action a , observe reward r and next state s (cid:48) in M if ˜ T (cid:54) =
T SOP then foreach g (
cid:48) ∈
˜ G do ¯ r ← r MIN if g (cid:48) (cid:54) = s ∈ ˜
G else r
Update ˜¯ Q with ( s, g (cid:48) , a, ¯ r, s (cid:48) )
according to A end if s is terminal then ˜
T ( s ) ← 1 r = r MAX ˜
G ← ˜
G ∪ { s } else s ← s (cid:48) end end end
B EXP ← SOP ( ˜ T , ˜ T ) ˜
T , ˜¯ Q ← ( ˜ T , ˜¯ Q ) if ˜ T = B EXP ( ˜ T ) else ( ˜ T ∪ { ˜ T } , ˜¯ Q ∪ { ˜¯ Q } ) return ˜ T , ˜¯ Q
C F OUR R OOMS E NVIRONMENT
We use the Four Rooms domain (Sutton et al., 1999), where an agent must navigate in a grid world to particular locations.
The goal locations are placed along the sides of the walls and at the centre of rooms (Figure 5).
This gives a goal space of size |G| = 40 and a task space of size |M| = 2 |G| ≈ 10 12 .
The agent can move in any of the four cardinal directions at each timestep, but colliding with a wall leaves the agent in the same location.
We add a 5th action for “stay” that the agent chooses to achieve goals.
A goal location only becomes terminal if the agent chooses to stay in it.
All rewards are 0 at non-terminal states, and 1 at the desirable goals.
The transition dynamics are stochastic with a slip probability ( sp = 0 . 1 ).
That is, with probability 1-sp the agent moves in the direction it chooses, and with probability sp it moves in one of the other three chosen uniformly at random.
D F UNCTION A PPROXIMATION E XPERIMENT D ETAILS D. E NVIRONMENT
The P ICK U P O BJ environment is fully observable, where each state observation is a 56 ∗ 56 ∗ 3 RGB image (Figure 1).
The agent has 7 actions it can take in this environment corresponding to: 1 - rotate left, 2 - rotate right, 3 - move one step forward if there is no wall or object in front, 4 - pickup object if there is an object in front and no object has been picked, 5 - drop the object in front if an object has been picked and there is no wall or object in front, 6 - open the door in front if there is a closed-door in front, and 7 - close the door in front if there is an opened door in front.
For each task, each episode starts with 1 desirable object and 4 other randomly chosen objects placed randomly in the environment.
The agent is also placed at a random position with a random orientation at the start of each episode.
The agent receives a reward of -0.1 at every timestep, and a reward of 2 when it picks up a desirable object.
The environment transitions to a terminal state once the agent picks up any object and the agent observes the picked object.
There are 15 types of objects (illustrated in Table 1) resulting in 15 possible goal states.
Hence, the dimension of the state space is |S| = 56 × 56 × 3 , the goal space is |G| = 15 , and the action space is |A| = 7 .
D. N ETWORK A RCHITECTURE AND H YPERPARAMETERS
In our function approximation experiments, we represent each extended value function ˜¯ Q ∗ with a list of |G| DQNs, such that the value function for each goal ˜
Q ∗ g ( s, a ) := ˜¯ Q ∗ ( s, g, a ) is approximated with a separate DQN.
The DQNs used have the following architecture, with the CNN part being identical to that used by Mnih et al.
Three convolutional layers: (a) Layer 1 has 3 input channels, 32 output channels, a kernel size of 8 and a stride of 4.
(b) Layer 2 has 32 input channels, 64 output channels, a kernel size of 4 and a stride of 2.
(c) Layer 3 has 64 input channels, 64 output channels, a kernel size of 3 and a stride of 1. 2.
Two fully-connected linear layers: (a) Layer 1 has input size 3136 and output size 512 and uses a ReLU activation function.
(b) Layer 2 has input size 512 and output size 7 with no activation function.
We used the ADAM optimiser with batch size 256 and a learning rate of 10 − 3 .
We started training after 1000 steps of random exploration and updated the target Q-network every 1000 steps.
Finally, we used (cid:15) -greedy exploration, annealing (cid:15) from 0 .
5 to 0 .
05 over 100000 timesteps.
Finally, we used the same DQN architecture and training hyperparameters for the baseline in all experiments.