G ENERALISATION IN L IFELONG R EINFORCEMENT L EARNING THROUGH L OGICAL C OMPOSITION
A BSTRACT
We leverage logical composition in reinforcement learning to create a framework that enables an agent to autonomously determine whether a new task can be immediately solved using its existing abilities, or whether a task-speciﬁc skill should be learned.
In the latter case, the proposed algorithm also enables the agent to learn the new task faster by generating an estimate of the optimal policy.
Importantly, we provide two main theoretical results: we give bounds on the performance of the transferred policy on a new task, and we give bounds on the necessary and sufﬁcient number of tasks that need to be learned throughout an agent’s lifetime to generalise over a distribution.
We verify our approach in a series of experiments, where we perform transfer learning both after learning a set of base tasks, and after learning an arbitrary set of tasks.
We also demonstrate that as a side effect of our transfer learning approach, an agent can produce an interpretable Boolean expression of its understanding of the current task.
Finally, we demonstrate our approach in the full lifelong setting where an agent receives tasks from an unknown distribution and, starting from zero skills, is able to quickly generalise over the task distribution after learning only a few tasks—which are sub-logarithmic in the size of the task space.
I NTRODUCTION
Reinforcement learning (RL) is a framework in artiﬁcial intelligence that enables agents to learn desired behaviours by maximising the rewards received through interaction with an environment (Sutton et al., 1998).
While it has achieved recent success in a number of difﬁcult, high-dimensional domains (Mnih et al., 2015; Levine et al., 2016; Lillicrap et al., 2016; Silver et al., 2017), these methods require millions of samples from the environment to learn optimal behaviours.
This is ultimately a fatal ﬂaw, since learning to solve complex, real-world tasks from scratch for every task of interest is typically infeasible.
Hence a major challenge in RL is building general-purpose agents that are able to use existing knowledge to solve new tasks quickly.
That is, after learning n tasks sampled from some distribution, the question of interest is: How can an agent leverage the skills learned from those n tasks to improve its starting performance or learning speed in task n + 1 ?
This is the problem setting formalised by lifelong RL (Thrun, 1996; Abel et al., 2018).
One approach to transfer in lifelong RL is composition (Todorov, 2009), which allows an agent to leverage its existing skills to build complex, novel behaviours, which can then be used to solve or speed up learning of a new task (Todorov, 2009; Saxe et al., 2017; Haarnoja et al., 2018;
Van Niekerk et al., 2019; Hunt et al., 2019; Peng et al., 2019).
Recently, Nangue Tasse et al.
(2020) proposed a framework for deﬁning a Boolean algebra over the space of tasks and optimal value functions.
This allowed for tasks and value functions to be composed using the union, intersection and negation operators in a principled manner to yield optimal skills zero-shot.
In this work, we propose a framework for lifelong RL that focuses not only on transfer between tasks for faster RL, but also gives guarantees on the generalisation of an agent’s skills over an unknown task distribution.
We ﬁrst extend the logical composition framework of Nangue Tasse et al.
(2020) to discounted and stochastic tasks.
This enables us to provide theoretical bounds for our approach in stochastic settings, and also enables us to easily compare our bounds to previous works in the discounted setting.
We then show how our framework leverages logical composition to tackle the lifelong RL problem.
The framework enables agents to iteratively solve tasks as they are given, while at the same time constructing a library of skills which can be composed to obtain behaviours for solving future tasks faster or even without further learning.
We empirically verify our framework in a series of experiments, where an agent is i) pretrained on a set of base tasks provided by the Boolean algebra framework, and ii) when the pretrained tasks are not base tasks.
We show that agents here are able to achieve good performance on new tasks before training even starts.
Finally, we demonstrate our framework in the lifelong RL setting where an agent receives tasks from an unknown non-stationary distribution and must determine what skills to learn and add to its library, and how to combine its current skills to solve new tasks.
Results demonstrate that this framework enables agents to quickly learn a set of skills, resulting in a combinatorial explosion in their abilities.
Consequently, even when tasks are sampled randomly from an unknown distribution, an agent is able leverage its existing skills to solve new tasks without further learning, thereby generalising over task distributions.
B ACKGROUND
We consider tasks modelled by Markov Decision Processes (MDPs).
An MDP is deﬁned by the tuple ( S , A , p, r, γ ) , where (i) S is the state space, (ii) A is the action space, (iii) p ( s (cid:48) | s, a ) is a
Markov transition probability, (iv) r is the real-valued reward function bounded by [ r MIN , r MAX ] , and (v) γ ∈ [0 , 1) is the discount factor.
In this work, we focus on goal-based tasks where an agent is required to reach a subset of desirable goals in a ﬁnite goal space G ⊆ S (boundary set of states).
Here, termination in G is modelled similarly to Van Niekerk et al.
by augmenting the state space with a virtual state, ω , such that p ( ω | s, a ) = 1 ∀ ( s, a ) ∈ ( G × A ) and the rewards are zero after reaching ω .
We hence consider the set of tasks
M such that the tasks are in the same environment—described by a background MDP ( S , A , p, γ, r 0 ) —and each task can be uniquely speciﬁed by a set of desirable and undesirable goals:
[Equation]
The goal of the agent is to compute a Markov policy π from S to A that optimally solves a given task.
A given policy π is characterised by a value function V π ( s ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t )] , specifying the expected return obtained under π starting from state s .
The optimal policy π ∗ is the policy that obtains the greatest expected return at each state:
V π ∗ ( s ) =
V ∗ ( s ) = max π V π ( s ) for all s in S .
A related quantity is the Q -value function, Q π ( s, a ) , which deﬁnes the expected return obtained by executing a from s , and thereafter following π .
Similarly, the optimal Q -value function is given by
Q ∗ ( s, a ) = max π Q π ( s, a ) for all s in S and a in A .
2. L OGICAL COMPOSITION
Nangue Tasse et al.
(2020) recently proposed the notion of a Boolean task algebra, which allows an agent to perform logical operations—conjunction ( ∧ ), disjunction ( ∨ ) and negation ( ¬ )—over the space of tasks and value functions.
1 To achieve this, they extend the standard deﬁnitions of the reward and value functions to deﬁne goal-oriented versions as follows:
Deﬁnition 1.
The extended reward function ¯ r : S × G ×
A → R is given by the mapping ( s, g, a ) (cid:55)→ (cid:26)
[Equation]
where ¯ r MIN ≤ min { r MIN , ( r MIN − r MAX ) D } , and D is the diameter of the MDP (Jaksch et al., 2010).
Deﬁnition 2.
The extended Q-value function ¯ Q : S × G ×
A → R is given by the mapping
[Equation]
where ¯ V ¯ π ( s, g )
=
E ¯ π [ (cid:80) ∞ t =0 ¯ r
( s t , g, a t )] .
By penalising the agent for achieving goals different from the one it wanted to reach— ¯ r MIN & if g (cid:54) = s ∈ G —, the extended reward function has the effect of driving the agent to learn how to separately achieve all desirable goals.
The agent can then act by simply maximising over ¯ Q : π
[Equation]
arg
[Equation]
The logic operators over tasks and extended action-value functions are then deﬁned as follows:
Deﬁnition 3.
Let M be a set of tasks with bounds M MIN ,
M MAX ∈ M such that,
[Equation]
Deﬁne the ¬ , ∨ , and ∧ operators over M as ¬ ( M ) :
= ( S , A , p, r ¬ M ) , where r ¬ M ( s, a ) :=
( r M MAX ( s, a ) + r M MIN ( s, a )) − r M ( s, a ) ∨ ( M 1 , M 2 ) :
= ( S , A , p, r M 1 ∨ M 2 ) , where r M 1 ∨ M 2 ( s, a ) :
= max { r M 1 ( s, a ) , r M 2 ( s, a ) } ∧ ( M 1 , M 2 ) :
= ( S , A , p, r M 1 ∧ M 2 ) , where r M 1 ∧ M 2 ( s, a ) :
=
min { r M 1 ( s, a ) , r M 2 ( s, a ) }
Deﬁnition 4.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M , with bounds ¯ Q ∗ MIN , ¯ Q ∗ MAX ∈ ¯ Q ∗ which are respectively the optimal ¯ Q -functions for the tasks
M MIN , M MAX ∈ M .
Deﬁne the ¬ , ∨ , and ∧ operators over ¯ Q ∗ as, ¬
[Equation]
Using the deﬁnitions for the logical operations over M and ¯ Q ∗ given above, Nangue Tasse et al.
(2020) construct a Boolean algebra over tasks and extended value functions.
Furthermore, by leveraging the goal-oriented deﬁnition of extended value functions, they also show that M and ¯ Q ∗ are homomorphic.
Proposition 1.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Then,
(i) M and ¯ Q ∗ respectively form a Boolean task algebra ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) and a Boolean EVF algebra ( ¯ Q ∗ , ∨ , ∧ , ¬ , ¯ Q ∗ MAX , ¯ Q ∗ MIN ) , (ii) A is a homomorphism between M and ¯ Q ∗ .
Proposition 1 allows one to compose existing tasks and skills together to create new ones in a principled way.
Furthermore, it guarantees that if we can write down a task under the Boolean algebra, we can immediately write down the optimal value function for the task.
This enables agents to solve any new task that is expressed as the logical combination of learned ones.
L IFELONG TRANSFER THROUGH COMPOSITION 3. E XTENDING THE LIFELONG REINFORCEMENT LEARNING PROBLEM
In lifelong RL, an agent is presented with a series of tasks sampled from some distribution D .
The agent then needs to not only transfer knowledge learned from previous tasks to solve new but related tasks quickly, but it also should not forget said knowledge in the process.
We formalise this lifelong learning problem as follows:
Deﬁnition 5.
Let D be an unknown non-stationary distribution over a set of tasks M ( S , A , p, γ, r 0 ) .
The lifelong learning problem consists of the repetition of the following steps for t ∈ N :
The agent samples a task M t ∼ D ( t ) , 2.
The agent interacts with the MDP M t until it is (cid:15) -optimal in M 0 , ..., M t .
This formulation of lifelong RL is similar to that of Abel et al.
(2018), with the main difference being that we do not assume that D is a stationary distribution,
and we explicitly require an agent to retain learned skills.
As discussed in the introduction, one of the main goals in this setting is that of transfer (Taylor &
Stone, 2009).
We add an important question to this setting: how many tasks should an agent learn during its lifetime in order to generalise over the task distribution—that is to be able to solve any new task without further learning?
While most approaches focus on the goal of transfer, the question of the number of tasks is often neglected by simply assuming the case where the agent has already learned n tasks (Abel et al., 2018; Barreto et al., 2018).
Consider for example a task space with only |G| =goals.
Then, considering the composition of goals, the size of the task space is |M| = 2 |G| ≈ 10 12 .
If D is a uniform distribution over |M| , then for most transfer learning methods an agent will have to learn most of the tasks it is presented with, since the probability of getting the same task will be approximately zero.
This is clearly impractical for a setting like RL where learning methods often have a high sample complexity even with transfer learning.
It is also extremely memory inefﬁcient since the learned skills of most tasks will have to be stored.
In this section, we show how logical composition can be leveraged to learn a subset of tasks that is sufﬁcient to generalise over the task distribution.
We ﬁrst extend the logical composition framework to discounted tasks.
3. E XTENDING THE B OOLEAN ALGEBRA FRAMEWORK
We extend Proposition 1 to the set of discounted stochastic tasks M (Equation 1).
To achieve this, we ﬁrst redeﬁne the extended reward function to use the simpler penalty ¯ r MIN = r MIN and include discounting in the action-value function:
Deﬁnition 6.
The extended reward function ¯ r : S × G ×
A → R is given by the mapping
[Equation]
Deﬁnition 7.
The extended Q-value function ¯ Q : S × G ×
A → R is given by the mapping
[Equation]
where ¯ V ¯ π ( s, g )
=
E ¯ π [ (cid:80) ∞ t =
γ t ¯ r ( s t , g, a t )] .
We now show that the Boolean algebra and zero-shot composition results of Nangue Tasse et al.
(2020) also hold for tasks in M .
We use the same deﬁnitions of ∨ and ∧ as in Deﬁnitions 3 and 4, but redeﬁne ¬ over ¯ Q ∗ as follows:
[Equation]
As we discuss in the supplementary material, this is equivalent to the previous deﬁnition for optimal ¯
Q -value functions of tasks in M , but it gives better bounds when composing (cid:15) -optimal ¯ Q -value functions.
Proposition 2.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Then,
(i) M and ¯ Q ∗ respectively form a Boolean task algebra ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) and a Boolean EVF algebra ( ¯
[Equation]
(ii) A is a homomorphism between M and ¯ Q ∗ .
We can now solve any new task in M zero-shot if we are given the correct Boolean expression that tells the agent how to compose its optimal skills.
This is essential for the following results.
3. T RANSFER BETWEEN TASKS
In this section, we leverage the logical composition results to address the following question of interest:
Given an arbitrary set of learned tasks, can we transfer their skills to solve new tasks faster?
As shown in Theorem 1, we answer this question in the afﬁrmative.
To achieve this, we ﬁrst realise that each task M ∈ M can be associated with a binary vector T ∈ { 0 , 1 } |G| which represents its set of desirable goals—as illustrated by the tasks in Table 1—such that
[Equation]
Now let ˜ T be an approximation of T for a given task M .
We can then use the sum of products method ( SOP ) to determine a candidate Boolean expression ( B EXP ) in terms of the learned binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } of a set of past tasks ˆ M = { M 1 , ..., M n } ⊆ M .
An estimate of the optimal ¯ Q -value function of M can then be obtained by composing the learned ¯ Q -value functions ˜¯
Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } according to B EXP .
Theorem 1 2 shows the optimality of this process.
Theorem 1.
Let M ∈ M be a task with binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T SOP =
B EXP ( ˜ T n ) and ¯ Q SOP =
B EXP ( ˜¯ Q ∗ n ) where B EXP = SOP ( ˜ T n , ˜ T ) .
Deﬁne,
[Equation]
Q SOP where Q SOP := max g
[Equation]
Q SOP ( s, g, a ) .
Then,
[Equation]
ii)
[Equation]
the dynamics [Equation] deterministic
[Equation]
where 1 is the indicator function, r g ( s, a ) :
=
¯ r
( s, g, a ) , r ∆ := r MAX − r MIN , and (cid:107) f − h (cid:107) ∞ :
= max s,g,a | f ( s, g, a ) − h ( s, g, a ) | .
Theorem 1(i) says that if ¯ Q SOP is close to optimal, then acting greedily with respect to it is also close to optimal.
Interestingly, this is similar to the bound obtained by Barreto et al.
(Proposition 1) for transfer learning using generalised policy improvement (GPI), but stronger.
This is unsurprising, since π ( s ) ∈ arg max a ∈A max g ∈G ¯ Q SOP ( s, g, a ) can be interpreted as generalised policy improvement on the set of goal policies of the extended value function ¯ Q SOP .
Importantly, if the environment is deterministic, then we obtain a strong bound on the composed value functions (Theorem 1(ii)).
This bound shows that transfer learning using the SOP method is (cid:15) -optimal—that is there is no loss in optimality—when the new task is expressible as a logical combination of past ones.
With the exponential nature of logical combination, this gives agents a strong generalisation ability over the task space—and hence over any task distribution—as shown in Theorem 2.
3. G ENERALISATION OVER A TASK DISTRIBUTION
We leverage Theorem 1 to design an algorithm that combines the SOP approach with goal-oriented learning to achieve fast transfer in lifelong RL.
Given an off-policy RL algorithm A , the agent initializes its extended value function ˜¯ Q , the task binary vector ˜ T , and a goal buffer according to A .
At the beginning of each episode, the agent
Goals
T a
T b
T c
T d               computes T SOP and Q SOP for ˜ T using the SOP method and its library of learned task vectors and extended Q-functions.
It then acts using the behaviour policy ( (cid:15) -greedy for example) of A with ¯ Q SOP for the action-value function if T SOP =
˜ T , and ¯ Q SOP ∨ ˜¯ Q otherwise.
4 If T SOP (cid:54) =
˜ T , the agent also updates ˜ ¯ Q for each goal in the goal buffer using A .
Finally, when the agent reaches a terminal state s , it adds it to the goal buffer and updates ˜ T ( s ) using the reward it receives (as per
Equation 6).
The full algorithm is included in Section 2 of the supplementary material.
We refer to this algorithm as SOPGOL ( Sum Of Products with Goal Oriented Learning ).
We now show that
SOPGOL generalises over any unknown non-stationary task distribution after learning only a number of tasks logarithmic in the size of the task space.
Theorem 2.
Let D be an unknown non-stationary distribution over a set of tasks M ( S , A , p, γ, r 0 ) , and let A :
[Equation]
∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Let ˜
T t +1 , ˜¯ Q ∗ t +1 =
SOPGOL ( A , M t , ˜ T t , ˜¯ Q ∗ t ) where M t ∼ D ( t ) and ˜ T
= ˜¯ Q ∗ 0 = ∅ ∀ t ∈ N .
Then,
[Equation]
Interestingly, Theorem 2 holds even in the case where a new task is expressible in terms of past tasks ( T SOP =
˜ T )
but we want to learn to solve it to a higher degree of optimality than past tasks.
In this case, we can pretend T SOP (cid:54) =
˜ T and learn a new ¯ Q -function to the desired degree of optimality.
We can then add it to our library, and remove any other skill from our library (the least optimal for example).
Notice that |G| is the maximum number of different skills we need to store because that is the dimensionality of the goal space.
E XPERIMENTS 4. T RANSFER AFTER PRETRAINING ON A SET OF TASKS
We consider the PickUpObj domain from the minigrid environment (Chevalier-Boisvert et al., 2018), where an agent must navigate in a 2D room to pickup objects of various shapes and colours from pixel observations (Figure 1).
There are |G| = 15 goals, each corresponding to picking up objects of 3 possible types—box, ball, key—and 5 possible colours—red, blue, green, purple, and yellow—.
Hence a set of (cid:100) log 2 |G|(cid:101)
= 4 base tasks can be selected which can be used to solve all 2 |G| = 32768 possible tasks under a Boolean composition of goals.
The agent receives a reward ofwhen it picks-up desired objects, and − 0 .
1 otherwise.
For all of our experiments in this section, we use deep Q-learning (Mnih et al., 2015) as the RL learning method for SOPGOL and as the performance baseline.
We also compare SOPGOL to or to SOPGOL-
SOPGOL to when no new skill is learned
[Figure] [Figure]
[Figure] [Figure]
[Equation]
and SOPGOL-continual refers to when a new skill is learned using the SOP Q estimate to speed up learning.
Since SOPGOL determines automatically which one to use, we compare whichever one it chooses with the other one in each of our experiments.
We ﬁrst demonstrate transfer learning after pretraining on a set of base tasks—a minimal set of tasks that span the task space—.
This can be done if the set of goals is known upfront, by ﬁrst assigning a Boolean label to each goal in a table and then using the rows of the table as base tasks.
These are illustrated in Table 1.
Having learned the (cid:15) -optimal extended value functions for our base tasks, we can now leverage logical composition to do transfer learning on test tasks.
We consider thetest tasks shown in Table 2.
For each, we run SOPGOL, SOPGOL-continual, and a standard DQN.
Figure 2 shows the results.
As expected from our theoretical results in Section 3.3, SOPGOL correctly determines that the current test tasks are solvable from the logical combinations of the learned base tasks.
Its performance from the start of training is hence the best.
Now that we have demonstrated how SOPGOL enables an agent to solve any new task in an environment after training on base tasks, we consider the more practical case where the new tasks are not fully expressible as a Boolean expression of previously learned tasks.
The agent in this case is pretrained on a set of tasks that do not span the task space, { , , , } , corresponding to the tasks for picking up green objects, blue objects, yellow objects, and keys.
We then train the agent with SOPGOL, SOPGOL-transfer, and a standard DQN on the same set of test tasks considered previously (Table 2).
Figure 3 shows our results.
We observe that SOPGOL now chooses to learn a task speciﬁc skill after transfer, and hence performs better than SOPGOL-transfer since the test tasks are not entirely expressible in terms of the pretrained ones.
Consider Figure 3a for example.
The test task is to pick up a yellow box, but the agent has only learned how to pick up red objects, blue objects, yellow objects, and keys.
It has not learned how to pick up boxes.
But we can see in the
Boolean expression it infers ( (cid:102) M 1 ) that it correctly identiﬁes that the desired objects are at least yellow.
Without further improving this transferred policy (SOPGOL-transfer), we can see that this gives the
[Figure]
The Boolean expressions generated by SOPGOL for the respective test tasks are:
[Equation]
agent better performance than just DQN from the get go.
This is because of two main factors: (i)
The transferred policy navigates to objects more reliably, so takes less random actions.
(ii)
Although the transferred policy does not have a complete understanding of which are the desirable objects, it at least navigates to yellow objects, which some of the times are yellow boxes.
Finally, since SOPGOL is able to determine that the current task is not entirely expressible in terms of its previous tasks (by checking if T SOP =
˜ T ), it is able to learn a new ¯ Q -value function that improves on the transferred policy.
Additionally, its returns are strictly higher than those of SOPGOL-transfer because SOPGOL learns the new ¯ Q -value function faster by using ¯ Q SOP ∨ ˜¯ Q in the behavior policy.
4. L IFELONG TRANSFER
In this section we consider the more general setting where the agent is not necessarily given pretrained skills upfront, but is rather presented with tasks sampled from some unknown non-stationary distribution.
Let us revisit the hypothetical example given in Section 3.1, but now more concretely by using a Four Rooms domain (Sutton et al., 1999) where an agent must navigate in a grid world to particular locations.
The goal locations are placed along the sides of the walls and at the center of rooms.
This gives a goal space of size |G| = 40 and a task space of size |M| = 2 |G| ≈ 10 12 .
The agent can move in any of the four cardinal directions at each timestep, but colliding with a wall leaves the agent in the same location.
We add a 5th action for “stay” that the agent chooses to achieve goals.
A goal location only becomes terminal if the agent chooses to stay in it.
All rewards are 0 at non-terminal states, and 1 at the desirable goals.
The transition dynamics are stochastic with a slip probability ( sp ).
That is, with probability 1-sp the agent moves in the direction it chooses, and with probability sp it moves in one of the other 3 chosen uniformly at random.
We demonstrate the ability of SOPGOL to generalise over task distributions by evaluating the approach with the following distributions: (i) D sampled : The goals for each task are chosen uniformly at random over G .
(ii) D best : The ﬁrst (cid:100) log 2 |G|(cid:101) tasks are the base tasks.
The rest follow D sampled .
This distribution gives the agent the minimum number of tasks to learn and store since the agent learns the base tasks ﬁrst before being presented with any other task.
(iii) D worst : The ﬁrst |G| tasks are each deﬁned by a single goal that differs from the previous tasks.
The rest follow
D sampled .
This distribution forces the agent to learn and store the maximum number of tasks, since none of the |G| tasks can be expressed as a logical combination of the others.
We use Q-learning (Watkins, 1989) as the RL method for SOPGOL, and Q-learning with max Q initialisation as a baseline.
This has been shown by previous works (Abel et al., 2018) to be a practical method of initialising value functions with a theoretically optimal optimism criterion that speeds-up convergence during training.
Our results (Figure 4) show that SOPGOL enables a lifelong agent to quickly generalise over an unknown task distribution.
Interestingly, both graphs show that the convergence speed during a randomly sampled task distribution D Sampled is very close to that of the best task distribution D Sampled .
This suggests that there is room to make the bound in Theorem 2 even tighter by making some assumptions on the task distribution—an interesting avenue for future work.
[Figure] [Figure]
R ELATED WORK
There have been several approaches in recent years for tackling the problem of transfer in lifelong
RL.
Most closely related to this work is the line of works on concurrent skill composition (Todorov, 2009;
Saxe et al., 2017; Haarnoja et al., 2018; Van Niekerk et al., 2019; Hunt et al., 2019; Peng et al., 2019).
These methods usually focus on multi-goal tasks, where they address the combinatorial amount of desirable goals by composing learned skills to create new ones.
Given that the reward function of each new task is well approximated by a linear function, Barreto et al.
(2020) proposed a scheme for few-shot transfer in RL by combining GPI and successor features (Barreto et al., 2017).
Recently Nangue Tasse et al.
(2020) proposed a framework to achieve zero-shot transfer (with no few-shot transfer) given that the Boolean expression for each new task is provided.
In contrast to these methods, our method is also able to cope with the case where a new task cannot be written as a combination of learned ones.
Other approaches like options (Sutton et al., 1999) and hierarchical RL (Barto & Mahadevan, 2003) address the lifelong RL problem via temporal compositions.
These methods are usually focused on single-goal tasks, where they address the potentially long trajectories needed to reach a desired goal by composing sub-goal skills sequentially (Levy et al., 2017; Bagaria & Konidaris, 2019).
While they do not consider the multi-goal setting, they can be used in conjunction with concurrent composition to learn how to achieve a combinatorial amount of desirable long horizon goals.
Finally, there is also non-compositional approaches like (Finn et al., 2017;
Abel et al., 2018; Singh et al., 2020).
These methods usually aim to learn the policy for a new task faster by initializing the networks with some pre-training procedure.
These can be used in combination with SOPGOL to learn new skills faster.
C ONCLUSION
In this work, we proposed an approach for efﬁcient transfer learning in RL.
Our framework, SOPGOL, leverages the Boolean algebra framework of Nangue Tasse et al.
(2020) to determine which skills should be reused in a new task.
We demonstrated that, if a new task is solvable using existing skills, an agent is able to solve it with no further learning.
However, even if this is not the case, an estimate of the optimal value function can still be obtained to speed up training.
This allows agents in a lifelong learning setting to quickly generalise over any unknown non-stationary task distribution.
Finally, we note that this framework can be complemented with MAXQINIT (Abel et al., 2018) or other few-shot methods to speed up the learning of new skills (over and above the transfer learning and task space generalisation shown here).
Our approach is a step towards the goal of truly general, long-lived agents, which are able to generalise both within tasks, as well as over the distribution of possible tasks it may encounter.
A P ROOFS OF THEORETICAL RESULTS A. B OOLEAN A LGEBRA D EFINITION
Deﬁnition 8.
A Boolean algebra is a set B equipped with the binary operators ∨ (disjunction) and ∧ (conjunction), and the unary operator ¬ (negation), which satisﬁes the following Boolean algebra axioms for a, b, c in B :
(i) Idempotence: a ∧ a = a ∨ a = a .
(ii) Commutativity: a
∧ b = b ∧ a and a ∨ b =
b ∨ a .
(iii) Associativity: a ∧ ( b ∧ c ) =
( a ∧ b ) ∧ c and a ∧ ( b ∨ c )
= ( a ∨ b ) ∨ c .
(iv) Absorption: a ∧ ( a ∨ b ) =
a ∨ ( a ∧ b ) = a .
(v) Distributivity: a ∧ ( b ∨ c )
= ( a ∧ b ) ∨ ( a ∧ c ) and a ∨ ( b ∧ c ) =
( a ∨ b ) ∧ ( a ∨ c ) .
(vi) Identity: there exists 0 , 1 in B such that
[Equation]
(vii) Complements: for every a in B , there exists an element a (cid:48) in B such that a ∧ a (cid:48)
= 0 and a ∨ a (cid:48)
= 1 .
A. P ROOFS FOR P ROPOSITION
Lemma 1.
Let M be a set of tasks.
Then ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) is a Boolean algebra.
Proof.
Let M 1 , M 2 ∈ M .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i)–(v):
These easily follow from the fact that the min and max functions satisfy the idempotent, commutative, associative, absorption and distributive laws.
(vi): Let r M MAX ∧ M 1 and r M 1 be the reward functions for M MAX ∧ M 1 and M 1 respectively.
Then for all ( s, a ) in
S × A , r M MAX ∧ M 1 ( s, a ) = (cid:26) min { r MAX , r M 1 ( s, a ) } , if s ∈ G min { r 0 ( s, a ) , r 0 ( s, a ) } , otherwise.
= (cid:26) r M 1 ( s, a ) , if s ∈ G r 0 ( s, a ) , otherwise.
( r M 1 ( s, a ) ∈ { r MIN , r MAX } for s ∈ G ) =
r M 1 ( s, a ) .
Thus M MAX ∧ M 1 = M 1 .
Similarly M MAX ∨ M 1 = M MAX , M MIN ∧ M 1 = M MIN , and
M MIN ∨ M 1 = M 1 .
Hence M MIN and M MAX are the universal bounds of M .
(vii): Let r M 1 ∧¬ M 1 be the reward function for M 1 ∧ ¬ M 1 .
Then for all ( s, a ) in S × A , r M 1 ∧¬ M 1 ( s, a ) = (cid:26) min { r M 1 ( s, a ) , ( r MAX + r MIN ) − r M 1 ( s, a ) } , if s ∈ G min { r 0 ( s, a ) , ( r 0 ( s, a ) + r 0 ( s, a )) − r 0 ( s, a ) } , otherwise.
=   r MIN , if s ∈ G and r M 1 ( s, a ) =
r MAX r MAX , if s ∈ G and r M 1 ( s, a ) =
r MIN r 0 ( s, a ) , otherwise.
= r M MIN ( s, a ) .
Thus M 1 ∧ ¬ M 1 = M MIN , and similarly M 1 ∨ ¬ M 1 = M MAX .
Lemma 2.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Then ( ¯ Q ∗ , ∨ , ∧ , ¬ , ¯ Q ∗ MAX , ¯ Q ∗ MIN ) is a Boolean Algebra.
Proof.
Let ¯ Q ∗ M 1 , ¯ Q ∗ M 2 ∈ ¯ Q ∗ be the optimal ¯ Q -value functions for tasks M 1 , M 2 ∈ M with reward functions r M 1 and r M 2 .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i) v
): These follow directly from the properties
of
the min and max functions.
(vi): For all ( s, g, a ) in S × G × A ,
[Equation]
[Equation]
(vii): For all ( . ) in S × G × A ,
[Equation]
min { ¯ Q ∗ M 1 ( . ) , ¬ ¯ Q ∗ M 1 ( . ) }
= (cid:26) min { ¯ Q ∗ MIN ( . ) , ¯ Q ∗ MAX
[Equation]
} if | ¯ Q ∗ ( . )
− ¯ Q
[Equation]
MIN ( .
[Equation]
| ≤ | ¯ Q ∗ ( .
[Equation]
− ¯ Q
[Equation]
MAX ( . )
| min { ¯ Q ∗ MAX ( . ) , ¯ Q ∗ MIN ( . ) } otherwise , = ¯ Q ∗ MIN ( . ) .
Similarly, ¯ Q ∗ M 1 ∨ ¬ ¯ Q ∗ M 1 = ¯ Q ∗ MAX .
Lemma 3.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M .
Then for all M 1 , M 2 ∈ M , we have (i) ¯ Q ∗¬ M 1 = ¬ ¯ Q ∗ M 1 , (ii) ¯ Q ∗ M 1 ∨ M 2 = ¯ Q ∗ M 1 ∨ ¯ Q ∗ M 2 , and (iii) ¯ Q ∗ M 1 ∧ M 2 = ¯ Q ∗ M 1 ∧ ¯ Q ∗ M 2 .
Proof.
Let M 1 , M 2 ∈ M .
Then for all ( s, g, a ) in S × G × A , (i): ¯ Q ∗¬ M 1 ( s, g, a ) = (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if r ¬ M 1 ( g, a (cid:48) )
=
r MAX ∀ a (cid:48)
∈
A ¯
Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if r M 1 ( g, a (cid:48) ) =
r MIN ∀ a (cid:48) ∈
A ¯
Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if ¯ Q ∗ M 1 ( s, g, a ) =
¯ Q ∗ MIN ( s, g, a ) ¯ Q ∗ MIN ( s, g, a ) , otherwise.
= (cid:26) ¯ Q ∗ MAX ( s, g, a ) , if | ¯ Q ∗ M 1 ( s, g, a ) − ¯ Q
[Equation] [Equation]
(iii): Follows similarly to
(ii).
Proposition 3.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Let A : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Then,
(i) M and ¯ Q ∗ respectively form a Boolean task algebra ( M , ∨ , ∧ , ¬ , M MAX , M MIN ) and
a Boolean
EVF algebra ( ¯ Q ∗ , ∨ , ∧ , ¬ , ¯ Q ∗ MAX , ¯ Q ∗ MIN ) , (ii)
A is a homomorphism between M and ¯ Q ∗ .
Proof.
(i): Follows from Lemma 1 and 2.
(ii): Follows from Lemma 3.
A. P ROOFS FOR T HEOREM
Lemma 4.
Let ¯ Q ∗ be the set of optimal ¯ Q -value functions for tasks in M .
Denote ˜¯ Q ∗ M as the (cid:15) -optimal
¯ Q -value function for a task M ∈ M such that
[Equation]
Then for all M 1 , M 2 in M and ( s, g, a ) in S × G × A ,
[Equation]
ii
[Equation]
(iii)
[Equation]
¯
[Equation]
a
[Equation] [Equation]
Proof.
(i):
[Equation]
[Table]
[Equation]
Lemma 5.
Let M ∈ M be a task with binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T SOP =
B EXP ( ˜ T n ) and ¯ Q SOP =
B EXP ( ˜¯ Q ∗ n ) where B EXP = SOP ( ˜ T n , ˜ T ) .
Deﬁne,
[Equation]
where
[Equation]
Then, (cid:107)
[Equation]
where 1 is the indicator function, r ∆ :=
r MAX − r MIN , and (cid:107) f − h (cid:107) ∞ := max s,g,a | f
( s, g, a ) − h
( s, g, a ) | .
Proof.
| ¯
[Equation]
(Using Lemma 4)
If T = T SOP , then ¯ Q ∗ ( s, g, a ) = ¯ Q ∗ SOP ( s, g, a ) , and we are done.
Let T (cid:54) =
T SOP .
Without loss of generality, let ¯ Q ∗ ( s, g, a ) =
¯ Q ∗ MAX ( s, g, a ) and ¯ Q ∗ SOP ( s, g, a ) =
¯ Q ∗ MIN ( s, g, a ) .
Then,
[Equation]
Lemma 6.
Let Q ∗ and ¯ Q ∗ be the optimal Q -value function and optimal extended Q -value function respectively for a deterministic task in M .
Then for all ( s, a ) in S × A , we have
[Equation]
Proof.
We ﬁrst note that
[Equation]
Now deﬁne
[Equation]
Then it follows that
[Equation]
p
[Equation]
Using Equation 7)
[Equation]
since ¯ r ( s, g, a ) =
r 0 ( s, a ) ∀ s / ∈ G and p ( s, a, ω )
= 1 with ¯ Q ∗ ( ω, g, a (cid:48) )
= 0 ∀ s ∈ G .
[Equation]
Hence ¯ Q ∗ max is a ﬁxed point of the Bellman optimality operator.
If s ∈ G , then
[Equation]
Since ¯ Q ∗ max = Q ∗ holds in G and ¯ Q ∗ max is a ﬁxed point of the Bellman operator, then ¯ Q ∗ max =
Q ∗ holds everywhere.
Theorem 1.
Let M ∈ M be a task with binary representation T and optimal extended action-value function ¯ Q ∗ .
Given (cid:15) -approximations of the binary representations ˜ T n = { ˜ T 1 , ..., ˜ T n } and optimal ¯ Q -functions ˜¯ Q ∗ n = { ˜¯ Q ∗ 1 , ..., ˜¯ Q ∗ n } for n tasks ˆ M = { M 1 , ..., M n }
⊆ M , let
T SOP =
B EXP ( ˜ T n ) and ¯ Q SOP =
B EXP ( ˜¯ Q ∗ n ) where B EXP = SOP ( ˜ T n , ˜ T ) .
Deﬁne, π
[Equation]
Then
[Equation]
(i) (cid:107) [Equation] (cid:107) [Equation]
1 T
[Equation]
T SOP
[Equation] r/ ∈ [Equation] g
[Equation] G [Equation] ) r ∆ + (cid:15) ) , (ii) If the dynamics are deterministic,
[Equation]
where 1 is the indicator function, r g ( s, a ) :
=
¯ r
( s, g, a ) , r ∆ := r MAX − r MIN , and (cid:107) f − h (cid:107) ∞ :
= max s,g,a | f ( s, g, a ) − h ( s, g, a ) | .
Proof.
(i): We ﬁrst note that each g in G can be thought of as deﬁning an MDP M g :=
( S , A , p, r g , γ ) with reward function r g ( s, a ) :
=
¯ r
( s, g, a ) , optimal policy π ∗ g ( s ) =
¯ π ∗ ( s, g ) and optimal Q-value function Q π ∗ g ( s, a ) =
¯ Q ∗ ( s, g, a ) .
Then this proof follows similarly to that of Barreto et al.
Theorem 2,
[Equation] [Equation]
( Since rewards only differ in G where r ( s, a ) , r
[Equation]
s, a
[Equation]
MIN ,
[Equation]
MAX
[Equation]
for s ∈ G )
[Equation]
Hence, (cid:107)
[Equation]
( Since min g 1 r (cid:54) =
r g = 0 only when r ∈ { r g } |G| ) .
[Equation]
ii):
[Equation]
A. C OMPARING THE BOUNDS OF T HEOREM 1 WITH THAT OF GPI IN B ARRETO ET AL . (2018)
We ﬁrst restate Proposition 1 (Barreto et al., 2018) here.
Proposition 4 ((Barreto et al., 2018)) .
Let M ∈ M and let Q π ∗
j
i be the action value function of an optimal policy of M j ∈ M when executed in M i ∈ M .
Given approximations { ˜ Q π 1 i , ..., ˜ Q π n i } such that | Q π j i − ˜ Q
π j i | ≤ (cid:15) for all s, a ∈ S × A , and j ∈ { 1 , ..., n } , let
[Equation]
then,
[Equation]
where Q ∗
We can simplify the bound in Proposition 4 as follows:
[Equation]
Since rewards only differ in G where r
[Equation]
MIN
[Equation]
r
[Equation] [Equation]
G
[Equation]
( Since min j 1 r i (cid:54) =
[Equation]
= 0 only when r i ∈ { r j } n )
[Equation]
where 1 is the indicator function, and r ∆ :=
r MAX − r MIN .
We can see that this bound is similar to that of Theorem 1(i) but weaker.
This because:
(i)
The ﬁrst term of this bound requires that the current task be identical to the task being approximated— 1 r (cid:54) =
r i —while the ﬁrst term of Theorem 1(i) only requires the current task to be expressible as a Boolean composition of past tasks— 1 T (cid:54) =
T SOP .
(ii)
The second term of this bound requires that the task being approximated is one of the past tasks— 1 r i / ∈{ r j } n —while the second term of Theorem 1(i) only requires the current task to have a single desirable goal—
[Equation]
iii) Barreto
et al.
[Equation]
assumes that the reward function of the current task is well approximated by a linear function over a ﬁxed set of rewards.
Hence while a new task may be expressed as the Boolean composition of past tasks— T = T SOP —, its rewards may not be expressible as a linear combination of a ﬁxed set of rewards— r (cid:54) =
r i where r i :=
[ r 0 , ..., r n ] ∗ w .
This suggests that we can can think of the SOP composition approach as an efﬁcient way of doing
GPI, one which leads to tight performance bounds on the transferred policy (Theorem 1(ii)).
A. P ROOFS FOR T HEOREM
Theorem 2.
Let D be an unknown non-stationary distribution over a set of tasks M ( S , A , p, γ, r 0 ) , and let A :
M → ¯
Q ∗ be any map from M to ¯ Q ∗ such that A ( M ) =
¯ Q ∗ M for all M in M .
Let ˜
[Equation]
, ˜¯ Q
[Equation]
= SOPGOL
[Equation]
A , M t , ˜ T t , ˜¯ Q ∗ t ) where M t ∼ D ( t ) and ˜ T 0 = ˜¯ Q ∗
= ∅ ∀ t ∈ N .
Then,
[Equation]
Proof.
Let ˜ T t be the approximate binary representation of task M t learned by SOPGOL.
We ﬁrst note that SOPGOL returns ˜ T t ∪ { ˜ T t } only if ˜ T t is not in the span of ˜ T t .
That is, ˜
T t +1 =
˜ T t ∪ { ˜ T t } iff ˜ T t (cid:54) =
B EXP ( ˜ T t ) where B EXP = SOP ( ˜ T t , ˜ T t ) .
Hence, it is sufﬁcient to show that the number, N , of linearly independent binary vectors, ˜ T ∈ { 0 , 1 } |G| , that span the Boolean vector space (Subrahmanyam, 1964), GF (2) |G| , 5 is bounded by
[Equation]
This follows from the fact that (cid:100) log |G|(cid:101) is the size of a minimal basis of GF (2) |G| (as can easily be seen with a Boolean table), and |G| is its dimensionality.
B S UM O F P RODUCTS WITH G OAL O RIENTED L EARNING
Input : off-policy RL algorithm A , /* e.g DQN */ task MDP M , set of (cid:15) -optimal task binary representations ˜ T , set of (cid:15) -optimal
¯ Q -value functions ˜¯ Q .
Initialise ˜ T : G → { 0 , 1 }
Initialise ˜¯ Q : S × G ×
A → R according to A
Initialise goal buffer ˜
G with terminal states observed from a random policy while ˜¯ Q is not converged do
Initialise state s from M
B EXP ← SOP ( ˜ T , ˜ T )
T SOP , ¯ Q SOP ← B EXP ( ˜ T ) , B EXP ( ˜¯ Q ∗ ) ¯ Q ← ¯ Q SOP if ˜ T
=
T SOP else
˜ ¯ Q ∨ ¯ Q SOP g ← arg max g (cid:48) ∈ ˜ G (cid:18) max a ∈A ¯ Q ( s, g (cid:48) , a ) (cid:19) while s is not terminal do
Select action a using the behaviour policy from A : a ← ¯ π ( s, g ) /* e.g (cid:15) -greedy */
Take action a , observe reward r and next state s (cid:48) in M if ˜ T (cid:54) =
T SOP then foreach g (
cid:48) ∈
˜ G do ¯ r ← r MIN if g (cid:48) (cid:54) = s ∈ ˜
G else r
Update ˜¯ Q with ( s, g (cid:48) , a, ¯ r, s (cid:48) )
according to A end if s is terminal then ˜
T ( s ) ← 1 r = r MAX ˜
G ← ˜
G ∪ { s } else s ← s (cid:48) end end end
B EXP ← SOP ( ˜ T , ˜ T ) ˜
T , ˜ ¯ Q ← ( ˜ T , ˜ ¯ Q ) if ˜ T = B EXP ( ˜ T ) else ( ˜ T ∪ { ˜ T } , ˜ ¯ Q ∪ { ˜ ¯ Q } ) return ˜ T , ˜¯ Q
C F UNCTION A PPROXIMATION E XPERIMENT D ETAILS C. E NVIRONMENT
The PickUpObj environment is fully observable, where each state observation is a 56 ∗ 56 ∗ 3 RGB image (Figure 1).
The agent has 7 actions it can take in this environment corresponding to: 1 - rotate left, 2 - rotate right, 3 - move one step forward if there is no wall or object in front, 4 - pickup object if there is an object in front and no object has been picked, 5 - drop the object in front if an object has been picked and there is no wall or object in front, 6 - open the door in front if there is a closed-door in front, and 7 - close the door in front if there is an opened door in front.
For each task, each episode starts with 1 desirable object and 4 other randomly chosen objects placed randomly in the environment.
The agent is also placed at a random position with a random orientation at the start of each episode.
The agent receives a reward of -0.1 at every timestep, and a reward of 2 when it picks up a desirable object.
The environment transitions to a terminal state once the agent picks up any object and the agent observes the picked object.
There are 15 types of objects (illustrated in Table 1) resulting in 15 possible goal states.
Hence, the dimension of the state space is |S| = 56 ∗ 56 ∗ 3 , the goal space is |G| = 15 , and the action space is |A| = 7 .
C. N ETWORK A RCHITECTURE AND H YPERPARAMETERS
In our function approximation experiments, we represent each extended value function ˜¯ Q ∗ with a list of |G| DQNs, such that the value function for each goal ˜
Q ∗ g ( s, a ) := ˜¯ Q ∗ ( s, g, a ) is approximated with a separate DQN.
The DQNs used have the following architecture, with the CNN part being identical to that used by Mnih et al.
Three convolutional layers: (a) Layer 1 has 3 input channels, 32 output channels, a kernel size of 8 and a stride of 4.
(b) Layer 2 has 32 input channels, 64 output channels, a kernel size of 4 and a stride of 2.
(c) Layer 3 has 64 input channels, 64 output channels, a kernel size of 3 and a stride of 1. 2.
Two fully-connected linear layers: (a) Layer 1 has input size 3136 and output size 512 and uses a ReLU activation function.
(b) Layer 2 has input size 512 and output size 7 with no activation function.
We used the ADAM optimiser with batch size 256 and a learning rate of 10 − 3 .
We started training after 1000 steps of random exploration and updated the target Q-network every 1000 steps.
Finally, we used (cid:15) -greedy exploration, annealing (cid:15) from 0 .
5 to 0 .
05 over 100000 timesteps.
Finally, we used the same DQN architecture and training hyperparameters for the baseline in all experiments.