A B OOLEAN T ASK A LGEBRA FOR R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a framework for deﬁning a
Boolean algebra over the space of tasks
This allows us to formulate new tasks in terms of the negation, disjunction and conjunction of a set of base tasks
We then show that by learning goal-oriented value functions and restricting the transition dynamics of the tasks
, an agent can solve these new tasks with no further learning.
We prove that by composing these value functions in speciﬁc ways, we immediately recover the optimal policies for all tasks expressible under the Boolean algebra.
We verify our approach in two domains—including a high-dimensional video game environment requiring function approximation—where an agent ﬁrst learns a set of base skills, and then composes them to solve a super-exponential number of new tasks.
I NTRODUCTION
Reinforcement learning (RL) has achieved recent success in a number of difﬁcult, high-dimensional environments (Mnih et al., 2015; Levine et al., 2016; Lillicrap et al., 2016; Silver et al., 2017).
However, these methods generally require millions of samples from the environment to learn optimal behaviours, limiting their real-world applicability.
A major challenge is thus in designing sample-efﬁcient agents that can transfer their existing knowledge to solve new tasks quickly.
This is particularly important for agents in a multitask or lifelong setting, since learning to solve complex tasks from scratch is typically infeasible.
One approach to transfer is composition (Todorov, 2009), which allows an agent to leverage existing skills to build complex, novel behaviours.
These newly-formed skills can then be used to solve or speed up learning in a new task.
In this work, we focus on concurrent composition, where existing base skills are combined to produce new skills (Todorov, 2009; Saxe et al., 2017; Haarnoja et al., 2018; Van Niekerk et al., 2019; Hunt et al., 2019; Peng et al., 2019).
This differs from other forms of composition, such as options (Sutton et al., 1999) and hierarchical RL (Bacon et al., 2017), where actions and skills are chained in a temporal sequence.
In this work, we deﬁne a Boolean algebra over the space of tasks and optimal value functions.
This extends previous composition results to encompass all Boolean operators: conjunction, disjunction, and negation.
We then prove that there exists a homomorphism between the task and value function algebras.
Given a set of base tasks that have been previously solved by the agent, any new task written as a Boolean expression can immediately be solved without further learning, resulting in a zero-shot super-exponential explosion in the agent’s abilities.
We illustrate our approach in a simple domain, where an agent ﬁrst learns to reach a number of rooms, after which it can then optimally solve any task expressible in the Boolean algebra.
We then demonstrate composition in high-dimensional video game environments, where an agent ﬁrst learns to collect different objects, and then compose these abilities to solve complex tasks immediately.
Our results show that, even when function approximation is required, an agent can leverage its existing skills to solve new tasks without further learning.
P RELIMINARIES
We consider tasks modelled by Markov Decision Processes (MDPs).
An MDP is deﬁned by the tuple ( S , A , ρ, r ) , where (i) S is the state space, (ii) A is the action space, (iii) ρ is a Markov transition kernel ( s, a ) (cid:55)→ ρ ( s,a ) from S × A to S , and (iv) r is the real-valued reward function bounded by [ r MIN , r MAX ] .
In this work, we focus on stochastic shortest path problems (Bertsekas & Tsitsiklis, 1991), which model tasks in which an agent must reach some goal.
We therefore consider the class of undiscounted MDPs with an absorbing set G ⊆ S .
The goal of the agent is to compute a Markov policy π from S to A that optimally solves a given task.
A given policy π is characterised by a value function V π ( s ) =
E π [ (cid:80) ∞ t =0 r ( s t , a t )] , specifying the expected return obtained under π starting from state s .
The optimal policy π ∗ is the policy that obtains the greatest expected return at each state:
V π ∗ ( s ) =
V ∗ ( s ) = max π V π ( s ) for all s in S .
A related quantity is the Q -value function, Q π ( s, a ) , which deﬁnes the expected return obtained by executing a from s , and thereafter following π .
Similarly, the optimal Q -value function is given by Q ∗ ( s, a ) = max π Q π ( s, a ) for all s in S and a in A .
Finally, we denote a proper policy to be a policy that is guaranteed to eventually reach the absorbing set G (James & Collins, 2006;
Van Niekerk et al., 2019).
We assume the value functions for improper policies—those that never reach absorbing states—are unbounded below.
B OOLEAN A LGEBRAS FOR T ASKS AND V ALUE F UNCTIONS
In this section, we develop the notion of a Boolean task algebra, allowing us to perform logical operations—conjunction ( ∧ ), disjunction ( ∨ ) and negation ( ¬ )—over the space of tasks.
We then show that, having solved a series of base tasks, an agent can use its knowledge to solve tasks expressible as a Boolean expression over those tasks, without any further learning.
We consider a family of related MDPs M restricted by the following assumptions:
Assumption 1.
For all tasks in a set of tasks M , (i) the tasks share the same state space, action space and transition dynamics, (ii) the transition dynamics are deterministic, (iii) reward functions between tasks differ only on the absorbing set G , and (iv) the set of possible terminal rewards consists of only two values. That is, for all ( g, a ) in G × A , we have that r ( g, a ) ∈ { r ∅ , r U } ⊂ R with r ∅ ≤ r U . For all non-terminal states, we denote the reward r s,a to emphasise that it is constant across tasks.
Assumption 2.
For all tasks in a set of tasks M which adhere to Assumption 1, the set of possible terminal rewards consists of only two values.
That is, for all ( g, a ) in G × A , we have that r ( g, a ) ∈ { r ∅ , r U } ⊂ R with r ∅ ≤ r U .
For all non-terminal states, we denote the reward r s,a to emphasise that it is constant across tasks.
Assumption 1 is similar to that of Todorov (2007) and identical to Van Niekerk et al.
, and imply that each task can be uniquely speciﬁed by its reward function. Furthermore, we note that Assumption 2 is only necessary to formally deﬁne the Boolean algebra. Although we have placed restrictions on the reward functions, the above formulation still allows for a large number of tasks to be represented.
Importantly, sparse rewards can be formulated under these restrictions.
3. A B OOLEAN A LGEBRA FOR T ASKS
An abstract Boolean algebra is a set B equipped with operators ¬ , ∨ , ∧ that satisfy the Boolean axioms of (i) idempotence, (ii) commutativity, (iii) associativity, (iv) absorption, (v) distributivity, (vi) identity, and (vii) complements.
2 Given the above deﬁnitions and the restrictions placed on the set of tasks we consider, we can now deﬁne a Boolean algebra over a set of tasks.
Theorem 1.
Let M be a set of tasks.
Deﬁne M U , M ∅ ∈ M to be tasks with the respective reward functions
[Equation]
Then M forms a Boolean algebra with universal bounds M ∅ and M U when equipped with the following operators:
[Equation]
Proof.
See Appendix.
Theorem 1 allows us to compose existing tasks together to create new tasks in a principled way.
Figure 1 illustrates the semantics for each of the Boolean operators in a simple environment.
3. E XTENDED V ALUE F UNCTIONS
The reward and value functions described in Section 2 are insufﬁcient to solve tasks speciﬁed by the Boolean algebra above.
We therefore extend these to deﬁne goal-oriented versions of the reward and value function, given by the following two deﬁnitions:
Deﬁnition 1.
The extended reward function ¯ r : S × G ×
A → R is given by the mapping
[Equation]
where N ≤ min { r MIN , ( r MIN − r MAX ) D } , and D is the diameter of the MDP (Jaksch et al., 2010).
To understand why standard value functions are insufﬁcient, consider two tasks that have multiple different goals, but at least one common goal.
Clearly, there is a meaningful conjunction between them—namely, achieving the common goal.
Now consider an agent that learns standard value functions for both tasks, and which is then required to solve their conjunction without further learning.
Note that this is impossible in general, since the regular value function for each task only represents the value of each state with respect to the nearest goal. That is, for all states where the nearest goal for each task is not the common goal, the agent has no information about that common goal. Conversely, by learning extended value functions, the agent is able to learn the value of achieving all goals, and not simply the nearest one. Because we require that tasks share the same transition dynamics, we also require that the absorbing set of states is shared.
Thus the extended reward function adds the extra constraint that, if the agent enters a terminal state for a different task, it should receive the largest penalty possible.
In practice, we can simply set N to be the lowest ﬁnite value representable by the data type used for rewards.
Deﬁnition 2.
The extended Q-value function ¯ Q : S × G ×
A → R is given by the mapping
[Equation]
where ¯ V ¯ π ( s, g )
= E ¯ π [ (cid:80) ∞ t =0 ¯ r ( s t , g, a t )] . The extended Q-value function is similar to universal value function approximators (UVFAs) (Schaul et al., 2015), but differs in that it uses the extended reward function deﬁnition.
It is also similar to DG functions (Kaelbling, 1993), except here we use task-dependent reward functions, as opposed to measuring distance between states.
The standard reward functions and value functions can be recovered from their extended versions through the following lemma. [Table] [Figure]
Lemma 1.
Let r M , ¯ r M , Q ∗ M , ¯ Q ∗ M be the reward function, extended reward function, optimal Qvalue function, and optimal extended Q-value function for a task M in M .
Then for all ( s, a ) in
S × A , we have (i) r M
( s, a )
[Equation]
s, g, a ) , and (ii) Q ∗ M ( s, a ) = max g ∈G ¯ Q ∗ M ( s, g, a ) .
Proof.
[Equation]
ρ
[Equation]
g function
[Equation]
[Equation] [Equation] g ( s, a ) =
¯ Q ∗ M ( s, g, a ) .
Then using (i) we have r M ( s, a ) =
max g ∈G r M g ( s, a ) and from Van Niekerk et al.
(2019, Corollary 1), we have that Q ∗ M ( s, a ) = max g ∈G Q ∗ M g ( s, a ) = max g ∈G ¯ Q [Equation] M
[Equation] s, g, a ) . In the same way, we can also recover the optimal policy from these extended value functions by ﬁrst applying Lemma 1, and acting greedily with respect to the resulting value function.
Lemma 2.
Denote S − =
S \ G as the non-terminal states of M .
Let M 1 , M 2 ∈ M , and let each g in
G deﬁne MDPs M 1 ,g and M 2 ,g with reward functions
[Equation]
Then for all g in G and s in S − ,
[Equation]
Proof.
See Appendix.
Combining Lemmas 1 and 2, we can extract the greedy action from the extended value function by ﬁrst maximising over goals, and then selecting the maximising action: π ∗ ( s ) ∈ arg max a ∈A max g ∈G ¯ Q ∗ ( s, g, a ) .
If we consider the extended value function to be a set of standard value functions (one for each goal), then this is equivalent to ﬁrst performing generalised policy improvement (Barreto et al., 2017), and then selecting the greedy action.
Finally, much like the regular deﬁnition of value functions, the extended Q-value function can be written as the sum of rewards received by the agent until ﬁrst encountering a terminal state.
Corollary 1.
Denote G ∗ s : g,a as the sum of rewards starting from s and taking action a up until, but not including, g .
Then let M ∈ M and ¯ Q ∗ M be the extended Q-value function.
Then for all s ∈ S , g ∈ G , a ∈ A , there exists a G ∗ s : g,a ∈ R such that
[Equation]
Proof.
This follows directly from Lemma 2.
Since all tasks M ∈ M share the same optimal policy π ∗ g up to (but not including) the goal state g ∈ G , their return G {+π ∗ g T − 1 = (cid:80) T − 1 t =
+}
r M ( s t , π ∗ g ( s t )) is the same up to (but not including) g .
3. A B OOLEAN A LGEBRA FOR V ALUE F UNCTIONS
In the same manner we constructed a Boolean algebra over a set of tasks, we can also do so for a set of optimal extended Q-value functions for the corresponding tasks.
Theorem 2.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M .
Deﬁne ¯
Q ∗ ∅ , ¯ Q ∗ U ∈ ¯ Q ∗ to be the optimal ¯ Q -functions for the tasks M ∅ , M U ∈ M .
Then ¯ Q ∗ forms a Boolean algebra when equipped with the following operators: ¬ : ¯
[Equation]
∗ → ¯
[Equation]
∗ ¯ Q ∗
[Equation]
¬ ¯ Q ∗ , where ¬ ¯ Q
[Equation]
× G
[Equation]
∨ : ¯ Q ∗ × ¯ Q ∗ → ¯ Q ∗ ( ¯ Q ∗ 1 , ¯ Q ∗ 2 ) (cid:55)→ ¯ Q ∗ 1 ∨ ¯ Q ∗ 2 , where ¯ Q ∗ 1 ∨ ¯ Q ∗ 2 : S × G ×
A → R ( s, g, a ) (cid:55)→ max { ¯ Q ∗ 1 ( s, g, a ) , ¯ Q ∗ 2 ( s, g, a ) } ∧ : ¯ Q ∗ × ¯ Q ∗ → ¯ Q ∗ ( ¯ Q ∗ 1 , ¯ Q ∗ 2 ) (cid:55)→ ¯ Q ∗ 1 ∧ ¯ Q ∗ 2 , where ¯ Q ∗ 1 ∧ ¯ Q ∗ 2 : S × G ×
A → R ( s, g, a ) (cid:55)→ min { ¯ Q ∗ 1 ( s, g, a ) , ¯ Q ∗ 2 ( s, g, a ) }
Proof.
See Appendix.
3. B ETWEEN T ASK AND V ALUE F UNCTION A LGEBRAS
Having established a Boolean algebra over tasks and extended value function, we ﬁnally show that there exists an equivalence between the two.
As a result, if we can write down a task under the
Boolean algebra, we can immediately write down the optimal value function for the task.
Theorem 3.
Let F : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that F ( M ) =
¯ Q ∗ M for all M in
M .
Then F is a homomorphism.
Proof.
See Appendix.
Z ERO - SHOT T RANSFER T HROUGH C OMPOSITION
We can use the theory developed in the previous sections to perform zero-shot transfer by ﬁrst learning extended value functions for a set of base tasks, and then composing them to solve new tasks expressible under the Boolean algebra.
To demonstrate this, we conduct a series of experiments in a Four Rooms domain (Sutton et al., 1999), where an agent must navigate in a grid world to a particular location.
The agent can move in any of the four cardinal directions at each timestep, but colliding with a wall leaves the agent in the same location.
The transition dynamics are deterministic, and rewards are − 0 .
1 for all non-terminal states, and 1 at the goal.
4. L EARNING B ASE T ASKS
We use a modiﬁed version of Q-learning (Watkins, 1989) to learn extended Q-value functions described previously.
Our algorithm differs in a number of ways from standard Q-learning: we keep track of the set of terminating states seen so far, and at each timestep we update the extended Q-value function with respect to both the current state and action, as well as all goals encountered so far.
We also use the deﬁnition of the extended reward function, and so if the agent encounters a terminal state of a different task, it receives reward N .
The full pseudocode is listed in the Appendix.
If we know the set of goals (and hence potential base tasks) upfront, then it is easy to select a minimal set of base tasks that can be composed to produce the largest number of composite tasks.
We ﬁrst assign a Boolean label to each goal in a table, and then use the columns of the table as base tasks.
The goals for each base task are then those goals with value 1 according to the table.
In this domain, the two base tasks we select are M T , which requires that the agent visit either of the top two rooms, and M L , which requires visiting the two left rooms.
We illustrate this selection procedure in the
Appendix.
4. B OOLEAN C OMPOSITION
Having learned the optimal extended value functions for our base tasks, we can now leverage Theorems 1–3 to solve new tasks with no further learning.
Figure 2 illustrates this composition, where an agent is able to immediately solve complex tasks such as exclusive-or.
We illustrate a few composite tasks here, but note that in general, if we have K base tasks, then a Boolean algebra allows for 2 2 K new tasks to be constructed.
Thus having trained on only two tasks, our agent has enough information to solve a total of 16 composite tasks.
[Figure]
By learning extended value functions, an agent can subsequently solve a massive number of tasks; however, the upfront cost of learning is likely to be higher.
We investigate the trade-off between the two approaches by investigating how the sample complexity scales with the number of tasks.
We compare to Van Niekerk et al.
(2019), who used regular value functions to demonstrate optimal disjunctive composition.
We note that while the upfront learning cost is therefore lower, the number of tasks expressible using only disjunction is 2 K − 1 , which is signiﬁcantly less than the full Boolean algebra.
We also test using an extended version of the Four Rooms domain, where additional goals are placed along the sides of all walls, resulting in a total of 40 goals.
Empirical results are illustrated by Figure 3.
Our results show that while additional samples are needed to learn an extended value function, the agent is able to expand the tasks it can solve super-exponentially.
Furthermore, the number of base tasks we need to solve is only logarithmic in the number of goal states.
For an environment with K goals, we need to learn only (cid:98) log 2 K (cid:99) + 1 base tasks, as opposed to the disjunctive approach which requires K base tasks.
Thus by sacriﬁcing sample efﬁciency initially, we achieve an exponential increase in abilities compared to previous work (Van Niekerk et al., 2019). [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure]  C OMPOSITION WITH F UNCTION A PPROXIMATION
Finally we demonstrate that our compositional approach can also be used to tackle high-dimensional domains where function approximation is required.
We use the same video game environment as
Van Niekerk et al.
(2019), where an agent must navigate a 2D world and collect objects of different shapes and colours.
The state space is an 84 × 84 RGB image, and the agent is able to move in any of the four cardinal directions.
The agent also possesses a pick-up action, which allows it to collect an object when standing on top of it.
There are two shapes (squares and circles) and three colours (blue, beige and purple) for a total of six unique objects.
The position of the agent is randomised at the start of each episode.
We modify deep Q-learning (Mnih et al., 2015) to learn extended action-value functions.
4 Our approach differs in that the network takes a goal state as additional input (again speciﬁed as an RGB image).
Additionally, when a terminal state is encountered, it is added to the collection of goals seen so far, and when learning updates occur, these goals are sampled randomly from a replay buffer.
We ﬁrst learn to solve two base tasks: collecting blue objects, and collecting squares, which can then be composed to solve new tasks immediately.
We demonstrate composition characterised by (i) disjunction, (ii) conjunction and (iii) exclusiveor.
This corresponds to tasks where the target items are: (i) blue or square, (ii) blue squares, and (iii) blue or squares, but not blue squares.
Figure 4 illustrates sample trajectories, as well as the subsequent composed value functions, for the respective tasks.
R ELATED W ORK
The ability to compose value functions was ﬁrst demonstrated using the linearly-solvable MDP framework (Todorov, 2007), where value functions could be composed to solve tasks similar to the disjunctive case (Todorov, 2009).
Van Niekerk et al.
(2019) show that the same kind of composition can be achieved using entropy-regularised RL (Fox et al., 2016), and extend the results to the standard RL setting, where agents can optimally solve the disjunctive case.
Using entropy-regularised
RL, Haarnoja et al.
(2018) approximates the conjunction of tasks by averaging their reward functions, and demonstrates that by averaging the optimal value functions of the respective tasks, the agent can achieve performance close to optimal.
Hunt et al.
(2019) extends this result by composing value functions to solve the average reward task exactly, which approximates the true conjunctive case.
More recently, Peng et al.
(2019) introduce a few-shot learning approach to compose policies [Figure] multiplicatively.
Although lacking theoretical foundations, results show that an agent can learn a weighted composition of existing base skills to solve a new complex task.
By contrast, we show that zero-shot optimal composition can be achieved for all Boolean operators.
C ONCLUSION
We have shown how to compose tasks using the standard Boolean algebra operators.
These composite tasks can be immediately solved by ﬁrst learning goal-oriented value functions, and then composing them in a similar manner.
Finally, we note that there is much room for improvement in learning the extended value functions for the base tasks.
In our experiments, we learned each extended value function from scratch, but it is likely that having learned one for the ﬁrst task, we could use it to initialise the extended value function for the second task to improve convergence times.
One area for improvement lies in efﬁciently learning the extended value functions, as well as developing better algorithms for solving tasks with sparse rewards. For example, it is likely that approaches such as hindsight experience replay (Andrychowicz et al., 2017) could reduce the number of samples required to learn extended value functions, while Mirowski et al. (2017) provides a method for learning complex tasks with sparse rewards using auxiliary tasks.
We leave incorporating these approaches to future work. Our proposed approach is a step towards both interpretable RL— since both the tasks and optimal value functions can be speciﬁed using Boolean operators—and the ultimate goal of lifelong learning agents, which are able to solve combinatorially many tasks in a sample-efﬁcient manner.
A A PPENDIX A. B OOLEAN A LGEBRA D EFINITION
Deﬁnition 3.
A Boolean algebra is a set B equipped with the binary operators ∨ (disjunction) and ∧ (conjunction), and the unary operator ¬ (negation), which satisﬁes the following Boolean algebra axioms for a, b, c in B :
(i) Idempotence: a ∧ a = a ∨ a = a .
(ii) Commutativity: a
∧ b = b ∧ a and a ∨ b =
b ∨ a .
(iii) Associativity: a ∧ ( b ∧ c ) =
( a ∧ b ) ∧ c and a ∧ ( b ∨ c )
= ( a ∨ b ) ∨ c .
(iv) Absorption: a ∧ ( a ∨ b ) =
a ∨ ( a ∧ b ) = a .
(v) Distributivity: a ∧ ( b ∨ c )
= ( a ∧ b ) ∨ ( a ∧ c ) and a ∨ ( b ∧ c ) =
( a ∨ b ) ∧ ( a ∨ c ) .
(vi) Identity: there exists 0 , 1 in B such that
[Equation]
(vii) Complements: for every a in B , there exists an element a (cid:48) in B such that a ∧ a (cid:48)
= 0 and a ∨ a (cid:48)
= 1 .
A. P ROOF FOR T HEOREM
Theorem 1.
Let M be a set of tasks.
Deﬁne M U , M ∅ ∈ M to be tasks with the respective reward functions
[Equation]
Then M forms a Boolean algebra with universal bounds M ∅ and M U when equipped with the following operators:
[Equation]
(cid:55)→ ( S , A , ρ, r ¬ M ) , where r ¬ M : S × A → R ( s, a ) (cid:55)→ (cid:0)
[Equation]
U
[Equation]
s,
[Equation]
s,
[Equation] [Equation]
∨ : M × M → M ( M 1 , M 2 ) (cid:55)→ ( S , A , ρ, r M 1 ∨ M 2 ) , where r M 1 ∨ M 2 : S ×
A → R ( s, a ) (cid:55)→ max { r M 1 ( s, a ) , r M 2 ( s, a ) } ∧ : M × M → M ( M 1 , M 2 )
(cid:55)→ ( S , A , ρ, r M 1 ∧ M 2 ) , where r M 1 ∧ M 2 : S ×
A → R
[Equation] [Equation]
(cid:55)→
[Equation] [Equation] [Equation] [Equation]
Proof.
Let M 1 , M 2 ∈ M .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i)–(v):
These easily follow from the fact that the min and max functions satisfy the idempotent, commutative, associative, absorption and distributive laws.
(vi): Let r M U
∧ M 1 and r M 1 be the reward functions for M U ∧ M 1 and M 1 respectively.
Then for all ( s, a ) in
[Equation]
A
[Equation]
s ∈
[Equation]
= r
[Equation]
s, a
[Equation]
Thus M
U
∧ M
1 =
M
Similarly M
U ∨
M 1 = M U , M ∅ ∧ M 1 = M ∅ , and M
∅ ∨ M
=
M
Hence M ∅ and M U are the universal bounds of M
(vii): Let r M 1 ∧¬ M 1 be the reward function for M 1 ∧ ¬ M 1 .
Then for all ( s, a ) in S × A ,
[Equation]
Thus M 1 ∧ ¬ M 1 = M ∅ , and similarly M 1 ∨ ¬ M 1 = M U .
A. P ROOF FOR L EMMA
Lemma 2.
Denote S − =
S \ G as the non-terminal states of M .
Let M 1 , M 2 ∈ M , and let each g in
G deﬁne MDPs M 1 ,g and M 2 ,g with reward functions
[Equation]
:= ¯ r M 1 ( s, g, a ) and r M 2 ,g :=
¯ r M 2 ( s, g, a ) for all ( s, a ) in S × A .
Then for all g in G and s in S − ,
[Equation]
Proof.
Let g ∈ G , s ∈ S − and let π ∗ g be deﬁned by
[Equation]
If g is unreachable from s , then we are done since for all ( s (cid:48) , a ) in S
× A we have
[Equation]
If g is reachable from s , then we show that following π ∗ g must reach g .
Since π ∗ g is proper, it must reach a terminal state g (cid:48) ∈ G .
Assume g (cid:48) (cid:54) =
g .
Let π g be a policy that produces the shortest trajectory to g .
Let G π ∗ g
and G π g be the returns for the respective policies.
Then,
[Equation]
where G π ∗ g T − 1 =
T −
[Equation]
t = r M 1 ,g ( s t , π ∗ g ( s t )) and T is the time at which g (cid:48) is reached.
[Equation]
since
[Equation]
deﬁnition of
[Equation]
But this is a contradiction since the result obtained by following an optimal trajectory up to a terminal state without the reward for entering the terminal state must be strictly less that receiving r MAX for every step of the longest possible optimal trajectory.
Hence we must have g (cid:48)
=
g .
Similarly, all optimal policies of M 2 ,g must reach g .
Hence π ∗ g ( s ) ∈ arg max a ∈A
Q ∗ M 2 ,g ( s, a ) .
Since M 1 and M 2 are arbitrary elements of M , the reverse implication holds too.
A. P ROOF FOR T HEOREM
Theorem 2.
Let ¯ Q ∗ be the set of optimal extended ¯ Q -value functions for tasks in M .
Deﬁne ¯
Q ∗ ∅ , ¯ Q ∗U ∈ ¯ Q ∗ to be the optimal ¯ Q -functions for the tasks M ∅ , M U ∈ M .
Then ¯ Q ∗ forms a Boolean algebra when equipped with the following operators:
[Equation]
¯ Q
[Equation]
¬ ¯ Q ∗ , where ¬ ¯ Q ∗ : S × G ×
A → R ( s, g, a ) (cid:55)→ (cid:0) ¯ Q ∗U ( s, g, a ) + ¯ Q ∗ ∅ ( s, g, a ) (cid:1) − ¯ Q ∗ ( s, g, a ) ∨ : ¯ Q ∗ × ¯ Q ∗ → ¯ Q ∗ ( ¯ Q ∗ 1 , ¯ Q ∗ 2 ) (cid:55)→ ¯ Q ∗ 1 ∨ ¯ Q ∗ 2 , where ¯ Q ∗ 1 ∨ ¯ Q ∗ 2 : S × G ×
A → R ( s, g, a ) (cid:55)→ max { ¯ Q ∗ 1 ( s, g, a ) , ¯ Q ∗ 2 ( s, a ) } ∧ : ¯ Q ∗ × ¯ Q ∗ → ¯ Q ∗ ( ¯ Q ∗ 1 , ¯ Q ∗ 2 ) (cid:55)→ ¯ Q ∗ 1 ∧ ¯ Q ∗ 2 , where ¯ Q ∗ 1 ∧ ¯ Q ∗ 2 : S × G ×
A → R ( s, g, a ) (cid:55)→ min { ¯ Q ∗ 1 ( s, g, a ) , ¯ Q ∗ 2 ( s, a ) }
Proof.
Let ¯ Q ∗ M
¯ Q ∗ M 2 ∈ ¯ Q ∗ be the optimal ¯ Q -value functions for tasks M 1 , M 2 ∈ M with reward functions r M 1 and r M 2 .
We show that ¬ , ∨ , ∧ satisfy the Boolean properties (i) – (vii).
(i)–(v): These follow directly from the properties of the min and max functions
[Equation]
[Equation] [Equation]
∗
[Equation]
Similarly, ¯ Q ∗ M 1 ∨ ¬ ¯ Q ∗ M 1 = ¯ Q ∗U .
A. P ROOF FOR T HEOREM
Theorem 3.
Let F : M → ¯ Q ∗ be any map from M to ¯ Q ∗ such that F ( M ) =
¯ Q ∗ M for all M in
M .
Then F is a homomorphism.
Proof.
Let M
[Equation]
M 2 ∈ M .
Then for all ( s, g, a ) in S × G × A , ¯
[Equation]
from Corollary 1)
[Equation]
= ⇒ F ( ¬ M 1 )
=
¬F ( M 1 )
[Equation]
⇒
[Equation]
[Equation]
[Equation]
[Equation]
Similarly F ( M 1 ∧ M 2 ) =
F ( M 1 ) ∧ F ( M 2 ) .
A. G OAL - ORIENTED Q- LEARNING
Below we list the pseudocode for the modiﬁed Q-learning algorithm used in the four-rooms domain.
Algorithm
Input: Learning rate α , discount factor γ , exploration constant ε , lower-bound return N
Initialise Q : S × S ×
A → R arbitrarily
G ← ∅ while Q is not converged do
Initialise state s while s is not terminal do if G = ∅ then Select random action a else a ←   arg max
[Equation]
(cid:18) max
[Equation]
Q
[Equation]
s, t, b ) (cid:19) with probability 1 − ε a random action with probability ε end
Choose a from s according to policy derived from Q
Take action a , observe r and s (cid:48) foreach g ∈ G do if s (cid:48) is terminal then if s (cid:48) (cid:54) =
g then δ ← N else δ ← r − Q ( s, g, a ) end else δ ←
[Equation]
end
Q
[Equation]
←
[Equation]
+ αδ end s ← s (cid:48) end
G ← G ∪ { s } endreturn Q
A. I NVESTIGATING P RACTICAL C ONSIDERATIONS
The theoretical results presented in this work rely on Assumptions 1 and 2, which restrict the tasks’ transition dynamics and reward functions in potentially problematic ways.
Although this is necessary to prove that Boolean algebraic composition results in optimal value functions, in this section we investigate whether these can be practically ignored.
In particular, we investigate two restrictions: the requirement that tasks share the same terminal states, and the impact of using dense rewards.
A.7. F OUR R OOMS E XPERIMENTS
We use the same setup as the experiment outlined in Section 4, but modify it in two ways.
We ﬁrst investigate the difference between using sparse and dense rewards.
Our sparse reward function is deﬁned as
[Equation]
and we use a dense reward function similar to Peng et al.
[Equation]
Using this dense reward function, we again learn to solve the two base task M T (reaching the centre of the top two rooms) and M L (reaching the centre of the left two rooms).
We then compose them to solve a variety of tasks, with the resulting value functions illustrated by Figure 6.
[Figure]
that produces logical nor.
We note that the resulting value functions are very similar to those produced in the sparse reward setting.
We also modify the domain so that tasks need not share the same terminating states (that is, if the agent enters a terminating state for a different task, the episode does not terminate and the agent can continue as if it were a normal state).
This results in four versions of the experiment:
(i) sparse reward, same absorbing set (ii) sparse reward, different absorbing set (iii) dense reward, same absorbing set (iv) dense reward, different absorbing set
We learn extended value functions for each of the above setups, and then compose them to solve each of the 2 4 tasks representable in the Boolean algebra.
We measure each composed value functions by evaluating its policy in the sparse reward setting, averaging results over 100000 episodes.
The results are given by Figure 7.
Our results indicate that extended value functions learned in the theoretically optimal manner ( sparse reward, same absorbing set ) are indeed optimal.
However, for the majority of the tasks, relaxing the restrictions on terminal states and reward functions results in policies that are either identical or very close to optimal.
Q * M T Q * M L Q * M T ¬ Q * M L Q * M L ¬ Q * M T
Q * M T Q * M L
Q * M T ¬ Q * M T
Q * M L ¬ Q * M L
Q * M T Q * M L Q * M T ¬ Q * M L Q * M L ¬ Q * M T
Q * M T Q * M L ¬( Q * M T Q * M L ) Q
* M T Q * M L Tasks 0.
A v e r a g e
R e t u r n s
[Figure]
A.7. F UNCTION A PPROXIMATION E XPERIMENTS
In this section we investigate whether we can again loosen some of the restrictive assumptions when tackling high-dimensional environments.
In particular, we run the same experiments as those presented in Section 5, but modify the domain so that (i) tasks need not share the same absorbing set, (ii) the pickup-up action is removed (the agent immediately collects an object when reaching it), and (iii) the position of every object is randomised at the start of each episode.
We ﬁrst learn to solve three base tasks: collecting blue objects, collecting purple objects, and collecting squares , which can then be composed to solve new tasks immediately.
We then demonstrate composition characterised by disjunction, conjunction and exclusive-or, with the resulting trajectories and value functions illustrated by Figure 8.
(a) Trajectories for disjunctive composition (collect blue or purple objects).
(b) Trajectories for
conjunc- tive composition (collect blue squares).
(c) Trajectories for exclusive- or composition (collect blue or square objects, but not blue squares).
(d) Value function
[Figure]
In summary, we have shown that our compositional approach offers strong empirical performance, even when the theoretical assumptions are violated.
Finally, we expect that, in general, the errors due to these violations will be far outweighed by the errors due to non-linear function approximation.
A. S ELECTING B ASE T ASKS
The Four Rooms domain requires the agent to navigate to one of the centres of the rooms in the environment.
Figure 9 illustrates the layout of the environment and the goals the agent must reach.
Since we know the goals upfront, we can select a minimal set of base tasks by assigning each goal a
Boolean number, and then using the columns of the table to select the tasks.
To illustrate, we assign
Boolean numbers to the goals as follows:
[Table]
As there are four goals, we can represent each uniquely with just two Boolean variables.
Each column in Table 1 represents a base task, where the set of goals for each task are those goals assigned a value r U .
We thus have two base tasks corresponding to
A. DQN A RCHITECTURE AND H YPERPARAMETERS
In our experiments, we used a DQN with the following architecture:
Three convolutional layers: (a) Layer 1 has 6 input channels, 32 output channels, a kernel size of 8 and a stride of 4.
(b) Layer 2 has 32 input channels, 64 output channels, a kernel size of 4 and a stride of 2.
(c) Layer 3 has 64 input channels, 64 output channels, a kernel size of 3 and a stride of 1. 2.
Two fully-connected linear layers: (a) Layer 1 has input size 3136 and output size 512 and uses a ReLU activation function.
(b) Layer 2 has input size 512 and output size 4 with no activation function.
We used the ADAM optimiser with batch size 32 and a learning rate of 10 − 4 .
We trained everytimesteps and update the target Q-network every 1000 steps.
Finally, we used (cid:15) -greedy exploration, annealing (cid:15) to 0 .
01 over 100000 timesteps.