U NIVERSAL S UCCESSOR F EATURES A PPROXIMATORS
Anonymous authors
Paper under double-blind review
A BSTRACT
The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential beneﬁts, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills.
We focus on one aspect in particular, namely the ability to generalise to unseen tasks.
Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs).
Another way to generalise to new tasks is to exploit structure in the RL problem itself.
Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs).
Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI.
We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical beneﬁts and transfer abilities on a large-scale domain in which the agent has to navigate in a ﬁrst-person perspective three-dimensional environment.
I NTRODUCTION
Reinforcement learning (RL) provides a general framework to model sequential decision-making problems with sparse evaluative feedback in the form of rewards.
The recent successes in deep RL have prompted interest in increasingly more complex tasks and a shift in focus towards scenarios in which a single agent must solve multiple related problems, either simultaneously or sequentially.
This paradigm is formally known as multitask RL (Taylor and Stone, 2009; Teh et al., 2017).
One of the beneﬁts of learning about multiple tasks at the same time is the possibility of transferring knowledge across tasks; in essence, this means that by jointly learning about a set of tasks one should be able to exploit their common structure to speed up learning and induce better generalisation (Taylor and
Stone, 2009; Lazaric, 2012).
A particularly interesting instance of transfer is the generalisation to new, unseen tasks.
This potentially allows an agent to perform a task with little or no learning by leveraging knowledge from previously learned tasks.
In this paper we will be exploring this scenario.
Consider an RL agent in a persistent environment trying to master a number of tasks.
In order to generalise to unseen tasks, the agent needs to be able to identify and exploit some common structure underlying the tasks.
Two possible sources of structure in this scenario are: i) some similarity between the solutions of the tasks, either in the policy or in the associated value-function space, and ii) the shared dynamics of the environment ( e.g. , physics).
In this paper we will attempt to build an agent that can make use of both types of structure.
For this, we will build on two frameworks that exploit these structures in isolation.
The ﬁrst one are Schaul et al.’s (2015) universal value function approximators (UVFAs).
UVFAs extend the notion of value functions to also include the description of a task, thus directly exploiting the common structure in the associated optimal value functions.
The second framework we build upon exploits the common structure in the environment and capitalises on the power of dynamic programming.
Barreto et al.
’s (2017) framework is based on two core concepts: successor features (SFs), a representation scheme that allows a policy to be evaluated on any task of a given format, and generalised policy improvement (GPI), a generalisation of dynamic programming’s classic operator that uses a set of policies instead of a single one.
UVFAs and SF&GPI generalise to new tasks in quite different, and potentially complementary, ways.
UVFAs aim to generalise across the space of tasks by exploiting structure in the underlying space of value functions.
In contrast, SF&GPI strategy is to exploit the structure of the RL problem itself.
In this paper we propose a model that exhibits both types of generalisation.
The basic insight is to note that SFs are multi-dimensional value functions, so we can extend them in the same way a universal value function extends their unidimensional counterparts.
We call the resulting model universal successor features approximators , or USFAs for short.
USFA is a strict generalisation of its precursors.
Speciﬁcally, we show that by combining USFAs and GPI we can recover both UVFAs and SF&GPI as particular cases.
This opens up a new spectrum of possible approaches in between these two extreme cases.
We discuss the challenges involved in training a USFA and demonstrate the practical beneﬁts of doing so on a large-scale domain in which the agent has to navigate in a three-dimensional environment using only images as observations.
B ACKGROUND
In this section we present some background material, formalise the scenario we are interested in, and brieﬂy describe the methods we build upon.
2. M ULTITASK REINFORCEMENT LEARNING
We consider the usual RL framework: an agent interacts with an environment and selects actions in order to maximise the expected amount of reward received in the long run (Sutton and Barto, 1998).
As usual, we assume that such an interaction can be modeled as a Markov decision process (MDP,
Puterman, 1994).
An MDP is deﬁned as a tuple M ≡ ( S , A , p, R, γ ) where S and A are the state and action spaces, p ( ·| s, a ) gives the next-state distribution upon taking action a in state s , R ( s, a, s (cid:48) ) is a random variable representing the reward received at transition s a −→ s (cid:48) , and γ ∈ [0 , 1) is a discount factor that gives smaller weights to future rewards.
As mentioned in the introduction, in this paper we are interested in the multitask RL scenario, where the agent has to solve multiple tasks .
Each task is deﬁned by a reward function R w ; thus, instead of a single MDP M , our environment is a set of MDPs that share the same structure except for the reward function.
Following Barreto et al.
(2017), we assume that the expected one-step reward associated with transition s a −→ s (cid:48) is given by
[Equation]
where φ ( s, a, s (cid:48) )
∈
R
d are features of ( s, a, s (cid:48) ) and w ∈ R d are weights.
The features φ ( s, a, s (cid:48) ) can be thought of as salient events that may be desirable or undesirable to the agent, such as for example picking up an object, going through a door, or knocking into something.
In this paper we assume that the agent is able to recognise such events—that is, φ is observable—, but the solution we propose can be easily extended to the case where φ must be learned (Barreto et al., 2017).
Our solution also applies to the case where (1) is only approximately satisﬁed, as discussed by Barreto et al.
Given w ∈ R d representing a task, the goal of the agent is to ﬁnd a policy π w : S (cid:55)→
A that maximises the expected discounted sum of rewards, also called the return G ( t ) w = (cid:80) ∞ i =
γ i R ( t + i )
w
[Equation]
where R ( t ) w = R w ( S t , A t , S t +1 ) is the reward received at the t th time step.
A principled way to address this problem is to use methods derived from dynamic programming (DP), which heavily rely on the concept of a value function (Puterman, 1994).
The action-value function of a policy π on task w is deﬁned as Q π w ( s, a ) ≡ E π (cid:104) G ( t ) w | S t =
s, A t = a (cid:105) , where E π [ · ] denotes expected value when following policy π .
Based on Q π w we can compute a greedy policy π (cid:48)
( s ) ∈ argmax a Q π w ( s, a ) ; one of the fundamental results in DP guarantees that Q π (cid:48) w ( s, a ) ≥ Q π w ( s, a ) for all ( s, a ) ∈ S × A .
The computation of Q π w ( s, a ) and π (cid:48) are called policy evaluation and policy improvement ; under certain conditions their successive application leads to the optimal value function Q ∗ w , from which one can derive an optimal policy for task w as π ∗ w ( s ) ∈ argmax a Q ∗ w ( s, a ) (Sutton and Barto, 1998).
As a convention, in this paper we will add a tilde to a symbol to indicate that the associated quantity is an approximation; we will then refer to the respective tunable parameters as θ .
For example, the agent computes an approximation ˜
Q
π w ≈ Q π w by tuning θ Q .
2. T RANSFER LEARNING
Here we focus on one aspect of multitask RL: how to transfer knowledge to unseen tasks (Taylor and
Stone, 2009; Lazaric, 2012).
Speciﬁcally, we ask the following question: how can an agent leverage knowledge accumulated on a set of tasks M ⊂ R d to speed up the solution of a new task w (cid:48) / ∈ M ?
In order to investigate the question above we recast it using the formalism commonly adopted in learning.
Speciﬁcally, we deﬁne a distribution D w over R d and assume the goal is for the agent to perform as well as possible under this distribution.
As usual, we assume a ﬁxed budget of sample transitions and deﬁne a training set M ∼ D w that is used by the agent to learn about the tasks of interest.
We also deﬁne a test set M (cid:48) ∼ D w and use it to assess the agent’s generalisation—that is, how well it performs on the distribution of MDPs induced by D w .
A natural way to address the learning problem above is to use Schaul et al.’s (2015) universal valuefunction approximators (UVFAs).
The basic insight behind UVFAs is to note that the concept of optimal value function can be extended to include as one of its arguments a description of the task; an obvious way to do so in the current context is to deﬁne the function Q ∗ ( s, a, w ) :
S ×
A × R d (cid:55)→ R as the optimal value function associated with task w .
The function Q ∗ ( s, a, w ) is called a universal value function (UVF); a UVFA is then the corresponding approximation, ˜ Q ( s, a, w ) .
When we deﬁne transfer as above it becomes clear that in principle a sufﬁciently expressive UVFA can identify and exploit structure across the joint space S ×
A × R d .
In other words, a properly trained UVFA should be able to generalise across the space of tasks.
A different way of generalising across tasks is to use Barreto et al.’s
(2017) framework, which builds on assumption (1) and two core concepts: successor features (SFs) and generalised policy improvement (GPI).
The SFs ψ ∈ R d of a state-action pair ( s, a ) under policy π are given by
[Equation]
SFs allow one to immediately compute the value of a policy π on any task w : it is easy to show that, when (1) holds, Q π w ( s, a ) =
ψ π ( s, a ) (cid:62) w .
It is also easy to see that SFs satisfy a Bellman equation in which φ play the role of rewards, so ψ can be learned using any RL method (Szepesv ´ ari, 2010).
GPI is a generalisation of the policy improvement step described in Section 2.1.
The difference is that in GPI the improved policy is computed based on a set of value functions rather than on a single one.
Suppose that the agent has learned the SFs
ψ π
i of policies π 1 , π 2 , ..., π n .
When exposed to a new task deﬁned by w , the agent can immediately compute Q π i w ( s, a ) =
ψ π i ( s, a ) (cid:62) w .
Let the
GPI policy be deﬁned as π ( s ) ∈ argmax a Q max ( s, a ) , where Q max = max i Q π i .
The GPI theorem states that Q π ( s, a ) ≥ Q max ( s, a ) for all ( s, a ) ∈ S × A .
The result also extends to the scenario where we replace Q π i with approximations
˜ Q π i (Barreto et al., 2017).
U NIVERSAL S UCCESSOR F EATURES A PPROXIMATORS
UVFAs and SF&GPI address the transfer problem described in Section 2.2 in quite different ways.
With UVFAs, one trains an approximator ˜ Q ( s, a, w ) by solving the training tasks w ∈ M using any RL algorithm of choice.
One can then generalise to a new task by plugging its description w (cid:48) into ˜ Q and then acting according to the policy π ( s ) ∈ argmax a ˜ Q ( s, a, w (cid:48) ) .
With SF&GPI one solves each task w ∈ M and computes an approximation of the SFs of the resulting policies π w , ˜ ψ π w ( s, a ) ≈ ψ π w ( s, a ) .
The way to generalise to a new task w (cid:48) is to use the GPI policy deﬁned as π
[Equation]
The algorithmic differences between UVFAs and SF&GPI reﬂect the fact that these approaches exploit distinct properties of the transfer problem.
UVFAs aim at generalising across the space of tasks by exploiting structure in the function Q ∗ ( s, a, w ) .
In practice, such strategy materialises in the choice of function approximator, which carries assumptions about the shape of Q ∗ ( s, a, w ) .
For example, by using a neural network to represent ˜ Q ( s, a, w ) one is implicitly assuming that
Q ∗ ( s, a, w ) is smooth in the space of tasks; roughly speaking, this means that small perturbations to w will result in small changes in Q ∗ ( s, a, w ) .
In contrast, SF&GPI’s strategy to generalise across tasks is to exploit structure in the RL problem itself.
GPI builds on the general fact that a greedy policy with respect to a value function will in general perform better than the policy that originated the value function.
SFs, in turn, exploit the structure (1) to make it possible to quickly evaluate policies across tasks—and thus to apply GPI in an efﬁcient way.
The difference between the types of generalisation promoted by UVFAs and
SF&GPI is perhaps even clearer when we note that the latter is completely agnostic to the way the approximations ˜ ψ
π w are represented, and in fact it can applied even with a tabular representation.
Obviously, both UVFAs and GPI have advantages and limitations.
In order to illustrate this point, consider two tasks w and w (cid:48) that are “similar”, in the sense that || w − w (cid:48)
|| is small ( || · || is a norm in R d ).
Suppose that we have trained a UVFA on task w and as a result we obtained a good approximation ˜ Q ( s, a, w ) ≈ Q ∗ ( s, a, w ) .
If the structural assumptions underlying ˜ Q ( s, a, w ) hold— for example, ˜ Q ( s, a, w ) is smooth with respect to w —, it is likely that ˜ Q ( s, a, w (cid:48) ) will be a good approximation of Q ∗ ( s, a, w (cid:48) ) .
On the other hand, if such assumptions do not hold, we should not expect UVFA to perform well.
A sufﬁcient condition for SF&GPI to generalise well from task w to task w (cid:48) is that the policy π ( s ) ← argmax a ˜ ψ π w ( s, a ) (cid:62) w (cid:48) does well on task w (cid:48) , where π w is a solution for task w .
On the downside, SF&GPI will not exploit functional regularities at all, even if they do exist.
Let policy π w (cid:48) be a solution for tasks w (cid:48) .
In principle we cannot say anything about ψ π w (cid:48)
( s, a ) , the SFs of π w (cid:48) , even if we have a good approximation ˜ ψ π w ( s, a ) ≈ ψ π w ( s, a ) .
As one can see, the types of generalisation provided by UVFAs and SF&GPI are in some sense complementary.
It is then natural to ask if we can simultaneously have the two types of generalisation.
In this paper we propose a model that provides exactly that.
The main insight is actually simple: since
SFs are multi-dimensional value functions, we can extend them in the same way as universal value functions extend regular value functions.
In the next section we elaborate on how exactly to do so.
3. U NIVERSAL SUCCESSOR FEATURES
As discussed in Section 2.2, UVFs are an extension of standard value functions deﬁned as Q ∗ ( s, a, w ) .
If π w is one of the optimal policies of task w , we can rewrite the deﬁnition as Q π w ( s, a, w ) .
This makes it clear that the argument w plays two roles in the deﬁnition of a UVF: it determines both the task w and the policy π w (which will be optimal with respect to w ).
This does not have to be the case, though.
Similarly to Sutton et al.’s (2011) general value functions (GVFs), we could in principle deﬁne a function Q ( s, a, w , π ) that “disentangles” the task from the policy.
This would provide a model that is even more general than UVFs.
In this section we show one way to construct such a model when assumption (1) (approximately) holds.
Note that, when (1) is true, we can revisit the deﬁnition of SFs and write Q ( s, a, w , π )
=
ψ π ( s, a ) (cid:62) w .
If we want to be able to compute Q ( s, a, w , π ) for any π , we need SFs to span the space of policies π .
Thus, we deﬁne universal successor features as ψ ( s, a, π ) ≡ ψ π ( s, a ) .
Based on such deﬁnition, we call ˜ ψ ( s, a, π ) ≈ ψ ( s, a, π ) a universal successor features approximator (USFA).
In practice, when deﬁning a USFA we need to deﬁne a representation for the policies π .
A natural choice is to embed π onto R k .
Let e : ( S (cid:55)→ A ) (cid:55)→ R k be a policy-encoding mapping , that is, a function that turns policies π into vectors in R k .
We can then see USFs as a function of e ( π ) :
ψ ( s, a, e ( π )) .
The deﬁnition of the policy-encoding mapping e ( π ) can have a strong impact on the structure of the resulting USF.
We now point out a general equivalence between policies and reward functions that will provide a practical way of deﬁning e ( π ) .
It is well known that any reward function induces a set of optimal policies (Puterman, 1994).
A point that is perhaps less immediate is that the converse is also true.
Given a deterministic policy π , one can easily deﬁne a set of reward functions that induce this policy: for example, we can have r π ( s, π ( s ) , · ) = 0 and r π ( s, a, · ) = c , with c < 0 , for any a (cid:54) =
π ( s ) .
Therefore, we can use rewards to refer to deterministic policies and vice-versa (as long as potential ambiguities are removed when relevant).
Since here we are interested in reward functions of the form (1), if we restrict our attention to policies induced by tasks
z ∈ R
d we end up with a conveniently simple encoding function e ( π z ) =
z .
From this encoding function it follows that Q ( s, a, w , π z ) =
Q ( s, a, w , z ) .
It should be clear that UVFs are a particular case of this deﬁnition when w = z .
Going back to the deﬁnition of USFs, we can ﬁnally write Q ( s, a, w , z ) =
ψ ( s, a, z ) (cid:62) w .
Thus, if we learn a USF ψ ( s, a, z ) , we have a value function that generalises over both tasks and policies, as promised.
3. USFA GENERALISATION
We now revisit the question as to why USFAs should provide the beneﬁts associated with both UVFAs and SF&GPI.
We will discuss how exactly to train a USFA in the next section, but for now suppose that we have trained one such model ˜ ψ ( s, a, z ) using the training tasks in M .
It is then not difﬁcult to see that we can recover the solutions provided by both UVFAs and SF&GPI.
Given an unseen task w (cid:48) , let π be the GPI policy deﬁned as
[Equation]
where C ⊂ R d .
Clearly, if we make C = { w (cid:48) } , we get exactly the sort of generalisation associated with UVFAs.
On the other hand, setting C
= M essentially recovers SF&GPI.
The fact that we can recover both UVFAs and SF&GPI opens up a spectrum of possibilities in between the two.
For example, we could apply GPI over the training set augmented with the current task, C
=
M ∪ { w (cid:48) } .
In fact, USFAs allow us to apply GPI over any set of tasks C ⊂ R d .
The beneﬁts of this ﬂexibility are clear when we look at the theory supporting SF&GPI, as we do next.
Barreto et al.
(2017) provide theoretical guarantees on the performance of SF&GPI applied to any task w (cid:48)
∈ M (cid:48)
based on a ﬁxed set of SFs.
Below we state a slightly more general version of this result that highlights the two types of generalisation promoted by USFAs (proof in Barreto et al.’s (2017)
Theorem 2).
Proposition 1 Let w (cid:48) ∈ M (cid:48) and let Q π w (cid:48) be the action-value function of executing policy π on task w (cid:48) .
Given approximations { ˜ Q π z w (cid:48)
=
˜ ψ ( s, a, z ) (cid:62) w (cid:48) } z ∈C , let π be the GPI policy deﬁned in 3.
Then,
[Equation]
[Table]
(cid:124)
[Table]
  , (4) where Q ∗ w (cid:48) is the optimal value of task w (cid:48) , ψ π z are the SFs corresponding to the optimal policy for task z , and (cid:107)
f − g (cid:107) ∞ =
max s,a | f ( s, a ) − g ( s, a ) | .
When we write the result in this form, it becomes clear that, for each policy π z , the right-hand side of (4) involves two terms: i) δ d ( z ) , the distance between the task of interest w (cid:48) and the task that induced π z , and ii) δ ψ ( z ) , the quality of the approximation of the SFs associated with π z .
In order to get the tightest possible bound (4) we want to include in C the policy z that minimises δ d ( z ) + δ ψ ( z ) .
This is exactly where the ﬂexibility of choosing C provided by USFAs can come in handy.
Note that, if we choose C
=
M , we recover Barreto et al.’s
(2017) bound, which may have an irreducible min z ∈C δ d ( z ) even with a perfect approximation of the SFs in M .
On the other extreme, we could query our USFA at the test point C
= { w (cid:48) } .
This would result in δ d ( w (cid:48) )
= 0 , but can potentially incur a high cost due to the associated approximation error δ ψ ( w (cid:48) ) .
3. H OW TO TRAIN A USFA
Now that we have an approximation ˜ Q ( s, a, w , z ) a natural question is how to train this model.
In this section we show that the decoupled nature of ˜ Q is reﬂected in the training process, which assigns clearly distinct roles for tasks w and policies π z .
In our scenario, a transition at time t will be ( s t , a t , φ t +1 , s t +1 ) .
Note that φ allows us to compute the reward for any task w , and since our policies are encoded by z , transitions of this form allow us to learn the value function of any policy π z on any task w .
To see why this is so, let us deﬁne the temporal-difference (TD) error (Sutton and Barto, 1998) used in learning these value functions.
Given transitions in the form above, the n -step TD error associated with policy π z on task w will be
[Equation]
where a t + n = argmax b ˜ Q ( s t + n , b, z , z ) =
argmax b ˜ ψ ( s t + n , b, z ) (cid:62) z .
As is well known, the TD error δ t,n wz allows us to learn the value of policy π z on task w ; since here δ t,n wz is a function of z and w only, we can learn about any policy π z on any task w by just plugging in the appropriate vectors.
Equation (5) highlights some interesting (and subtle) aspects involved in training a USFA.
Since the value function ˜ Q ( s, a, w , z ) can be decoupled into two components, ˜ ψ ( s, a, z ) and w , the process of evaluating a policy on a task reduces to learning ˜ ψ
( s, a, z ) using the vector-based TD error δ t,n z showing up in (5).
Since δ t,n z is a function of z only, the updates to ˜ Q ( s, a, w , z ) will not depend on w .
How do the tasks w inﬂuence the training of a USFA, then?
If sample transitions are collected by a behaviour policy, as is usually the case in online RL, a natural choice is to have this policy be induced by a task w .
When this is the case the training tasks w ∈ M will deﬁne the distribution used to collect sample transitions.
Whenever we want to update ψ ( s, a, z ) for a different z than the one used in generating the data, we ﬁnd ourselves under the off-policy regime (Sutton and Barto, 1998).
Assuming that the behaviour policy is induced by the tasks w ∈ M , training a USFA involves two main decisions: how to sample tasks from M and how to sample policies π z to be trained through (5) or some variant.
As alluded to before, these decisions may have a big impact on the performance of the resulting USFA, and in particular on the trade-offs involved in the choice of the set of policies C used by the GPI policy (3).
As a form of illustration, Algorithm 1 shows a possible regime to train a
USFA based on particularly simple strategies to select tasks w ∈ M and to sample policies z ∈ R d .
One aspect of Algorithm 1 worth calling attention to is the fact that the distribution D z used to select policies can depend on the current task w .
This allows one to focus on speciﬁc regions of the policy space; for example, one can sample policies using a Gaussian distribution centred around w . Require:
, training tasks M , distribution D z over R d , number of policies n z do
, 2 , ..., n z do z i ∼ D z ( ·| w ) 6: if Bernoulli( (cid:15) )
=
1 then a ← Uniform( A ) 7: else a ← argmax b max i ˜ ψ
( s, b, z i ) (cid:62) w { GPI } 8:
Execute action a and observe φ and s (cid:48) 9: for i ← 1 , 2 , ..., n z do { Update ˜ ψ } 10: a (cid:48) ← argmax b ˜ ψ ( s, b, z i ) (cid:62) z i { a (cid:48) ≡ π i ( s (cid:48) ) } 11: θ α ←− (cid:104) φ + γ ˜ ψ
( s (cid:48) , a (cid:48) , z i ) − ˜ ψ
( s, a, z i ) (cid:105) ∇ θ ˜ ψ 12: s ← s (cid:48) 13: return θ
E XPERIMENTS
In this section we describe the experiments conducted to test the proposed architecture in a multitask setting and assess its ability to generalise to unseen tasks.
4. I LLUSTRATIVE EXAMPLE : T RIP MDP
We start with a simple illustrative example to provide intuition on the kinds of generalisation provided by UVFAs and SF&GPI.
We also show how in this example USFAs can effectively leverage both types of generalisation and outperform its precursors.
For this we will consider the simple two-state
MDP depicted in Figure 2.
To motivate the example, suppose that state s 1 of our MDP represents the arrival of a traveler to a new city.
The traveler likes coffee and food and wants to try what the new city has to offer.
In order to model that, we will use features φ ∈ R 2 , with φ 1 representing the quality of the coffee and φ 2 representing the quality of the food, both ranging from 0 to 1 .
The traveler has done some research and identiﬁed the places that serve the best coffee and the best food; in our MDP these places are modelled by terminal states associated with actions ‘ C ’ and ‘ F ’ whose respective associated rewards are φ ( · , C )
=
φ ( C ) =
[1 , 0] and φ ( F ) =
As one can infer from these feature vectors, the best coffee place does not serve food and the best restaurant does not serve coffee (at least not a very good one).
Nevertheless, there are other places in town that serve both; as before, we will model these places by actions P i associated with features φ ( P i ) .
We assume that (cid:107) φ ( P i ) (cid:107) 2 = 1 and consider N = 5 alternative places P i evenly spaced on the preference spectrum.
We model how much the traveler wants coffee and food on a given day by w ∈ R 2 .
If the traveler
[Figure]
happens to want only one of these (i.e. w ∈ { [1 , 0] , [0 , 1] } ), she can simply choose actions ‘ C ’ or ‘ F ’ and get a reward r = φ ( · ) (cid:62) w = 1 .
If instead she wants both coffee and food (i.e. if w is not an “one-hot” vector), it may actually be best to venture out to one of the other places.
Unfortunately, this requires the traveler to spend some time researching the area, which we model by an action ‘ E ’ associated with feature φ ( E ) =
[ − (cid:15), − (cid:15) ] .
After choosing ‘ E ’ the traveler lands on state s 2 and can now reach any place in town: C , F , P 1 , ..., P N .
Note that, depending on the vector of preferences w , it may be worth paying the cost of φ ( E ) (cid:62) w to subsequently get a reward of φ ( P i ) (cid:62) w (here γ = 1 ).
In order to assess the transfer ability of UVFAs, SF&GPI and USFAs, we deﬁne a training set
M = { 10 , 01 } and K = 50 test tasks corresponding to directions in the two-dimensional w -space:
M (cid:48)
= { w (cid:48) | w (cid:48) =
[cos( πk 2 K ) , sin( πk 2 K )] , k = 0 , 1 , ..., K } .
We start by analysing what SF&GPI would do in this scenario.
We focus on training task w C =
[1 , 0] , but an analogous reasoning applies to task w F
= [0 , 1] .
Let π C be the optimal policy associated with task w C .
It is easy to see that π ( s 1 ) =
π ( s 2 ) =
C .
Thus, under π C it should be clear that Q π C w (cid:48) ( s 1 , C ) >
Q π C w (cid:48)
( s 1 , E ) for all test tasks w (cid:48) .
Since the exact same reasoning applies to task w F if we replace action C with action F , the
GPI policy (3) computed over { ψ π C , ψ π F } will be suboptimal for most test tasks in M (cid:48) .
Training a
UVFA on the same set M , will not be perfect either due to the very limited number of training tasks.
Nonetheless the smoothness in the approximation allows for a slightly better generalisation in M (cid:48) .
Alternatively, we can use Algorithm 1 to train a USFA on the training set M .
In order to do so we sampled n z = 5 policies z ∈ R 2 using a uniformly random distribution D z ( ·| w ) =
U ([0 , 1] 2 ) (see line 5 in Algorithm 1).
When acting on the test tasks w (cid:48)
we considered two choices for the candidates set: C = { w (cid:48) } and C
= { z i |
z i ∼ U ([0 , 1] 2 ) , i = 1 , 2 , ..., 5 } .
Empirical results are provided in
Figure 2.
As a reference we report the performance of SF&GPI using the true { ψ π C , ψ π F } – no approximation.
We also show the learning curve of a UVFA.
As shown in the ﬁgure, USFA clearly outperforms its precursors and quickly achieves near optimal performance.
This is due to two factors.
First, contrary to vanilla SF&GPI, USFA can discover and exploit the rich structure in the policyspace, enjoying the same generalisation properties as UVFAs but now enhanced by the combination of the off-policy and off-task training regime.
Second, the ability to sample a candidate set C that induces some diversity in the policies considered by GPI overcomes the suboptimality associated with the training SFs ψ π C and ψ π F .
We explore this effect in a bit more detail in the suppl.
material.
L ARGE SCALE EXPERIMENTS
[Figure]
Environment and tasks .
We used the
DeepMind Lab platform to design a 3D environment consisting of one large room containing four types of objects:
TVs, balls, hats, and balloons (Beattie et al., 2016; Barreto et al., 2018).
A depiction of the environment through the eyes of the agent can be seen in Fig.
Features φ
i are indicator functions associated with object types, i.e. , φ i ( s, a, s (cid:48) )
=
if and only if the agent
[Figure]
UVFA (offpolicy)
USFA (GPI over C = M)
USFA (GPI over C =
M + {w'})
USFA (GPI over C
[Figure]
w [Figure] [Figure] collects an object of type i (say, a TV) on the transition s a −→ s (cid:48) .
A task is deﬁned by four real numbers w ∈ R 4 indicating the rewards attached to each object.
Note that these numbers can be negative, in which case the agent has to avoid the corresponding object type.
For instance, in task w =
[1-100] the agent is interested in objects of the ﬁrst type and should avoid objects of the second type.
Agent architecture .
A depiction of the architecture used for the USFA agent is illustrated in Fig.
The architecture has three main modules: i)
A fairly standard input processing unit composed of three convolution layers and an LSTM followed by a non-linearity (Schmidhuber, 1996); ii) A policy conditioning module that combines the state embedding coming from the ﬁrst module, s ≡ f ( h ) , and the policy embedding, z , and produces | A | · d outputs corresponding to the SFs of policy π z , ˜ ψ
( s, a, z ) ; and iii)
The evaluation module , which, given a task w and the SFs ˜ ψ ( s, a, z ) , will construct the evaluation of policy π z on w , ˜ Q ( s, a, w , z ) =
˜ ψ ( s, a, z ) (cid:62) w .
Training and baselines .
We trained the above architecture end-to-end using a variation of Alg.that uses Watkins’s (1989) Q ( λ ) to apply Q -learning with eligibility traces.
As for the distribution D z used in line 5 of Alg.
1 we adopted a Gaussian centred at w : z ∼ N ( w , 0 .
1 I ) , where I is the identity matrix.
We used the canonical vectors of R 4 as the training set, M = { 1000 , 0100 , 0010 , 0001 } .
Once an agent was trained on M we evaluated it on a separate set of unseen tasks, M (cid:48) , using the GPI policy (3) over different sets of policies C .
Speciﬁcally, we used: C = { w (cid:48) } , which corresponds to a
UVFA with an architecture specialised to (1); C = M , which corresponds to doing GPI on the SFs of the training policies (similar to (Barreto et al., 2017)), and C =
M ∪ { w (cid:48) } , which is a combination of the previous two.
We also included as baselines two standard UVFAs that do not take advantage of the structure (1); one of them was trained on-policy and the other one was trained off-policy (see supplement).
The evaluation on the test tasks M (cid:48) was done by “freezing” the agent at different stages of the learning process and using the GPI policy (3) to select actions.
To collect and process the data we used an asynchronous scheme similar to IMPALA (Espeholt et al., 2018).
4. R ESULTS AND DISCUSSION
Fig.
4 shows the results of the agents after being trained on M .
One thing that immediately stands out in the ﬁgure is the fact that all architectures generalise quite well to the test tasks.
This is a surprisingly good result when we consider the difﬁculty of the scenario considered: recall that the agents are solving the test tasks without any learning taking place .
This performance is even more
[Figure]
impressive when we note that some test tasks contain negative rewards,
something never experienced by the agents during training.
When we look at the relative performance of the agents, it is clear that USFAs perform considerably better than the unstructured UVFAs.
This is true even for the case where C = { w (cid:48) } , in which USFAs essentially reduce to a structured UVFA that was trained by decoupling tasks and policies.
The fact that USFAs outperform UVFAs in the scenario considered here is not particularly surprising, since the former exploit the structure (1) while the latter cannot.
In any case, it is reassuring to see that our model can indeed exploit such a structure effectively.
This result also illustrates a particular way of turning prior knowledge about a problem into a favourable inductive bias in the UVFA architecture
It is also interesting to see how the different instantiations of USFAs compare against each other.
As shown in Fig.
4, there is a clear advantage in including M to the set of policies C used in GPI (3).
This suggests that, in the speciﬁc instantiation of this problem, the type of generalisation provided by SF&
GPI is more effective than that associated with UVFAs.
One result that may seem counter-intuitive at ﬁrst is the fact that USFAs with C =
M + { w (cid:48) } sometimes perform worse than their counterparts using C = M , especially on tasks with negative rewards.
Here we note two points.
First, although including more tasks to C results in stronger guarantees for the GPI policy, strictly speaking there are no guarantees that the resulting policy will perform better (see Barreto et al.’s
Theorem 1, 2017).
Another explanation very likely in the current scenario is that errors in the approximations ˜ ψ ( s, a, z ) may have a negative impact on the resulting GPI policy (3).
As argued previously, adding a point to
C can sometimes increase the upper bound in (4), if the approximation at this point is not reliable.
On the other hand, comparing USFA’s results using C
=
M + { w (cid:48) } and C = { w (cid:48) } , we see that by combining the generalisation of UVFAs and GPI we can boost the performance of a model that only relies on one of them.
This highlights the ﬁne balance between the two error terms in (4) and emphasizes how critical selecting low-error candidates in C can be.
In the above scenario, SF&GPI on the training set M seems to provide a more effective way of generalising, as compared to UVFAs, even when the latter has a structure specialised to (1).
Nevertheless, with less conservative choices of D z that provide a greater coverage of the z space we expect the structured UVFA ( C = { w (cid:48) } ) to generalise better.
Note that this can be done without changing M and is not possible with conventional UVFAs.
One of the strengths of USFAs is exactly that: by disentangling tasks and policies, one can learn about the latter without ever having to actually try them out in the environment.
We exploit this possibility to repeat our experiments now using
D z = N ( w , 0 .
5 I ) .
Results are shown in Fig.5.
As expected, the generalisation of the structured
UVFA improves considerably, almost matching that of GPI.
This shows that USFAs can operate in two regimes: i) with limited coverage of the policy space, GPI over M will provide a reliable generalisation; ii) with a broader coverage of the space structured UVFAs will do increasingly better.
R ELATED W ORK
Multitask RL is an important topic that has generated a large body of literature.
Solutions to this problem can result in better performance on the training set (Espeholt et al., 2018), can improve data efﬁciency (Teh et al., 2017) and enable generalisation to new tasks.
For a comprehensive presentation of the subject please see Taylor and Stone (2009) and Lazaric (2012) and references therein.
There exist various techniques that incorporate tasks directly into the deﬁnition of the value function for multitask learning (Kaelbling, 1993; Ashar, 1994; Sutton et al., 2011).
UVFAs have been used for zero-shot generalisation to combinations of tasks (Mankowitz et al., 2018; Hermann et al., 2017), or to learn a set of ﬁctitious goals previously encountered by the agent (Andrychowicz et al., 2017).
Many recent multitask methods have been developed for learning subtasks or skills for a hierarchical controller (Vezhnevets et al., 2017;
Andreas et al., 2016;
Oh et al., 2017).
In this context, Devin et al.
(2017) and Heess et al.
(2016) proposed reusing and composing sub-networks that are shared across tasks and agents in order to achieve generalisation to unseen conﬁgurations.
Finn et al.
(2017) uses meta-learning to acquire skills that can be ﬁne-tuned effectively.
Sequential learning and how to retain previously learned skills has been the focus of a number of investigations (Kirkpatrick et al., 2016; Rusu et al., 2016).
All of these works aim to train an agent (or a sub-module) to generalise across many subtasks.
All of these can be great use-cases for USFAs.
USFAs use a UVFA to estimate SFs over multiple policies.
The main reason to do so is to apply GPI, which provides a superior zero-shot policy in an unseen task.
There have been previous attempts to combine SFs and neural networks, but none of them used GPI
(Kulkarni et al., 2016; Zhang et al., 2016).
Recently, Ma et al.
(2018) have also considered combining SFs and UVFAs.
They propose building a goal-conditioned policy that aims to generalise over a collection of goals. In their work, the SFs are trained to track this policy and only used to build the critic employed in training the goal-conditioned policy.
Thus, they are considering the extrapolation in π ( s, g ) and using the SFs as an aid in training.
Moreover, as both the training and SFs and π ( s, g ) are done on-policy, the proposed system has only seen instances where the SFs, critic and policy are all conditioned on the same goal.
In contrast, in this work we argue and show the beneﬁts of decoupling the task and policy to enable generalisation via GPI when appropriate, while preserving the ability to exploit the structure in the policy space.
We use the SFs as a way to factorize and exploit effectively the structure in value function space.
And we will use these evaluations directly to inform our action selection
C ONCLUSION
In this paper we presented USFAs, a generalisation of UVFAs through SFs.
The combination of
USFAs and GPI results in a powerful model capable of exploiting the same types of regularity exploited by its precursors: structure in the value function, like UVFAs, and structure in the problem itself, like SF&GPI.
This means that USFAs can not only recover their precursors but also provide a whole new spectrum of possible models in between them.
We described the choices involved in training and evaluating a USFA and discussed the trade-offs associated with these alternatives.
To make the discussion concrete, we presented two examples aimed to illustrate different regimes of operation.
The ﬁrst example embodies a MDP where the generalisation in the optimal policy space is fairly easy but the number of optimal policies we would want to represent can be large.
This is a scenario where UVFAs would strive, while vanilla SF&GPI will struggle due to the large number of policies needed to build a good GPI policy.
In this case, we show that USFAs can leverage the sort of parametric generalisation provided by UVFAs and even improve on it, due to its decoupled training regime and the use of GPI in areas where the approximation is not quite perfect.
Our second example is in some sense a reciprocal one, where we know from previous work that the generalisation provided via GPI can be very effective even on a small set of policies, while generalising in the space of optimal policies, like UVFAs do, seems to require a lot more data.
Here we show that USFAs can recover the type of generalisation provided by SFs when appropriate.
This example also highlights some of the complexities involved in training at scale and shows how USFAs are readily applicable to this scenario.
Overall, we believe USFAs are a powerful model that can exploit the available structure effectively: i) the structure induced by the shared dynamics (via SFs), ii) the structure in the policy space (like UVFAs) and ﬁnally iii) the structure in the RL problem itself (via GPI), and could potentially be useful across a wide range of RL applications that exhibit these properties
A I LLUSTRATIVE EXAMPLE : T RIP MDP
In this section we provide some additional analysis and results omitted from the main text.
As a reminder, this is a two state MDP, where the ﬁrst state is a root state, the transition from s 1 → scomes at a cost r w ( s 1 , E ) =
φ ( s 1 , E ) T w = − (cid:15) ( w 1 + w 2 ) and all other actions lead to a ﬁnal positive reward corresponding to how much the resulting state/restaurant alligns with our preferences (our task) right now.
For convenience, we provide below the depiction of the Trip MDP introduced in
Section 4.1.
Arrived in new city
Explored neighborhood
-ε
-ε .
S ₁
S ₂ E
C F
In the experiments run we considered a ﬁxed set of training tasks M = { 01 , 10 } for all methods.
The set of outcomes from the exploratory state s 2 is deﬁned as φ ( s 2 , a ) =
[cos( θ ) , sin( θ )] for θ ∈ { kπ/ 2 N } k =0 ,N .
Note that this includes the binary states for k = 0 and respectively k
= N .
We ran this MDP with N = 6 , and (cid:15) = 0 .
Thus outside the binary outcomes, the agent can select
N − 1 = 5 other mixed outcomes and, as argued in the main text, under these conditions there will be a selection of the w -space in which each of these outcomes will be optimal.
Thus the space of optimal policies, we hope to recover, is generally N + 1 .
Nevertheless, there is a lot of structure in this space, that the functional approximators can uncover and employ in their generalization.
A. A DDITIONAL RESULTS
In the main paper, we reported the zero-shot aggregated performance over all direction M (cid:48)
= { w (cid:48) | w (cid:48) =
[cos( πk 2 K ) , sin( πk 2 K )] , k ∈ Z K } .
This should cover most of the space of tasks/trade-offs we would be interest in.
In this section we include the generalization for other sets M (cid:48) .
First in Fig.we depict the performance of the algorithms considered across the whole w (cid:48) space M (cid:48) =
Fig.
7 is just a different visualization of the previous plot, where we focus on how far these algorithms are from recovering the optimal performance.
This also shows the subtle effect mentioned in the discussion in the main text, induced by the choice of C in the USFA evaluation.
Perfomance: UVFA 0.0 0.2 0.4 0.6 0.8 1.0 0.
Perfomance: SF&GPI 0.0 0.2 0.4 0.6 0.8 1.0 0.
1.0 Perfomance: USFA C=random(5) 0.0 0.2 0.4 0.
1.0 Perfomance: USFA C={w} 0.000 0.
Optimal Perfomance
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Optimality Gap: SF&GPI 0.0 0.2 0.4 0.6 0.8 1.0 0.
1.0 Optimality Gap: USFA C=random(5) 0.
Optimality Gap: USFA C={w}
A particularly adversarial choice of test tasks for the vanilla SF&GPIwould be the diagonal in the [0 , 1] 2 quadrant depicted in the plot above: M (cid:48)
= { w (cid:48) | w (cid:48) 1 = w (cid:48) 2 , w 1 ∈ [0 , 1] } .
This is, in a sense, maximally away from the training tasks and both of the precursor models are bound to struggle in this portion of the space.
This intuition was indeed empirically validated.
Results are provided in Fig.
As mentioned above, this is an adversarial evaluation, mainly to point out that, in general, there might be regions of the space were the generalization of the previous models can be very bad, but where the combination of them can still recover close to optimal performance
[Figure]
B L ARGE SCALE EXPERIMENTS : D ETAILS B. A GENT ’ S ARCHITECTURE
This section contains a detailed description of the USFA agent used in our experimental section.
As a reminder, we include the agent’s architecture below (Figure 1 in the main text).
As highlighted in Section 4.2, our agent comprises of three main modules:
• Input processing module : computes a state representation f
( h t ) from observation o t .
This module is made up of three convolutional layers (structure identical to the one used in (Mnih et al., 2015)), the output of which then serves as input to a LSTM (256).
This LSTM takes as input the previously executed action a t − 1 .
The output of the LSTM is passed through a non-linearity f (chosen here to be a ReLu) to produce a vector of 128 units, f ( h t ) .
• Policy conditioning module: compute the SFs ˜ ψ
( s, a, z ) , given a (sampled) policy embedding z and the state representation f
( h t ) .
This module ﬁrst produces n z number of z ∼ D z samples ( n z = 30 in our experiments).
Each of these is then transformed via a 2-layer MLP(32,32) to produce a vector of size 32 , for each sample z .
This vector g ( z ) gets concatenated with the state representation f
( h t ) and the resulting vector is further processed by a 2-layer MLP that produces a tensor of dimensions d × |A| for each z , where d = dim ( φ ) .
These correspond to SFs ˜
ψ
( s, a, z ) for policy π z .
Note that this computation can be done quite efﬁciently by reusing the state embedding f ( h t ) , doing the downstream computation in parallel for each policy embedding z .
• Task evaluation module: computes the value function Q ( s, a, z , w ) =
˜ ψ ( s, a, z ) T w for a given task description w .
This module does not have any parameters as the value functions are simply composable from ˜ ψ ( s,
a, z ) and the task description w via assumption (1).
This module with output n z value functions that will be used to produce a behavior via GPI.
An important decision in this design was how and where to introduce the conditioning on the policy.
In all experiments shown here the conditioning was done simply by concatenating the two embeddings w and z , although stronger conditioning via an inner product was tried yielding similar performance.
The ’where’ on the other hand is much more important.
As the conditioning on the policy happens quite late in the network, most of the processing (up to f ( h t ) ) can be done only once, and we can sample multiple z and compute the corresponding ˜ ψ π z at a fairly low computational cost.
As mentioned above, these will be combined with the task vector w to produce the candidate action value functions for GPI.
Note that this helps both in training and in acting, as otherwise the unroll of the LSTM would be policy conditioned, making the computation of the SFs and the off-policy n -step learning quite expensive.
UVFA baseline agents have a similar architecture, but now the task description w is fed in as an input to the network.
The conditioning on the task of UVFAs is done in a similar fashion as we did the conditioning on the policies in USFAs, to make the computational power and capacity comparable.
The input processing module is the same and now downstream, instead of conditioning on the policy embedding z , we condition on task description w .
This conditioning if followed by a 2-layer MLP that computes the value functions ˜ Q ∗ ( s, a, w ) , which induces the greedy policy π ( UV FA ) w = arg max a ˜ Q ∗ ( s, a, w ) .
B. A GENT ’ S TRAINING
The agents’ training was carried out using the IMPALA architecture (Espeholt et al., 2018).
On the learner side, we adopted a simpliﬁed version of IMPALA that uses Q ( λ ) as the RL algorithm.
In our experiments, for all agents we used λ = 0 . 9 .
Depending on the sampling distribution D z , in learning we will be often off-policy.
That is, most of the time, we are going to learn about a policy π z 1 and update its corresponding SFs approximations ˜ ψ
( s, a, z 1 ) , using data generated by acting in the environment according to some other policy π z 2 .
In order to account for this off-policiness, whenever computing the n-step return required in eq.
5, we are going to cut traces whenever the policies start to disagree and bootstrap from this step on (Sutton and Barto, 1998).
Here we can see how the data distribution induce by the choice of training tasks M can inﬂuence the training process.
If the data distribution D z is very close to the set M , as in our ﬁrst experiment, most of the policies we are going to sample will be close to the policies that generated the data.
This means that we might be able to make use of longer trajectories in this data, as the policies will rarely disagree.
On the other hand, by staying close to the training tasks, we might hurt our ability to generalise in the policy space, as our ﬁrst experiment suggest (see Figure 4).
By having a broader distribution D z = N ( w , 0 .
5 I ) , we can learn about more diverse policies in this space, but we will also increase our off-policiness.
We can see from Figure 5, that our algorithm can successfully learn and operate in both of these regimes.
For the distributed collection of data we used 50 actors per task.
Each actor gathered trajectories of length 32 that were then added to the common queue.
The collection of data followed an (cid:15) -greedy policy with a ﬁxed (cid:15)
= 0 .
The training curves shown in the paper correspond to the performance of the the (cid:15) -greedy policy (that is, they include exploratory actions of the agents).
B. A GENT ’ S EVALUATION
All agents were evaluated in the same fashion.
During the training process, periodically (every 20M frames) we will evaluate the agents performance on a test of held out test tasks.
We take these intermediate snapshots of our agents and ’freeze’ their parameters to assess zero-shot generalisation.
Once a test task w (cid:48) is provided, the agent interacts with the environment for 20 episodes, one minute each and the average (undiscounted) reward is recorded.
These produce the evaluation curves in
Figure 4.
Evaluations are done with a small (cid:15) = 0 .
001 , following a GPI policy with different instantiations of C .
For the pure UVFA agents, the evaluation is similar: (cid:15) -greedy on the produced value functions ˜ Q ∗ ( s, a, w ) , with the same evaluation (cid:15) = 0 .
C A DDITIONAL RESULTS
In our experiments we deﬁned a set of easy test tasks (close to M ) and a set of harder tasks, in order to cover reasonably well a few distinct scenarios:
• Testing generalisation to tasks very similar to the training set, e.g. w (cid:48)
=
• Testing generalisation to harder tasks with different reward proﬁles: only positive rewards, only negative rewards, and mixed rewards.
In the main text, we included only a selection of these for illustrative purposes.
Here we present the full results.
C. C ANONICAL BASIS : Z ERO - SHOT GENERALISATION
This section contains the complete results of the ﬁrst experiment conducted.
As a reminder, in this experiment we were training a USFA agent on M = { 1000 , 0100 , 0010 , 0001 } , with D z =
N ( w , 0 .
1 I )
and compare its performance with two conventional UVFA agents (one trained on-policy and the other one using all the data generated to learn off-policy) on a range of unseen test tasks.
Complete set of result is included below, as follows:
Figure 10 includes results on easy tasks, close to the tasks contained in the training set M (generalisation to those should be fairly straightforward);
Figure 11 and Figure 12 present results on more challenging tasks, quite far away from the training set, testing out agents ability to generate to the whole 4D hypercube.
[Figure] [Figure]
(Part 1)
[Figure]
(Part
C. C ANONICAL BASIS : USFA S IN DIFFERENT TRAINING REGIMES .
In this section
[Figure]
we include the omitted results from our second experiment.
As a reminder, in this experiment we were training two USFA agents on the same set of canonical tasks
[Figure]
but employing different distributions D z , one will low variance σ = 0 .
1 , focusing in learning policies around the training set
M
[Figure]
and another one with larger variance σ = 0 .
5 , that will try to learn about a lot more policies away from the training set, thus potentially facilitating the generalisation provided by the
UVFA component.
Results are displayed in Figures 13-14 on all tasks in the hard evaluation set
[Figure]
of two USFA agent trained on the canonical training set M = { 1000 , 0100 , 0010 , 0001 } , with D z = N ( w , 0 .
1 I ) , and
D z = N ( w , 0 .
5 I )
[Figure] [Figure]
C. L ARGER COLLECTION OF TRAINING TASKS
We also trained our USFA agent on a larger set of training tasks that include the previous canonical tasks, as well as four other tasks that contain both positive and negative reward M = { 1000, 0100, 0010, 0001, 1-100, 01-10, 001-1, -1000 } .
Thus we expect this agent to generalises better as a result of its training.
A selection of these results and sample performance in training are included in Fig.
[Figure]