X I -L EARNING : S UCCESSOR F EATURE T RANSFER L EARNING FOR G ENERAL R EWARD F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks.
Successor features (SF) are a prominent transfer mechanism in domains where the reward function changes between tasks.
They reevaluate the expected return of previously learned policies in a new target task and to transfer their knowledge.
A limiting factor of the SF framework is its assumption that rewards linearly decompose into successor features and a reward weight vector.
We propose a novel SF mechanism, ξlearning, based on learning the cumulative discounted probability of successor features.
Crucially, ξ -learning allows to reevaluate the expected return of policies for general reward functions.
We introduce two ξ -learning variations, prove its convergence, and provide a guarantee on its transfer performance.
Experimental evaluations based on ξ -learning with function approximation demonstrate the prominent advantage of ξ -learning over available mechanisms not only for general reward functions, but also in the case of linearly decomposable reward functions.
I NTRODUCTION
Reinforcement Learning (RL) successfully addressed many complex problems such as playing computer games, chess, and even Go with superhuman performance (Mnih et al., 2015; Silver et al., 2018).
These impressive results are possible thanks to a vast amount of interactions of the RL agent with its environment/task.
Such strategy is unsuitable in settings where the agent has to perform and learn at the same time.
Consider, for example, a care giver robot in a hospital that has to learn a new task, such as a new route to deliver meals.
In such a setting, the agent can not collect a vast amount of training samples but has to adapt quickly instead.
Transfer learning aims to provide mechanisms quickly to adapt agents in such settings (Taylor and Stone, 2009; Lazaric, 2012; Zhu et al., 2020).
The rationale is to use knowledge from previously encountered source tasks for a new target task to improve the learning performance on the target task.
The previous knowledge can help reducing the amount of interactions required to learn the new optimal behavior.
For example, the care giver robot could reuse knowledge about the layout of the hospital it learned in previous source tasks (e.g. guiding a person) to learn to deliver meals.
The Successor Feature (SF) and General Policy Improvement (GPI) framework (Barreto et al., 2020) is a prominent transfer learning mechanism for tasks where only the reward function differs.
Its basic premise is that the rewards which the RL agent tries to maximize are deﬁned based on a lowdimensional feature descriptor φ ∈ R n .
For our care-giver robot this could be ID’s of beds or rooms that it is visiting, in difference to its high-dimensional visual state input from a camera.
The rewards are then computed not based on its visual input but on the ID’s of the beds or rooms that it visits.
The expected cumulative discounted successor features ( ψ ) are learned for each behavior that the robot learned in the past.
It represents the dynamics in the feature space that the agent experiences for a behavior.
This corresponds to the rooms or beds the care-giver agent would visit if using the behavior.
This representation of feature dynamics is independent from the reward function.
A behavior learned in a previous task and described by this SF representation can be directly re-evaluated for a different reward function.
In a new task, i.e. for a new reward function, the GPI procedure re-evaluates the behaviors learned in previous tasks for it.
It then selects at each state the behavior of a previous task if it improves the expected reward.
This allows to reuse behaviors learned in previous source tasks
Source code at https://tinyurl.com/3xuzxff  for a new target task.
A similar transfer strategy can also be observed in the behavior of humans (Momennejad et al., 2017; Momennejad, 2020; Tomov et al., 2021) .
The classical SF&GPI framework (Barreto et al., 2017; 2018) makes the assumption that rewards r are a linear composition of the features φ ∈ R n via a reward weight vector w i ∈ R n that depends on the task i : r i = φ (cid:62) w i .
This assumption allows to effectively separate the feature dynamics of a behavior from the rewards and thus to re-evaluate previous behaviors given a new reward function, i.e. a new weight vector w j .
Nonetheless, this assumption also restricts successful application of SF&GPI only to problems where such a linear decomposition is possible.
We investigate the application of the
SF&GPI framework to general reward functions: r i =
R i ( φ ) over the feature space.
We propose to learn the cumulative discounted probability over the successor features, named ξ -function, and refer to the proposed framework as ξ -learning.
Our work is related to Janner et al.
Touati and
Ollivier (2021), and brings two important additional contributions.
First, we provide mathematical proof of the convergence of ξ -learning.
Second, we demonstrate how ξ -learning can be used for meta-RL, using the ξ -function to re-evaluate behaviors learned in previous tasks for a new reward function R j .
Furthermore, ξ -learning can also be used to transfer knowledge to new tasks using GPI.
The contribution of our paper is three-fold:
• We introduce a new RL algorithm, ξ -learning, based on a cumulative discounted probability of successor features, and two variants of its update operator.
• We provide theoretical proofs of the convergence of ξ -learning to the optimal policy and for a guarantee of its transfer learning performance under the GPI procedure.
• We experimentally compare ξ -learning in tasks with linear and general reward functions, and for tasks with discrete and continuous features to standard Q-learning and the classical SF framework, demonstrating the interest and advantage of ξ -learning.
B ACKGROUND 2. R EINFORCEMENT L EARNING
RL
investigates algorithms to solve multi-step decision problems, aiming to maximize the sum over future rewards (Sutton and Barto, 2018).
RL problems are modeled as Markov Decision
Processes (MDPs) which are deﬁned as a tuple M ≡ ( S , A , p, R, γ ) , where S and A are the state and action set.
An agent transitions from a state s t to another state s t +1 using action a t at time point t collecting a reward r t : s t a t ,r t −−−→ s t +1 .
This process is stochastic and the transition probability p ( s t +1 | s t , a t ) describes which state s t +1 is reached.
The reward function R deﬁnes the scalar reward r t =
R ( s t , a t , s t +1 )
∈ R for the transition.
The goal in an MDP is to maximize the expected return
G t = E (cid:2)(cid:80) ∞ k =0 γ k R t + k (cid:3) , where R t =
R ( S t , A t , S t +1 ) .
The discount factor γ ∈ [0 , 1) weights collected rewards by discounting future rewards stronger.
RL provides algorithms to learn a policy π : S
→ A deﬁning which action to take in which state to maximise G t .
Value-based RL methods use the concept of value functions to learn the optimal policy.
The stateaction value function, called Q-function, is deﬁned as the expected future return taking action a t in s t and then following policy π :
[Equation]
The Q-function can be recursively deﬁned following the Bellman equation
such that the current
Q-value Q π ( s t , a t ) depends on the maximum Q-value of the next state Q π ( s t +1 , a t +1 ) .
The optimal policy for an MDP can then be expressed based on the Q-function, by taking at every step the maximum action: π ∗ ( s ) ∈ argmax a Q ∗ ( s, a ) .
The optimal Q-function can be learned using a temporal difference method such as Q-learning (Watkins and Dayan, 1992).
Given a transition ( s t , a t , r t , s t +1 ), the Q-value is updated according to:
[Equation]
where α k ∈ (0 , 1] is the learning rate at iteration k .
2. T RANSFER L EARNING AND THE SF&GPI F RAMEWORK
We are interested in the transfer learning setting where the agent has to solve a set of tasks
M = { M 1 , M 2 , . . .
, M m } , that in our case differ only in their reward function.
The Successor Feature (SF) framework provides a principled way to perform transfer learning (Barreto et al., 2017; 2018).
SF assumes that the reward function can be decomposed into a linear combination of features φ ∈ Φ ⊂ R n and a reward weight vector w i ∈ R n that is deﬁned for a task M i :
[Equation]
We refer to such reward functions as linear reward functions .
Since the various tasks differ only in their reward functions, the features are the same for all tasks in M .
Given the decomposition above, it is also possible to rewrite the Q-function into an expected discounted sum over future features ψ π i ( s, a ) and the reward weight vector w i :
[Equation]
This decouples the dynamics of the policy
π i in the feature space of the MDP from the expected rewards for such features.
Thus, it is now possible to evaluate the policy
π i in a different task M j using a simple multiplication of the weight vector w j with the ψ -function: Q π i j ( s, a ) =
ψ π i ( s, a ) (cid:62) w j .
Interestingly, the ψ function also follows the Bellman equation:
[Equation]
and can therefore be learned with conventional RL methods.
Moreover, (Lehnert and Littman, 2019) showed the equivalence of SF-learning to Q-learning.
Being in a new task M j the Generalized Policy Improvement (GPI) can be used to select the action over all policies learned so far that behaves best:
[Equation]
(Barreto et al., 2018) proved that under the appropriate conditions for optimal policy approximates, the policy constructed in (6) is close to the optimal one, and their difference is upper-bounded:
[Equation]
where (cid:107)
f − g (cid:107) ∞ =
max s,a | f ( s, a ) − g ( s, a ) | .
For an arbitrary reward function r the result can be interpreted in the following manner.
Given the arbitrary task M , we identify the theoretically closest possible linear reward task M i with r i .
For this theoretically closest task, we search the linear task
M j in our set of task M (from which we also construct the GPI optimal policy (6)) which is closest to it.
The upper bound between Q ∗ and Q is then deﬁned by 1) the difference between task M and the theoretically closest possible linear task M i : || r − r
i || ∞ ; and by 2) the difference between theoretical task M i and the closest task M j : min j || r i − r j || ∞ .
If our new task M is also linear then r = r
i and the ﬁrst term in (7) would vanish.
Very importantly, this result shows that the SF framework will only provide a good approximation of the true Q-function if the reward function in a task can be represented using a linear decomposition.
If this is not the case then the error in the approximation increases with the distance between the true reward function r and the best linear approximation of it r i as stated by || r − r
i || ∞ .
M ETHOD : ξ - LEARNING 3. D EFINITION AND FOUNDATIONS OF ξ - LEARNING
The goal of this paper is to investigate the application of SF&GPI to tasks with general reward functions R : Φ (cid:55)→ R over state features φ ∈ Φ :
[Equation]
where we deﬁne φ t ≡ φ ( s t , a t , s t +1 ) .
Under this assumption the Q-function can not be linearly decomposed into a part that describes feature dynamics and one that describes the rewards as in the linear SF framework (4).
To overcome this issue, we propose to deﬁne the expected cumulative discounted probability of successor features or ξ -function, which is going to be the central mathematical object of the paper, as:
[Equation]
where p ( φ t + k = φ | s t = s, a t = a ; π ) , or in short p ( φ t + k = φ
| s t , a t ; π ) , is the probability density function of the features at time t + k , following policy π and conditioned to s and a being the state and action at time t respectively.
Note that ξ π depends not only on the policy π but also on the state transition (constant through the paper).
With the deﬁnition of the ξ -function, the Q-function rewrites (this is compatible with SFQL in the linear reward case, see Appendix A.6):
[Equation]
Depending on the reward function R , there are several ξ -functions that correspond to the same Q function.
Formally, this is an equivalence relationship, and the quotient space has a one-to-one correspondence with the Q -function space.
Proposition 1.
(Equivalence between functions ξ and Q) Let Q =
Q : S ×
A → R s.t.
(cid:107) Q (cid:107) ∞ < ∞} .
Let ∼ be deﬁned as ξ 1 ∼ ξ 2 ⇔ (cid:82) Φ Rξ 1 = (cid:82) Φ Rξ 2 .
Then, ∼ is an equivalence relationship, and there is a bijective correspondence between the quotient space Ξ ∼ and Q .
Corollary 1.
The bijection between Ξ ∼ and Q allows to induce a norm (cid:107) · (cid:107) ∼ into Ξ ∼ from the supremum norm in Q , with which Ξ ∼ is a Banach space (since Q is Banach with (cid:107) · (cid:107) ∞ ):
[Equation]
Similar to the Bellman equation for the Q-function, we can deﬁne a Bellman operator for the ξ -function, denoted by T ξ , as:
[Equation]
As in the case of the Q -function, we can use T ξ to construct a contractive operator:
Proposition 2.
( ξ -learning has a ﬁxed point)
The operator T ξ is well-deﬁned w.r.t.
the equivalence ∼ , and therefore induces an operator T ∼ deﬁned over Ξ ∼ .
T ∼ is contractive w.r.t.
(cid:107) · (cid:107) ∼ .
Since Ξ ∼ is Banach, T ∼ has a unique ﬁxed point and iterating T ∼ starting anywhere converges to that point.
In other words, successive applications of the operator T ∼ converge towards the class of optimal ξ functions [ ξ ∗ ] or equivalently to an optimal ξ function deﬁned up to an additive function k satisfying (cid:82) Φ k ( s, a, φ ) R ( φ ) d φ = 0 , ∀ ( s, a ) ∈ S × A (i.e. k ∈ Ker ( [Equation] → (cid:82) Φ Rξ ) ).
While these two results state (see Appendix A for the proofs) the theoretical links to standard
Q-learning formulations, the T ξ operator deﬁned in (12) is not usable in practice, because of the expectation.
In the next section, we deﬁne the optimisation iterate, prove its convergence, and provide two variants to perform the ξ updates.
3. ξ - LEARNING ALGORITHMS
In order to learn the ξ -function, we introduce the ξ -learning update operator, which is an offpolicy temporal difference method analogous to Q-learning.
Given a transition ( s t , a t , s t +1 , φ t ) the ξ -learning update operator is deﬁned as: [Equation] where ¯ a t +1 = argmax [Equation]
(cid:82)
[Equation]
The following is one of the main results of the manuscript, stating the convergence of ξ -learning:
Theorem 1.
(Convergence of ξ -learning)
For a sequence of state-action-feature { s t , a t , s t +1 , φ t } ∞ t = consider the ξ -learning update given in (13).
If the sequence of state-action-feature triples visits each state, action inﬁnitely often, and if the learning rate α k is an adapted sequence satisfying the
Robbins-Monro conditions:
[Equation]
then the sequence of function classes corresponding to the iterates converges to the optimum, which corresponds to the optimal Q-function to which standard Q-learning updates would converge to:
[Equation]
The proof is provided in Appendix A and follows the same ﬂow as for Q-learning.
The previous theorem provides convergence guarantees under the assumption that either p ( φ t = φ | s t , a t ; π ) is known, or an unbiased estimate can be constructed.
We propose two different ways to approximate p ( φ t = φ | s t , a t ; π ) from a given transition ( s t , a t , s t +1 , φ t ) so as to perform the ξ -update (13).
The ﬁrst instance is a model-free version and detailed in the following section.
A second instance uses a one-step SF model, called One-Step Model-based (MB) ξ -learning , which is further described in Sec.
B.
Model-free (MF) ξ -Learning: MF ξ -learning uses the same principle as standard model-free temporal difference learning methods.
The update assumes for a given transition ( s t , a t , s t +1 , φ t ) that the probability for the observed feature is p ( φ = φ t | s t , a t ) = 1 .
Whereas for all other features ( ∀ φ (cid:48) ∈ Φ , φ (cid:48) (cid:54) =
φ t ) the probability is p ( φ (cid:48) =
φ t | s t , a t ) = 0 , see Appendix D for continuous features.
The resulting updates are:
[Equation]
Due to the stochastic update of the ξ -function and if the learning rate α ∈ (0 , 1] discounts over time, the ξ -update will learn the true probability of p ( φ = φ t | s t , a t ) .
A potential problem with the MF procedure is that it might induce a high variance when the true feature probabilities are not binary.
3. M ETA ξ - LEARNING
After discussing ξ -learning on a single task and showing its theoretical convergence, we can now investigate how it can be applied in transfer learning.
Similar to the linear SF framework the ξ -function allows to reevaluate a policy learned for task M i , ξ π i , in a new environment M j :
[Equation]
This allows us to apply GPI in (6) for arbitrary reward functions in a similar manner to what was proposed for linear reward functions in (Barreto et al., 2018).
We extend the GPI result to the ξ -learning framework as follows:
Theorem 2.
(Generalised policy improvement in ξ -learning)
Let M be the set of tasks, each one associated to a (possibly different) weighting function R i ∈
L 1 (Φ) .
Let ξ π ∗ i be a representative of the optimal class of ξ -functions for task
M i , i ∈ { 1 , . . .
, I } , and let ˜ ξ π i be an approximation to the optimal ξ -function, (cid:107) ξ π ∗ i − ˜ ξ π i (cid:107) R i ≤ ε, ∀ i .
Then, for another task M with weighting function R , the policy deﬁned as:
[Equation]
satisﬁes:
[Equation]
where (cid:107) f
(cid:107) g
= sup s,a (cid:82) Φ | f · g | d φ .
The proof is provided in Appendix A.
E XPERIMENTS
We evaluated ξ -learning in two environments.
The ﬁrst has discrete features.
It is a modiﬁed version of the object collection task by Barreto et al.
(2017) having more complex features to allow general reward functions.
See Appendix E.1 for experimental results in the original environment.
The second environment, the racer environment, evaluates the agents in tasks with continuous features.
4. D ISCRETE F EATURES - O BJECT C OLLECTION E NVIRONMENT
Environment: The environment consist of 4 rooms (Fig. 1 - a).
The agent starts an episode in position S and has to learn to reach the goal position G. During an episode, the agent can collect objects to gain further rewards.
Each object has 2 properties: 1) color: orange or blue, and 2) form: box or triangle.
The state space is a high-dimensional vector s ∈ R 112 .
It encodes the agent’s position using a 10 × 10 grid of two-dimensional Gaussian radial basis functions.
Moreover, it includes a memory about which object has been already collected.
Agents can move in 4 directions.
The features φ ∈ Φ = { 0 , 1 } 5 are binary vectors.
The ﬁrst 2 dimensions encode if an orange or a blue object was picked up.
The 2 following dimensions encode the form.
The last dimension encodes if the agent reached goal G.
For example, φ (cid:62) =
[1 , 0 , 1 , 0 , 0] encodes that the agent picked up an orange box.
Tasks:
Each agent learns sequentially 300 tasks which differ in their reward for collecting objects.
We compared agents in two settings: either in tasks with linear or general reward functions.
For each linear task M i , the rewards r = φ (cid:62) w i are deﬁned by a linear combination of features and a weight vector w i ∈ R 5 .
The weights w i,k for the ﬁrst 4 dimensions deﬁne the rewards for collecting an object with a speciﬁc property.
They are randomly sampled from a uniform distribution: w i,k ∼ U ( − 1 , 1) .
The ﬁnal weight deﬁnes the reward for reaching the goal position which is w i, 5 = 1 for each task.
The general reward functions are sampled by assigning a different reward to each possible combination of object properties φ j ∈
Φ using uniform sampling: R i ( φ j ) ∼ U ( − 1 , 1) , such that picking up an orange box might result in a reward of R i ( φ (cid:62) =
= 0 .
[Figure]
T o t a l
R e t u r n R a ti o
[Figure]
Agents: We compared ξ -learning to Q-learning (QL), and classical SF Q-learning (SFQL) (Barreto et al., 2017).
All agents use function approximation for their state-action functions (Q, ψ , or ξfunction).
An independent linear mapping is used to map the values from the state for each of theactions.
As the features are discrete, the ξ -function and ˆ p φ -model are approximated by an independent mapping for each action and possible feature φ ∈ Φ .
The Q-value Q ( s, a ) for the ξ -agents (Eq. 10) is computed by:
Q π ( s, a ) = (cid:80) φ ∈ Φ R ( φ ) ξ π ( s, a, φ ) .
The reward functions of each task are given to the ξ -agents.
For SFQL, the sampled reward weights w i were given in tasks with linear reward functions.
For general reward functions, a linear model r = φ (cid:62) ˜ w i approximating the rewards was learned for each task and its weights ˜ w i given to SFQL (see Appendix C.3 for details).
Each tasks was executed for 20 , 000 steps, and the average performance over 10 runs per algorithm was measured.
We performed a grid-search over the parameters of each agent, reporting here the performance of the parameters with the highest total reward over all tasks.
Results: ξ -learning outperformed SFQL and QL for tasks with linear and general reward functions (Fig.
1 - b; d).
MF showed a slight advantage over MB ξ -learning in both settings.
We further studied the effect non-linearity of general reward functions on the performance of classical SF compared to ξ -learning by evaluating them in tasks with different levels of non-linearity.
We sampled general reward functions that resulted in different levels of mean absolute model error if they are linearly approximated with min ˜ w | r ( φ ) − φ (cid:62) ˜ w | .
We trained SFQL and MF ξ -learning in each of these conditions on 300 tasks and measured the ratio between the total return of SFQL and MF ξ (Fig. 1).
The relative performance of SFQL compared to MF ξ reduces with higher non-linearity of the reward functions.
For reward functions that are nearly linear (mean error of 0 . 125 ), both have a similar performance.
Whereas, for reward functions that are difﬁcult to model with a linear relation (mean error of 1 . 625 ) SFQL reaches only less than 50% of the performance of ξ -learning.
This follows
SFQL’s theoretical limitation in (7) and shows the advantage of ξ learning over SFQL in non-linear reward tasks.
4. C ONTINUOUS F EATURES - R ACER E NVIRONMENT
Environment and Tasks: We further evaluated the agents in an environment with continuous features (Fig. 2 - a).
The agent is randomly placed in the environment and has to drive around fortimesteps before the episode ends.
Similar to a car, the agent has an orientation and momentum, so that it can only drive straight, or in a right or left curve.
The agent reappears on the opposite side if it exits one side.
The distance to 3 markers are provided as features φ ∈ R 3 .
Rewards depend on the distances r = (cid:80) 3 k =
r k φ k , where each component r k has 1 or 2 preferred distances deﬁned by
Gaussian functions.
For each of the 37 tasks, the number of Gaussians and their properties ( µ , σ ) are randomly sampled for each feature dimension.
Fig.
2 (a) shows a reward function with dark areas depicting higher rewards.
The agent has to learn to drive around in such a way as to maximize its trajectory over positions with high rewards.
The state space is a high-dimensional vector s ∈
Rencoding the agent’s position and orientation.
As before, the 2D position is encoded using a 10 ×grid of two-dimensional Gaussian radial basis functions.
Similarly, the orientation is also encoded using 20 Gaussian radial basis functions.
[Figure]
Agents: We introduce a MF ξ -agent for continuous features (CMF ξ ) (Appendix D.2.3).
CMF ξ discretizes each feature dimension φ k ∈ [0 , 1] in 11 bins with the bin centers:
X = { 0 . 0 , 0 . 1 , . . .
It learns for each dimension k and bin i the ξ -value ξ πk ( s, a, X i ) .
Q-values (Eq. 10) are computed by: Q π ( s, a ) = (cid:80) 3 k =1 (cid:80) 11 i =1 r k ( X i ) ξ πk ( s, a, X i ) .
SFQL learns ψ for the continuous, non-discretized feature space.
It received an approximated weight vector ˜ w i that was trained before the task started on several uniformly sampled features and rewards.
Results: ξ -learning reached the highest performance of all agents (Fig.
2 - b) outperforming QL and SFQL.
SFQL reaches only a low performance below QL, because it is not able to sufﬁciently well approximate the general reward functions with its linear reward model.
This shows the advantage of ξ -learning over SFQL in environments with general reward functions.
D ISCUSSION
Performance of ξ -learning compared to classical SF&GPI: ξ -learning allows to disentangle the dynamics of policies in the feature space of a task from the associated reward, see (10).
The experimental evaluation in tasks with general reward functions (Fig. 1 - d, and Fig. 2) shows that ξ -learning can therefore successfully apply GPI to transfer knowledge from learned tasks to new ones.
Given a general reward function it can re-evaluate successfully learned policies for knowledge transfer.
Instead, classical SFQL based on a linear decomposition (3) can not be directly applied given a general reward function.
In this case a linear approximation has to be learned which shows inferior performance to ξ -learning that directly uses the true reward function.
ξ -learning also shows an increased performance over SFQL in environments with linear reward functions (Fig.
1 - a).
This effect can not be attributed to differences in their computation of a policy’s expected return as both are correct (Appendix A.6).
A possible explanation is that ξ -learning reduces the complexity for the function approximation of the ξ -function compared to the ψ -function in SFQL.
Continuous Feature Spaces: For tasks with continuous features (racer environment), ξ -learning used successfully a discretization of each feature dimension, and learned the ξ -values independently for each dimension.
This strategy is viable for reward functions that are cumulative over the feature dimensions: r ( φ )
= (cid:80) k
r k φ k .
The Q-value can be computed by summing over the independent dimensions and the bins X : Q π ( s, a ) = (cid:80)
k (cid:80) x
∈ X r k ( x ) ξ π ( s, a, x ) .
For more general reward functions, the space of all feature combinations would need to be discretized, which grows exponentially with each new dimension.
As a solution the ξ -function could be directly deﬁned over the continuous feature space, but this yields some problems.
First, the computation of the expected return requires an integral Q ( s, a )
= (cid:82) φ ∈ Φ R ( φ ) ξ ( s, a, φ ) over features instead of a sum, which is a priori intractable.
Second, the representation and training of the ξ -function, which would be deﬁned over a continuum thus increasing the difﬁculty of approximating the function.
Janner et al.
(2020) and Touati and
Ollivier (2021) propose methods that might allow to represent a continuous ξ -function, but it is unclear if they converge and if they can be used for transfer learning.
Learning of Features: In principle, classical SF&GPI can also optimize general reward functions if features and reward weights are learned.
This is possible if the learned features describe the non-linear effects in the reward functions.
Nonetheless, learning of features adds further challenges and shows to reduce performance.
Barreto et al.
(2017) learns features from observations sampled from several tasks before the SF&GPI starts.
Therefore, novel non-linearities potentially introduced at later tasks are not well represent by the learned features.
If instead features are learned alongside the SF&GPI procedure, the problem on how to coordinate both learning processes needs to be investigated.
Importantly, ψ -functions for older tasks would become unusable for the GPI procedure on newer task, because the feature representation changed between them.
Moreover, our replication (Sec. E.1) of the object collection task from Barreto et al.
(2017) shows the performance of learned features is below the performance of given features.
MF Xi reaches a ﬁnal average reward per task of 850 with given features and reward functions.
The best performance of SFQL with learned features only reaches a ﬁnal performance of 575 (Fig. 2 in (Barreto et al., 2017)).
In summary, if features and reward functions are known then ξ -learning outperforms SFQL.
And using given features and reward functions is natural for many applications as these are often known, for example in robotic tasks where they are usually manually designed (Akalin and Loutﬁ, 2021).
Computational Complexity: The improved performance of SFQL and ξ -learning over QL in the transfer learning setting comes at the cost of an increased computational complexity.
The GPI procedure (6) of both approaches requires to evaluate at each step the ψ π i -function or ξ π i -function over all previous experienced tasks in M .
As a consequence, the computational complexity increases linearly with each new environment that is added.
A solution is to apply GPI only over a subset of learned policies.
Nonetheless, an open question is still how to optimally select this subset.
R ELATED WORK
Transfer Learning: Transfer methods in RL can be generally categorized according to the type of tasks between which transfer is possible and the type of transferred knowledge (Taylor and Stone, 2009; Lazaric, 2012; Zhu et al., 2020).
In the case of SF&GPI which ξ -learning is part of, tasks only differ in their reward functions.
The type of knowledge that is transferred are policies learned in source tasks which are re-evaluated in the target task and recombined using the GPI procedure.
A natural use-case for ξ -learning are continual problems (Khetarpal et al., 2020) where an agent has continually adapt to changing tasks, which are in our setting different reward functions.
Successor Features: SF are based on the concept of successor representations (Dayan, 1993;
Momennejad, 2020).
Successor representations predict the future occurrence of all states for a policy in the same manner as SF for features.
Their application is restricted to low-dimensional state spaces using tabular representations.
SF extended them to domains with high-dimensional state spaces (Kulkarni et al., 2016;
Zhang et al., 2017; Barreto et al., 2017; 2018), by predicting the future occurrence of low-dimensional features that are relevant to deﬁne the return.
Several extensions to the SF framework have been proposed.
One direction aims to learn appropriate features from data such as by optimally reconstruct rewards (Barreto et al., 2017), using the concept of mutual information (Hansen et al., 2019), or the grouping of temporal similar states (Madjiheurem and
Toni, 2019).
Another direction is the generalization of the ψ -function over policies (Borsa et al., 2018) analogous to universal value function approximation (Schaul et al., 2015).
Similar approaches use successor maps (Madarasz, 2019), goal-conditioned policies (Ma et al., 2020), or successor feature sets (Brantley et al., 2021).
Other directions include their application to POMDPs (Vértes and Sahani, 2019), combination with max-entropy principles (Vertes, 2020), or hierarchical RL (Barreto et al., 2021).
In difference to ξ -learning all these approaches build on the assumption of linear reward functions, whereas ξ -learning allows the SF&GPI framework to be used with general reward functions.
Nonetheless, most of the extensions for linear SF can be combined with ξ -learning.
Model-based RL
: SF represent the dynamics of a policy in the feature space that is decoupled from the rewards allowing to reevaluate them under different reward functions.
It shares therefore similar properties with model-based RL (Lehnert and Littman, 2019).
In general, model-based RL methods learn a one-step model of the environment dynamics p ( s t +1 | s t , a t ) .
Given a policy and an arbitrary reward function, rollouts can be performed using the learned model to evaluate the return.
In practice, the rollouts have a high variance for long-term predictions rendering them ineffective.
Recently, (Janner et al., 2020) proposed the γ -model framework that learns to represent ξ -values in continuous domains.
Nonetheless, the application to transfer learning is not discussed and no convergence is proven as for ξ -learning.
This is the same case for the forward-backward MPD representation proposed in Touati and Ollivier (2021).
(Tang et al., 2021) also proposes to decouple the dynamics in the state space from the rewards, but learn an internal representation of the rewards.
This does not allow to reevaluate an policy to a new reward function without relearning the mapping.
C ONCLUSION
The introduced ξ -learning framework learns the expected cumulative discounted probability of successor features which disentangles the dynamics of a policy in the feature space of a task from the expected rewards.
This allows ξ -learning to reevaluate the expected return of learned policies for general reward functions and to use it for transfer learning utilizing GPI.
We proved that ξ -learning converges to the optimal policy, and showed experimentally its improved performance over Q-learning and the classical SF framework for tasks with linear and general reward functions.
E THICS S TATEMENT
ξ -learning and its associated optimization algorithms represent general RL procedures similar to
Q-learning.
Their potential negative societal impact depends on their application domains which range over all possible societal areas in a similar manner as for other general RL procedures.
Beyond the topic of the paper, we did our best to cite the relevant literature and to fairly compare with previous ideas, concepts and methods.
To that aim, all agents are trained and evaluated within the same software environment, and under the very same experimental settings.
R EPRODUCIBILITY S TATEMENT
In order to ensure high changes of reproducibility we provided lots of details of the method and experiments associated to the paper.
In particular, we have provided the proofs for all mathematical results announced in the main paper (see Appendix A).
These constitute the theoretical foundation of the proposed ξ -learning methodology.
Secondly, we have provided all experimental details (methods, and environments) required for reproducing our experiments, namely: appendix C for the object collection and D for the racer environment respectively.
In addition, we provide additional results in appendix E, to completely illustrate the interest of the proposed method.
Finally, we provided an anonymous link to the source code, so that reviewers can run it if necessary.
A T HEORETICAL P ROOFS A. P ROOF OF P ROPOSITION
Let us start by recalling the original statement in the main paper.
Proposition 1.
(Equivalence between functions ξ and Q) Let Q =
Q : S ×
A → R s.t.
(cid:107) Q (cid:107) ∞ < ∞} .
Let ∼ be deﬁned as ξ 1 ∼ ξ 2 ⇔ (cid:82) Φ Rξ 1 = (cid:82) Φ Rξ 2 .
Then, ∼ is an equivalence relationship, and there is a bijective correspondence between the quotient space Ξ ∼ and Q .
Proof.
We will proof the statements sequentially.
∼ is an equivalence relationship: To prove this we need to demonstrate that ∼ is symmetric, reciprocal and transitive.
The three are quite straightforward since: ξ ∼ ξ, ∀ ξ , ξ ∼ η ⇔ η ∼ ξ, ∀ ξ, η and ξ ∼ η, η ∼ ν ⇒ ξ ∼ ν .
Bijective correspondence:
To prove the bijectivity, we will ﬁrst prove that it is injective, then surjective.
Regarding the injectivity: [ ξ ] (cid:54) =
[ η ] ⇒ Q ξ (cid:54) =
Q η , we prove it by contrapositive:
[Equation]
In order to prove the surjectivity, we start from a function Q ∈ Q and select an arbitrary ξ ∈ Ξ , then the following function:
[Equation]
(21) satisﬁes that ξ Q ∈ Ξ and that (cid:82) Φ R ( φ ) ξ Q ( s, a, φ ) d φ = Q ( s, a ) , ∀ ( s, a ) ∈ S × A .
We conclude that there is a bijective correspondence between the elements of Ξ ∼ and of Q .
A. P ROOF OF C OROLLARY
Let us recall the result:
Corollary 1.
The bijection between Ξ ∼ and Q allows to induce a norm (cid:107) · (cid:107) ∼ into Ξ ∼ from the supremum norm in Q , with which Ξ ∼ is a Banach space (since Q is Banach with (cid:107) · (cid:107) ∞ ):
[Equation]
Proof.
The norm induced in the quotient space is deﬁned from the correspondence between Ξ ∼ and
Q and is naturally deﬁned as in the previous equation.
The norm is well deﬁned since it does not depend on the class representative.
Therefore, all the metric properties are transferred, and Ξ ∼ is immediately Banach with the norm (cid:107) · (cid:107) ∼ .
A. P ROOF OF P ROPOSITION
Let’s restate the result:
Proposition 2.
( ξ -learning has a ﬁxed point)
The operator T ξ is well-deﬁned w.r.t.
the equivalence ∼ , and therefore induces an operator T ∼ deﬁned over Ξ ∼ .
T ∼ is contractive w.r.t.
(cid:107) · (cid:107) ∼ .
Since Ξ ∼ is Banach, T ∼ has a unique ﬁxed point and iterating T ∼ starting anywhere converges to that point.
Proof.
We prove the statements above one by one:
The operator T ∼ is well deﬁned: Let us ﬁrst recall the deﬁnition of the operator T ξ in (12), where we removed the dependency on π for simplicity:
[Equation]
Let ξ 1 , ξ 2 ∈ [ ξ ] two different representatives of class [ ξ ] , we can write:
[Equation]
because ξ 1 , ξ 2 ∈ [ ξ ] .
Therefore the operator T ∼ ([ ξ ]) =
T ξ ( ξ ) is well deﬁned in the quotient space, since the image of class does not depend on the function chosen to represent the class.
Contractive operator T ∼ : The contractiveness of T ∼ can be proven directly:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The contractiveness of T ∼ can also be understood as being inherited from the standard Bellmann operator on Q .
Indeed, given a ξ function, one can easily see that applying the standard Bellman operator to the Q function corresponding to ξ leads to the Q function corresponding to T ∼ ([ ξ ]) .
Fixed point of T ∼ : To conclude the proof, we use the fact that any contractive operator on a Banach space, in our case: T ∼ : Ξ ∼ →
Ξ ∼ has a unique ﬁxed point [ ξ ∗ ] , and that for any starting point [ ξ 0 ] , the sequence [ ξ n ] = T ∼ ([ ξ n − 1 ]) converges to [ ξ ∗ ] w.r.t.
to the corresponding norm (cid:107) [ ξ ] (cid:107) ∼ .
A. P ROOF OF T HEOREM
These two propositions will be useful to prove that the ξ learning iterates converge in Ξ ∼ .
Let us restate the deﬁnition of the operator from (13):
[Equation]
and the theoretical result:
Theorem 1.
(Convergence of ξ -learning)
For a sequence of state-action-feature { s t , a t , s t +1 , φ t } ∞ t = consider the ξ -learning update given in (13).
If the sequence of state-action-feature triples visits each state, action inﬁnitely often, and if the learning rate α k is an adapted sequence satisfying the
Robbins-Monro conditions:
[Equation]
then the sequence of function classes corresponding to the iterates converges to the optimum, which corresponds to the optimal Q-function to which standard Q-learning updates would converge to:
[Equation]
Proof.
The proof re-uses the ﬂow of the proof used for Q-learning (Tsitsiklis, 1994).
Indeed, we rewrite the operator above as:
[Equation]
with ε deﬁned as:
[Equation]
Obviously ε satisﬁes E { ε } = 0 , which, together with the contractiveness of T ∼ , is sufﬁcient to demonstrate the convergence of the iterative procedure as done for Q-learning.
In our case, the optimal function ξ ∗ is deﬁned up to an additive kernel function κ ∈ Ker.
The correspondence with the optimal Q learning function is a direct application of the correspondence between the ξ - and
Q-learning problems.
A. P ROOF OF T HEOREM
Let us restate the result.
Theorem 2.
(Generalised policy improvement in ξ -learning)
Let M be the set of tasks, each one associated to a (possibly different) weighting function R i ∈
L 1 (Φ) .
Let ξ π ∗ i be a representative of the optimal class of ξ -functions for task
M i , i ∈ { 1 , . . .
, I } , and let ˜ ξ π i be an approximation to the optimal ξ -function, (cid:107) ξ π ∗ i − ˜ ξ π i (cid:107) R i ≤ ε, ∀ i .
Then, for another task M with weighting function R , the policy deﬁned as:
[Equation]
satisﬁes:
[Equation]
where (cid:107) f
[Equation]
= sup
[Equation]
Proof.
The proof is stated in two steps.
First, we exploit the proof of Proposition 1 of (Barreto et al., 2017), and in particular (13) that states:
[Equation]
where Q ∗ and Q π are the Q-functions associated to the optimal and π policies in the environment
R .
The conditions on the Q functions required in the original proposition are satisﬁed because the ξ -functions satisfy them, and there is an isometry between Q and ξ functions.
Because the above inequality is true for all training environments i , we can rewrite as:
[Equation] [Equation]
We now realise that, in the case of ξ -learning, the reward functions rewrite as:
[Equation]
and therefore we have:
[Equation]
Similarly, due to the isometry between ξ and Q-learning, i.e. Proposition 2, we can write that:
[Equation]
which proves the generalised policy improvement for ξ -learning.
A. R ELATION BETWEEN CLASSICAL SF AND ξ - LEARNING FOR LINEAR REWARD FUNCTIONS
In the case of linear reward functions, i.e. where assumption (3) holds, it is possible to show that ξlearning can be reduced to classical SF.
Classical SF represents therefore a speciﬁc case of ξ -learning under this assumption.
Theorem 3.
(Equality of classical SF and ξ -learning for linear reward functions)
Given the assumption that reward functions are linearily decomposable with
[Equation]
where φ ∈ R
n are features for a transition and w i ∈ R n are the reward weight vector of task m i ∈ M , then the classical SF and ξ -learning framework are equivalent.
Proof.
We start with the deﬁnition of the Q-value according to ξ -learning from (10).
After replacing the reward function R i with our linear assumption, the deﬁnition of the Q-function according to classical SF with the ψ -function can be recovered:
[Equation]
Please note, although both methods are equal in terms of their computed values, how these are represented and learned differs between them.
Thus, it is possible to see a performance difference of the methods in the experimental results where ξ -learning outperforms SFQL in our environments.
B O NE -S TEP SF M ODEL - BASED (MB) ξ -L EARNING
Besides the MF ξ -learning update operator (16), we introduce a second ξ -learning procedure called
One-step SF Model-based (MB) ξ -Learning that attempts to reduce the variance of the update.
To do so, MB ξ -learning estimates the distribution over the successor features over time.
Let ˜ p
( φ t = φ | s t , a t ; π ) denote the current estimate of the feature distribution.
Given a transition ( s t , a t , s t +1 , φ t ) the model is updated according to:
[Equation]
where β ∈ [0 , 1] is the learning rate.
After updating the model ˜ p φ , it can be used for the ξ -update as deﬁned in (13).
Since the learned model ˜ p φ is independent from the reward function and from the policy, it can be learned and used over all tasks.
C E XPERIMENTAL D ETAILS : O BJECT C OLLECTION E NVIRONMENT
The object collection environment (Fig. 1 - a) was brieﬂy introduced in Section 4.1.
This section provides a formal description.
C. E NVIRONMENT
The environment is a continuous two-dimensional area in which the agent moves.
The position of the agent is a point in the 2D space: ( x, y ) ∈ [0 , 1] 2 .
The action space of the agent consists of four movement directions:
A = { up, down, left, right } .
Each action changes the position of the agent in a certain direction and is stochastic by adding a Gaussian noise.
For example, the action for going right updates the position according to x t +1 = x t + N ( µ = 0 .
05 , σ = 0 .
If the new position ends in a wall (black areas in Fig.
1 - a) that have a width of 0 . 04 ) or outside the environment, the agent is set back to its current position.
Each environment has 12 objects.
Each object has two properties with two possible values: color (orange, blue) and shape (box, triangle).
If the agent reaches an object, it collects the object which then disappears.
The objects occupy a circular area with radius 0 . 04 .
At the beginning of an episode the agent starts at location S with ( x, y ) S = (0 . 05 , 0 . 05) .
An episode ends if the agent reaches the goal area G which is at position ( x, y )
G =
(0 . 86 , 0 . 86) and has a circular shape with radius 0 .
After an episode the agent is reset to the start position S and all collected objects reappear.
The state space of the agents consist of their position in the environment and the information about which objects they already collected during an episode.
Following (Barreto et al., 2017), the position is encoded using a radial basis function approach.
This upscales the agent’s ( x, y ) position to a high-dimensional vector s pos ∈ R 100 providing a better signal for the function approximation of the different functions such as the ψ or ξ -function.
The vector s pos is composed of the activation of two-dimensional Gaussian functions based on the agents position ( x, y ) :
[Equation]
where c j ∈ R 2 is the center of the j th Gaussian.
The centers are laid out on a regular 10 × 10 grid over the area of the environment.
The state also encodes the memory about the objects that the agent has already collected using a binary vector s mem ∈ { 0 , 1 } 12 .
The j th dimension encodes if the j th object has been taken ( s mem ,j = 1 ) or not ( s mem ,j = 0 ).
An additional constant term was added to the state to aid the function approximation.
As a result, the state received by the agents is a column vector with s =
[ s (cid:62) pos , s (cid:62) mem , 1] (cid:62) ∈ R 113 .
The features φ ( s t , a t , s t +1 ) ∈ { 0 , 1 } 5 in the environment describe the type of object that was collected by an agent during a step or if it reached the goal position.
The ﬁrst four feature dimensions encode binary the properties of a collected object and the last dimension if the goal area was reached.
In total | Φ | = 6 possible features exists: φ 1 = [0 , 0 , 0 , 0 , 0] (cid:62) - standard observation, φ 2 = [1 , 0 , 1 , 0 , 0] (cid:62) - collected an orange box, φ 3 =
[1 , 0 , 0 , 1 , 0] (cid:62) - collected an orange triangle, φ 4 = [0 , 1 , 1 , 0 , 0] (cid:62) - collected a blue box, φ 5 = [0 ,
Two types of tasks were evaluated in this environment that have either 1) linear or 2) general reward functions.
300 tasks, i.e. reward functions, were sampled for each type.
For linear tasks, the rewards r = φ (cid:62) w i are deﬁned by a linear combination of discrete features φ ∈ N 5 and a weight vector w i ∈ R 5 .
The ﬁrst four dimensions in w i deﬁne the reward that the agent receives for collecting objects having speciﬁc properties, e.g. being blue or being a box.
The weights for each of the four dimensions are randomly sampled from a uniform distribution: w k ∈ [1 , 2 , 3 , 4] ∼ U ( − 1 , 1) for each task.
The ﬁnal weight deﬁnes the reward for reaching the goal state which is w 5 = 1 for each task.
For training agents in general reward tasks, general reward functions R i for each task M i were sampled.
These reward functions deﬁne for each of the four features ( φ 2 , . . .
, φ 5 ) that represent the collection of a speciﬁc object type an individual reward.
Their rewards were sampled from a uniform distribution: R i ( φ k ∈{ 2 ,..., 5 } ) ∼ U ( − 1 , 1) .
The reward for collecting no object is R i ( φ 1 )
= 0 and for reaching the goal area is R i ( φ 6 )
= 1 for all tasks.
Reward functions of this form can not be linearly decomposed in features and a weight vector.
C. A LGORITHMS
This section introduces the details of each evaluated algorithm.
First the common elements are discussed before introducing their speciﬁc implementations.
All agents experience the tasks M ∈ M of an environment sequentially.
They are informed when a new task starts.
All algorithms receive the features φ ( s, a, s (cid:48) ) of the environment.
For the action selection and exploration, all agents use a (cid:15) -greedy strategy.
With probability (cid:15) ∈ [0 , 1] the agent performs a random action.
Otherwise it selects the action that maximizes the expected return.
As the state space ( s ∈ R 113 ) of the environments is high-dimensional and continuous, all agents use an approximation of their respective functions such as for the Q-function ( ˜ Q ( s, a ) ≈ Q ( s, a ) ) or the ξ -function ( ˜ ξ ( s, a, φ ) ≈ ξ ( s, a, φ ) ).
We describe the general function approximation procedure on the example of ξ -functions.
If not otherwise mentioned, all functions are approximated by a single linear mapping from the states to the function values.
The parameters θ ξ ∈ R 113 ×|A|×| Φ | of the mapping have independent components θ ξa,φ ∈ R 113 for each action a ∈ A and feature φ ∈ Φ :
[Equation]
To learn ˜ ξ we update the parameters θ ξ using stochastic gradient descent following the gradients ∇ θ ξ L ξ ( θ ξ ) of the loss based on the ξ -learning update (13):
[Equation]
ξ ( s t +1 , ¯ a t +1 , φ ; ¯ θ ξ ) − ˜ ξ ( s t , a t , φ ; θ ξ ) (cid:17) 2 (cid:27) with ¯ a t +1 = argmax a (cid:88) φ ∈
Φ
R ( φ )˜ ξ ( s t +1 , a, φ ; ¯ θ ξ ) , (36) where ¯ θ ξ = θ ξ but ¯ θ ξ is treated as a constant for the purpose of calculating the gradients ∇ θ ξ L ξ ( θ ξ ) .
We used PyTorch 2 for the computation of gradients and its stochastic gradient decent procedure (SGD) for updating the parameters.
C.2. QL
The Q-learning (QL) agent (Algorithm 1) represents standard Q-learning (Watkins and Dayan, 1992).
The Q-function is approximated and updated using the following loss after each observed transition:
[Equation]
(37) where ¯ θ Q = θ Q but ¯ θ Q is treated as a constant for the purpose of optimization, i.e no gradients ﬂow through it.
Following (Barreto et al., 2017) the parameters θ Q are reinitialized for each new task.
C.2. SFQL
The classical successor feature algorithm (SFQL) is based on a linear decomposition of rewards in features and reward weights (Barreto et al., 2017) (Algorithm 2).
If the agent is learning the reward weights ˜ w i for a task M i then they are randomly initialized at the beginning of a task.
For the case of general reward functions and where the reward weights are given to the agents, the weights are learned to approximate a linear reward function before the task.
See Section C.3 for a description of the training procedure.
After each transition the weights are updated by minimizing the error between the predicted rewards φ ( s t , a t , s t +1 ) (cid:62) ˜ w i and the observed reward r t :
[Equation]
Algorithm
Input : exploration rate: (cid:15) learning rate for the Q-function: α for i ← 1 to num_tasks do initialize ˜ Q : θ Q ← small random initial values new_episode ← true for t ← 1 to num_steps do if new_episode then new_episode ← false s t ← initial state
With probability (cid:15) select a random action a t , otherwise a t ← argmax a ˜ Q ( s t , a )
Take action a t and observe reward r t and next state s t + if s t +1 is a terminal state then new_episode ← true γ t ←else γ t ← γ y ← r t + γ t max a t +1 ˜ Q
( s t +
[Equation]
, a t +1 )
Update θ Q using SGD( α ) with L Q =
( y − ˜ Q ( s t , a t ))s t ← s t +
SFQL learns an approximated ˜ ψ i -function for each task M i .
The parameters of the ˜ ψ -function for the ﬁrst task θ ψ 1 are randomly initialized.
For consecutive tasks, they are initialized by copying them from the previous task ( θ ψi ← θ ψi − 1 ).
The ˜ ψ i -function of the current task
M i is updated after each observed transition with the loss based on (5):
[Equation]
with ¯
[Equation]
(39) where ¯ θ ψi = θ ψi but ¯ θ ψi is treated as a constant for the purpose of optimization, i.e no gradients ﬂow through it.
Besides the current ˜ ψ i -function, SFQL also updates the ˜ ψ c -function which provided the GPI optimal action for the current transition: c = argmax k ∈{ 1 , 2 ,...,i } max b ˜ ψ k ( s, b ) (cid:62) ˜ w i .
The update uses the same loss as for the update of the active ˜ ψ i -function (39), but instead of using the
GPI optimal action as next action, it uses the optimal action according to its own policy: ¯ a t +1 = argmax a ˜ ψ c ( s t +1 , a ) (cid:62) ˜ w c
C.2. ξ - LEARNING
The ξ -learning agents (Algorithms 3, 4) allow to reevaluate policies in tasks with general reward functions.
If the reward function is not given, an approximation ˜ R i of the reward function for each task M i is learned.
The parameters for the approximation are randomly initialized at the beginning of each task.
After each observed transition the approximation is updated according to the following loss:
[Equation]
In the case of tasks with linear reward functions the reward approximation becomes ˜
R i ( φ ( s t , a t , s t +1 ); θ R i ) =
φ ( s t , a t , s t +1 )
(cid:62) θ R i .
Thus with θ R i = ˜
w i we recover the same procedure as for SFQL (38).
For non-linear, general reward functions we represented ˜ R with a neural network.
The input of the network is the feature φ ( s t , a t , s t +1 ) .
The network has one hidden layer with 10 neurons having ReLu activations.
The output is a linear mapping to the scalar reward r t ∈ R .
All ξ -learning agents learn an approximation of the ˜ ξ i -function for each task M i .
Analogous to
SFQL, the parameters of the ˜ ξ -function for the ﬁrst task θ ξ 1 are randomly initialized.
For con-
Algorithm
Input : exploration rate: (cid:15) learning rate for ψ -functions: α learning rate for reward weights w : α w features φ optional: reward weights for tasks: { ˜ w 1 , ˜ w 2 , . .
˜ w num_tasks } for i ← 1 to num_tasks do if ˜ w i not provided then ˜ w i ← small random initial values if i = 1 then initialize ˜ ψ
i : θ ψi ← small random initial values else θ ψi ← θ ψi −new_episode ← true for t ← 1 to num_steps do if new_episode then new_episode ← false s t ← initial state c ← argmax k ∈{ 1 , 2 ,...,i } max a ˜ ψ k ( s t , a ) (cid:62)
˜
w i // GPI optimal policy
With probability (cid:15) select a random action a t , otherwise a t ← argmax a ˜ ψ c ( s t , a ) (cid:62) ˜ w i
Take action a t and observe reward r t and next state s t +
Update ˜ w i using SGD( α w ) with L w =
( r t − φ ( s t , a t , s t +1 ) (cid:62) ˜ w i )if s t +1 is a terminal state then new_episode ← true γ t ←else γ t ← γ // GPI optimal
next action for task i
¯ a t +1 ← argmax a arg k ∈{ 1 , 2 ,...,i } ˜ ψ k ( s t +1 , a ) (cid:62) ˜ w i
y ← φ ( s t , a t , s t +1 )
+ γ t ˜ ψ
i ( s t +1 , ¯ a t +1 )
Update θ ψi using SGD( α ) with L ψ =
( y − ˜ ψ
i ( s t , a t ))
if c (cid:54) =
i then ¯ a t +1 ← argmax a ˜ ψ c ( s t +1 ,
[Equation]
optimal next action for task
[Equation]
Update θ ψc using SGD( α ) with L ψ = ( y − ˜ ψ c ( s t , a t ))
s t ← s t + secutive tasks, they are initialized by copying them from the previous task ( θ ξi ← θ ξi − 1 ).
The ˜ ξ i -function of the current task
M i is updated after each observed transition with the loss given in (46).
The ξ -learning agents differ in their setting for p ( φ t = φ | s t , a t ) in the updates which is described in the upcoming sections.
Besides the current ˜ ξ i -function, the ξ -learning agents also update the ˜ ξ c -function which provided the GPI optimal action for the current transition: c = argmax k ∈{ 1 , 2 ,...,i } max a t (cid:80) φ ∈ Φ ˜ ξ k
( s t , a t , φ ) ˜
R i ( φ ) .
The update uses the same loss as for the update of the active ˜ ξ i -function (46), but instead of using the GPI optimal action as next action, it uses the optimal action according to its own policy: ¯ a t +1 = max a (cid:80) φ ∈ Φ ˜ ξ c ( s t +1 , a, φ )
˜ R c ( φ ) .
MF ξ -learning: The model-free ξ -learning agent (Algorithm 3) uses a stochastic update for the ˜ ξfunctions.
Given a transition, we set p ( φ t = φ | s t , a t ) ≡ 1 for the observed feature φ = φ ( s t , a t , s t +1 ) and p ( φ t = φ | s t , a t ) ≡ 0 for all other features φ (cid:54) =
φ ( s t , a t , s t +1 ) .
MB ξ
-learning: The one-step SF model-based ξ -learning agent (Algorithm 4) uses an approximated model ˜ p to predict p ( φ t = φ | s t , a t ) to reduce the variance of the ξ -function update.
The model is by a linear mapping for each action.
It uses a softmax activation to produce a valid distribution over Φ :
[Equation]
41) where θ pa,φ ∈ R 113 .
As ˜ p is valid for each task in M , its weights θ p are only randomly initialized at the beginning of the ﬁrst task.
For each observed transition, the model is updated using the following loss:
[Equation] [Equation]
where we set p ( φ t = φ | s t , a t ) ≡ 1 for the observed feature φ = φ ( s t , a t , s t +1 ) and p ( φ t = φ | s t , a t ) ≡ 0 for all other features φ (cid:54) =
φ ( s t , a t , s t +1 ) .
C. E XPERIMENTAL P ROCEDURE
All agents were evaluated in both task types (linear or general reward function) on 300 tasks.
The agents experienced the tasks sequentially, each for 20.000 steps.
The agents had knowledge when a task change happened.
Each agent was evaluated for 10 repetitions to measure their average performance.
Each repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when taking steps, and d) the (cid:15) -greedy action selection of the agents.
The tasks, i.e. the reward functions, were different between the repetitions of a particular agent, but identical to the same repetition of a different agent.
Thus, all algorithms were evaluated over the same tasks.
The SF agents (SFQL, ξ -learning) were evaluated under two conditions.
First, that they have to learn the reward weights or the reward function online during the training (indicated by (O) in ﬁgures).
Second, the reward weights or the reward function is given to them.
As the SFQL does not support general reward functions, it is not possible to provide the SFQL agent with the reward function in the second condition.
As a solution, before the agent was trained on a new task M i , a linear model of the reward R i ( φ ) =
φ (cid:62) ˜
w i was ﬁtted.
The initial approximation ˜ w i was randomly initialized and then ﬁtted for 10 .
iterations using a gradient descent procedure based on the absolute mean error (L1 norm):
[Equation]
with a learning rate of η = 1 . 0 that yielded the best results tested over several learning rates.
Hyperparameters: The hyperparameters of the algorithms were set to the same values as in (Barreto et al., 2017).
A grid search over the learning rates of all algorithms was performed.
Each learning rate was evaluated for three different settings which are listed in Table 1.
If algorithms had several learning rates, then all possible combinations were evaluated.
This resulted in a different number of evaluations per algorithm and condition: QL - 3, SFQL (O) - 9, MF Xi (O) - 9,
MB Xi
(O) - 27, SFQL - 3, MF Xi - 3, MB Xi - 9.
In total, 63 parameter combinations were evaluated.
The reported performances in the ﬁgures are for the parameter combination that resulted in the highest cumulative total reward averaged over all 10 repetitions in the respective environment.
Please note, the learning rates α
and α w are set to half of the rates deﬁned in (Barreto et al., 2017).
This is necessary due to the differences in calculating the loss and the gradients in the current paper.
We use mean squared error loss formulations, whereas (Barreto et al., 2017) uses absolute error losses.
The probability for random actions of the (cid:15) -Greedy
action selection was set to (cid:15) = 0 .
15 and the discount rate to γ = 0 .
The initial weights θ for the function approximators were randomly sampled from a standard distribution with θ init ∼ N ( µ = 0 , σ = 0 . 01) .
Computational Resources: Experiments were conducted on a cluster with a variety of node types (Xeon SKL Gold 6130 with 2.10GHz, Xeon SKL Gold 5218 with 2.30GHz, Xeon SKL Goldwith 2.60GHz, Xeon SKL Gold 6244 with 3.60GHz, each with 192 GB Ram, no GPU).
The time for evaluating one repetition of a certain parameter combination over the 300 tasks depended on the algorithm and the task type.
Linear reward function tasks:
QL ≈ 1 h , SFQL (O)
≈ 4 h , MF ξ (O)
≈ 42 h , MB ξ (O)
≈ 43 h , SFQL ≈ 4 h , MF ξ ≈ 15 h , and MB ξ ≈ 16 h .
General reward function
[Table]
tasks:
QL ≈ 1 h , SFQL (O)
≈ 4 h , MF ξ (O)
≈ 68 h , MB ξ
(O)
≈ 67 h , SFQL ≈ 5 h , MF ξ ≈ 14 h , and MB ξ ≈ 18 h .
Please note, the reported times do not represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some use different software packages (numpy or pytorch) for their individual computations.
C. E FFECT OF I NCREASING N ON - LINEARITY IN G ENERAL R EWARD T ASK
We further studied the effect of general reward functions on the performance of classical SF compared to ξ -learning (Fig.
1 - c).
We evaluated the agents in tasks with different levels of difﬁculty in relation to how well their reward functions can be approximated by a linear model.
Seven difﬁculty levels have been evaluated.
For each level, the agents were trained sequentially on 300 tasks as for the experiments with general reward functions.
The reward functions for each level were sampled with the following procedure.
Several general reward functions were randomly sampled as previously described.
For each reward function, a linear model of a reward weight vector ˜ w was ﬁtted using the same gradient descent procedure as in Eq.
The ﬁnal average absolute model error after 10 . 000 iterations was measured.
Each of the seven difﬁculty levels deﬁnes a range of model errors its tasks have with the following increasing ranges: { [0 . 0 , 0 .
For each difﬁculty level, 300 reward functions were selected that yield a linear model are in the respective range of the level.
Q-Learning, SFQL, and MF Xi-learning were each trained on 300 tasks, i.e. reward functions, on each difﬁculty level.
As hyperparameters were the best performing parameters from the previous general reward task experiments used.
We measured the ratio between the total return over 300 tasks of QL and MF Xi-learning (QL/MF Xi), and SFQL and MF Xi-learning (SFQL/MF Xi).
Fig.
1c shows the results, using as x-axis the mean average absolute model error deﬁned by the bracket of each difﬁculty level.
The results show that the relative performance of SFQL compared to MF
Xi reduces with higher non-linearity of the reward functions.
For reward functions that are nearly linear (mean error of 0.125), both have a similar performance.
Whereas, for reward functions that are difﬁcult to model with a linear relation (mean error of 1.625) SFQL reaches only less than 50% of the performance of MF Xi-learning.
Input : exploration rate: (cid:15) learning rate for ξ -functions: α learning rate for reward models R : α R features φ optional: reward functions for tasks: { ˜ R 1 , ˜ R 2 , . . .
, ˜ R num_tasks } for i ← 1 to num_tasks do if ˜ R i not provided then initialize ˜ R
i : θ Ri ← small random initial values if i = 1 then initialize ˜ ξ
i : θ ξi ← small random initial values else θ ξi ← θ ξi −new_episode ← true for t ← 1 to num_steps do if new_episode then new_episode ← false s t ← initial state c ← argmax k ∈{ 1 , 2 ,...,i } max a (cid:80) φ ˜ ξ k
( s t , a, φ ) ˜
R i ( φ ) // GPI
optimal policy
With probability (cid:15) select a random action a t , otherwise a t ← argmax a (cid:80) φ ˜ ξ c ( s t , a, φ )
˜ R i ( φ )
Take action a t and observe reward r t and next state s t + if ˜ R i not provided then
Update θ Ri using SGD( α R ) with L R =
( r t − ˜ R
i ( φ ( s t , a t , s t +1 ))
if s t +1 is a terminal state then new_episode ← true γ t ←else γ t ← γ //
GPI optimal next action for task i
¯ a t +1 ← argmax a arg k ∈{ 1 , 2 ,...,i } (cid:80) φ ˜ ξ k
( s t +1 , a, φ )
˜
R i ( φ ) foreach φ ∈ Φ do if φ = φ ( s t , a t , s t +1 )
then y φ ← 1 + γ t ˜ ξ i ( s t +1 , ¯ a t +1 , φ ) else y φ ← γ t ˜ ξ i ( s t +1 , ¯ a t +1 , φ )
Update θ ξi using SGD( α ) with L ξ =
(cid:80) φ ( y φ − ˜ ξ
i ( s t , a t , φ ))if c (cid:54) =
i then // optimal next action for task c ¯ a t +1 ← argmax a (cid:80) φ ˜ ξ c ( s t +1 , a, φ )
˜ R c ( φ ) foreach φ ∈ Φ do if φ = φ ( s t , a t , s t +1 )
then y φ ← 1 + γ t ˜ ξ c ( s t +1 , ¯ a t +1 , φ ) else
y φ ← γ t ˜ ξ c ( s t +1 , ¯ a t +1 , φ )
Update θ ξc using SGD( α ) with L ξ = (cid:80) φ ( y φ − ˜ ξ c
( s t , a t , φ ))s t ← s t +
Algorithm
Input : exploration rate: (cid:15) learning rate for ξ -functions: α learning rate for reward models R : α R learning rate for the one-step SF model ˜ p : β features φ optional: reward functions for tasks: { ˜ R 1 , ˜ R 2 , . . .
, ˜ R num_tasks } initialize ˜ p : θ p ← small random initial values for i ← 1 to num_tasks do if ˜ R i not provided then initialize ˜ R
i : θ Ri ← small random initial values if i = 1 then initialize ˜ ξ
i : θ ξi ← small random initial values else θ ξi ← θ ξi −new_episode ← true for t ← 1 to num_steps do if new_episode then new_episode ← false s t ← initial state c ← argmax k ∈{ 1 , 2 ,...,i } max a (cid:80) φ ˜ ξ k
( s t , a, φ ) ˜
R i ( φ ) // GPI optimal policy
With probability (cid:15) select a random action a t , otherwise a t ← argmax a (cid:80) φ ˜ ξ c ( s t , a, φ )
˜ R i ( φ )
Take action a t and observe reward r t and next state s t + if ˜ R i not provided then
Update θ Ri using SGD( α R ) with L R =
( r t − ˜ R
i ( φ ( s t , a t , s t +1 ))foreach φ ∈
Φ do if φ = φ ( s t , a t , s t +1 )
then y φ ← 1 else y φ ←
Update θ p using SGD( β ) with L p = (cid:80) φ ( y φ − ˜ p
( s t , a t , φ ))
if s t +1 is a terminal state then new_episode ← true γ t ←else γ t ← γ // GPI
optimal next action for task i
¯ a t +1 ← argmax a arg k ∈{ 1 , 2 ,...,i } (cid:80) φ ˜ ξ k
( s t +1 , a, φ )
˜
R i ( φ ) foreach φ ∈ Φ do y φ ← ˜ p
( s t , a t , φ ) + γ t ˜ ξ i ( s t +1 , ¯ a t +1 , φ )
Update θ ξi using SGD( α ) with L ξ =
(cid:80) φ ( y φ − ˜ ξ
i ( s t , a t , φ ))if c (cid:54) =
i then //
optimal next action for task c
¯ a t +1 ← argmax a (cid:80) φ ˜ ξ c ( s t +1 , a, φ ) ˜ R c ( φ ) foreach φ ∈ Φ do y φ ← ˜ p
( s t , a t , φ ) + γ t ˜ ξ c ( s t +1 , ¯ a t +1 , φ )
Update θ ξc using SGD( α ) with L ξ = (cid:80) φ ( y φ − ˜ ξ c
( s t , a t , φ ))s t ← s t +
D E XPERIMENTAL D ETAILS : R ACER E NVIRONMENT
This section extends the brief introduction to the racer environment (Fig.
2 - a) given in Section 4.2.
D. E NVIRONMENT
The environment is a continuous two-dimensional area in which the agent drives similar to a car.
The position of the agent is a point in the 2D space: p = ( x, y ) ∈ [0 , 1] 2 .
Moreover, the agent has an orientation which it faces: θ ∈ [ − π, π 1] .
The action space of the agent consists of three movement directions:
A = { right, straight, left } .
Each action changes the position of the agent depending on its current position and orientation.
The action straight changes the agent’s position by 0 .
075 towards its orientation θ .
The action right changes the orientation of the agent to θ + 17 π and its position 0 .
06 towards this new direction, whereas left the direction to θ −
π changes.
The environment is stochastic by adding Gaussian noise with σ = 0 .
005 to the ﬁnal position x , y , and orientation θ .
If the agent drives outside the area ( x, y ) ∈ [0 , 1] 2 , then it reappears on the other opposite side.
The environment resembles therefore a torus (or donut).
As a consequence, distances d ( p x , p y ) are also measure in this space, so that the positions p x = (0 . 1 , 0 . 5) and p y =
(0 . 9 , 0 . 5) have not a distance of 0 .
8 but d ( p x , p y ) = 0 .
The environment has 3 markers at the positions m 1 = (0 . 25 , 0 . 75) , m 2 = (0 .
75 , 0 . 25) , and m
=
The features measure the distance of the agent to each marker: φ ∈ R 3 with φ k = d ( p, m k ) .
Each feature dimensions is normalized to be φ k ∈ [0 , 1] .
At the beginning of an episode the agent is randomly placed and oriented in environment.
An episode ends after 200 time steps.
The state space of the agents is similarly constructed as for the object collection environment.
The agent’s position is encoded with a 10 × 10 radial basis functions s pos ∈ R 100 as deﬁned in 34.
In difference, that the distances are measure according to the torus shape.
A similar radial basis function approach is also used to encode the orientation s ori ∈ R 20 of the agent using 20 equally distributed gaussian centers in [ − π, π ] and σ = 15 π .
Please note, π and − π are also connected in this space, i.e. d ( π, − π ) = 0 .
The combination of the position and orientation of the agent is the ﬁnal state: s
[Equation]
The reward functions deﬁne preferred positions in the environment based on the features, i.e. the distance of the agent to the markers.
A preference function r k exists for each distance.
The functions are composed of a maximization over m Gaussian components that evaluate the agents distance:
[Equation]
Reward functions are randomly generated by sampling the number of Gaussian components m to be 1 or 2 .
The properties of each component are sampled according to µ j ∼ U (0 . 0 , 0 . 7) , and σ j ∼ U (0 . 001 , 0 . 01) .
Fig.
2 - a illustrates one such randomly sampled reward function where dark areas represent locations with high rewards.
D. A LGORITHMS
We evaluated Q-learning, SFQL (O), SFQL, and MF ξ -learning (see Section C.2 for their full description) in the racer environment.
In difference to their implementation for the object collection environment, they used a different neural network architecture to approximate their respective value functions.
D.2. QL
Q-learning uses a fully connected feedforward network with bias and a
ReLU activation for hidden layers.
It has 2 hidden layers with 20 neurons each.
D.2. SFQL
Q-learning uses a feedforward network with bias and a ReLU activation for hidden layers.
It has for each of the three feature dimensions a separate fully connected subnetwork.
Each subnetwork hashidden layers with 20 neurons each.
D.2. C ONTINUOUS M ODEL - FREE ξ - LEARNING
The racer environment has continuous features φ ∈ R 3 .
Therefore, the MF ξ -learning procedure (Alg. 3) can not be directly applied as it is designed for discrete feature spaces.
We introduce here a
MF ξ -learning procedure for continuous feature spaces (CMF ξ -learning).
It is a feature dimension independent, and discretized version of ξ -learning.
As the reward functions (44) are a sum over the individual feature dimensions, the Q-value can be computed as:
[Equation]
where Φ k is the feature space for each feature dimension which is Φ k = [0 , 1] in the racer environment.
ξ πk is a ξ -function for the feature dimension k .
(45) shows that the ξ -function can be independently represented over each individual feature dimension φ k , instead of over the full features φ .
This reduces the complexity of the approximation.
Moreover, we introduce a discretization of the ξ -function that discretizes the space of each feature dimension k in U =
11 bins with the centers:
[Equation]
where ∆ φ k is the distance between the centers, and φ mink = 0 . 0 is the lowest center, and φ maxk = 1 .the largest center.
Given this discretization and the decomposition of the Q-function according to (45), the Q-values can be computed by:
[Equation]
Alg.
5 lists the complete CMF ξ -learning procedure with the update steps for the ξ -functions.
Similar to the SFQL agent, the CMF Xi uses a feedforward network with bias and a ReLU activation for hidden layers.
It has for each of the three feature dimensions a separate fully connected subnetwork.
Each subnetwork has 2 hidden layers with 20 neurons each.
The discretized outputs per feature dimension share the last hidden layer per subnetwork.
The ξ -function is updated according to the following procedure.
Instead of providing a discrete learning signal to the model, we encode the observed feature using continuous activation functions around each bin center.
Given the j ’th bin center of dimension k , x k,j , its value is encoded to be 1 .if
the feature value of this dimension aligns with the center ( φ k = x k,j ) .
Otherwise, the encoding for the bin decreases linearily based on the distance between the bin center and the value ( | x k,j − φ k | ) and reaches 0 if the value is equal to a neighboring bin center, i.e. has a distance ≥ ∆ φ k .
We represent this encoding for each feature dimension k by u k ∈ (0 , 1) U with:
[Equation]
To learn ˜ ξ we update the parameters θ ξ using stochastic gradient descent following the gradients ∇ θ ξ L ξ ( θ ξ ) of the loss based on the ξ -learning update (13):
[Equation]
where n = 3 is the number of feature dimensions and ˜ ξ k is the vector of the U discretized ξ -values for dimension k .
D. E XPERIMENTAL P ROCEDURE
All agents were evaluated on 37 tasks.
The agents experienced the tasks sequentially, each for 1000 episodes ( 200 , 000 steps per task).
The agents had knowledge when a task change happened.
Each agent was evaluated for 10 repetitions to measure their average performance.
Each repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when taking steps, and d) the (cid:15) -greedy action selection of the agents.
The tasks, i.e. the reward functions, were different between the repetitions of a particular agent, but identical to the same repetition of a different agent.
Thus, all algorithms were evaluated over the same tasks.
SFQL was evaluated under two conditions.
First, by learning the reward weights online during the training (indicated by (O) in ﬁgures).
Second, the reward weights were trained with the iterative gradient decent method in (43).
The weights were trained for 10 , 000 iterations with an learning rate of 1 . 0 .
At each iteration, 50 random points in the task were sampled and their features and rewards are used for the training step.
Hyperparameters A grid search over the learning rates of all algorithms was performed.
Each learning rate was evaluated for three different settings which are listed in Table 2.
If algorithms had several learning rates, then all possible combinations were evaluated.
This resulted in a different number of evaluations per algorithm and condition: QL - 4, SFQL (O) - 12, SFQL - 4, CMF Xi 4.
In total, 24 parameter combinations were evaluated.
The reported performances in the ﬁgures are for the parameter combination that resulted in the highest cumulative total reward averaged over allrepetitions in the respective environment.
The probability for random actions of the (cid:15) -Greedy
action selection was set to (cid:15) = 0 .
15 and the discount rate to γ = 0 .
The initial weights and biases θ for the function approximators were initialized according to an uniform distribution with θ i ∼ U
( −√ k, √ k ) , where k = 1 in_features .
[Table]
5 } α w
Learning rate of the reward weights { 0 . 025 , 0 . 05 , 0 . 075 }
Computational Resources: Experiments were conducted on the same cluster as for the object collection environment experiments.
The time for evaluating one repetition of a certain parameter combination over the 37 tasks depended on the algorithm:
QL ≈ 9 h , SFQL (O)
≈
70 h , SFQL ξ ≈ 73 h , and CMF ξ ≈ 88 h .
Please note, the reported times do not represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some use different software packages (numpy or pytorch) for their individual computations.
Input : exploration rate: (cid:15) learning rate for ξ -functions: α learning rate for reward models R : α R features φ ∈ R n components of reward functions for tasks: { R 1 = { r 11 , r 12 , ..., r 1 n } , R 2 , . . .
, R num_tasks } discretization parameters: X , ∆ φ for i ← 1 to num_tasks do if i
= 1 then ∀ k ∈{ 1 ,...,n } : initialize ˜
ξ ik : θ ξi,k ← small random values else ∀ k ∈{ 1 ,...,n } : θ ξ i,k ← θ ξ i − 1 ,k new_episode ← true for t ← 1 to num_steps do if new_episode then new_episode ← false s t ← initial state c ← argmax j ∈{ 1 , 2 ,...,i } max a (cid:80) nk
=1 (cid:80) x ∈ X k ˜ ξ jk ( s t , a, x ) r ik ( x ) //
GPI policy
With probability (cid:15) select a random action a t , otherwise a t ← argmax a (cid:80) nk
=1 (cid:80) x ∈ X k ˜ ξ jk ( s t , a, x ) r ik ( x )
Take action a t and observe reward r t and next state s t + if s t +1 is a terminal state then new_episode ← true γ t ←else γ t ← γ // GPI
optimal next action for task i
¯ a t +1 ← argmax a arg j ∈{ 1 , 2 ,...,i } (cid:80) nk =1 (cid:80) x ∈ X k ˜ ξ jk ( s t , a, x ) r ik ( x ) φ t ← φ ( s t , a t , s t +1 ) for k ← 1 to n do foreach x ∈ X k do y k,x ← max (cid:16) 0 , 1 − | x − φ t,k | ∆ φ (cid:17) + γ t ˜ ξ
ik ( s t +1 , ¯ a t +1 , x )
Update θ ξi using SGD( α ) with L ξ =
(cid:80) nk
=1 (cid:80) x ∈ X
k
( y k,x − ˜ ξ ik ( s t , a t , x ))
if c (cid:54) =
i then //
optimal next action for task
c ¯ a t +1 ← argmax a (cid:80) nk =1 (cid:80) x ∈ X k ˜ ξ ck
( s t , a, x ) r ck ( x ) for k ← 1 to n do foreach x ∈ X k do y k,x ← max (cid:16) 0 , 1 − | x − φ t,k | ∆ φ (cid:17) + γ t ˜ ξ ck
( s t +1 , ¯ a t +1 , x )
Update θ ξc using SGD( α ) with L ξ = (cid:80) nk
=1 (cid:80) x ∈ X
k
( y k,x − ˜ ξ ck
( s t , a t , x ))s t ← s t +
E A DDITIONAL E XPERIMENTAL R ESULTS
This section reports additional results and experiments:
Report of the total return and the statistical signiﬁcance of differences between agents for all experiments 2.
Evaluation of the agents in the original object collection task by Barreto et al.
E. O BJECT C OLLECTION T ASK BY B ARRETO ET AL . (2017)
We additionally evaluated all agents in the original object collection task by Barreto et al.
[Figure]
Environment: The environment differs to the modiﬁed object collection task (Section.
C) only in terms of the objects and features.
The environment has 3 object types: orange, blue, and pink (Fig. 3).
The feature encode if the agent has collected one of these object types or if it reached the goal area.
The ﬁrst three dimensions of the features φ ( s t , a t , s t +1 ) ∈ { 0 , 1 } 4 encode which object type is collected.
The last dimension encodes if the goal area was reached.
In total | Φ | = 5 possible features exists: φ 1 = [0 , 0 , 0 , 0] (cid:62) - standard observation, φ 2 = [1 , 0 , 0 , 0] (cid:62) - collected an orange object, φ 3 =
[0 , 1 , 0 , 0] (cid:62) - collected a blue object, φ 4 = [0 , 0 , 1 , 0] (cid:62) - collected a pink object, and φ 5 =
[0 , 0 , 0 , 1] (cid:62) - reached the goal area.
The rewards r = φ (cid:62) w i are deﬁned by a linear combination of discrete features φ ∈ N 4 and a weight vector w ∈ R 4 .
The ﬁrst three dimensions in w deﬁne the reward that the agent receives for collecting one of the object types.
The ﬁnal weight deﬁnes the reward for reaching the goal state which is w 4 = 1 for each task.
All agents were trained in on 300 randomly generated linear reward functions with the same experimental procedure as described in Section.
C. For each task the reward weights for the 3 objects are randomly sampled from a uniform distribution: w k ∈{ 1 , 2 , 3 } ∼ U ( − 1 , 1) .
Results: The results (Fig. 4) follow closely the results from the modiﬁed object collection task (Fig. 1 - b, and 5 - a).
MF ξ reaches the highest performance outperforming SFQL in terms of learning speed and asymptotic performance.
It is followed by MB ξ and SFQL which show no statistical signiﬁcant difference between each other in their ﬁnal performance.
Nonetheless, MB Ξ has a higher learning speed during the initial 40 tasks.
The results for the agents that learn the reward weights online (SFQL (O), MF ξ (O), and MB ξ (O)) follow the same trend with MF ξ (O) outperforming SFQL (O) slightly.
Nonetheless, the ξ -agents have a much stronger learning speed during the initial 70 tasks compared to SFQL (O), due to the errors in the approximation of the weight vectors, especially at the beginning of a new task.
All agents can clearly outperform standard
Q-learning.
E. T OTAL R ETURN IN T RANSFER L EARNING E XPERIMENTS AND S TATISTICAL S IGNIFICANT D IFFERENCES
Fig.
5 shows for each of the transfer learning experiments in the object collection and the racer environment the total return that each agent accumulated over all tasks.
Each dot besides the boxplot shows the total return for each of the 10 repetitions.
The box ranges from the upper to the lower quartile.
The whiskers represent the upper and lower fence.
The mean and standard deviation are indicated by the dashed line and the median by the solid line.
The tables in Fig.
5 report the p-value of pairwise Mann–Whitney U test.
A signiﬁcant different total return can be expected if p < 0 . 05 .
For the object collection environment (Fig.5 - a; b), ξ -learning outperforms SFQL in both conditions, in tasks with linear and general reward functions.
However, the effect is stronger in tasks with general reward functions where SFQL has more problems to correctly approximate the reward function with its linear approach.
For the condition, where the agents learn a reward model online (O), the difference between the algorithms in the general reward case is not as strong due the effect of their poor approximated reward models for all agents.
In the racer environment (Fig.5 - c) SFQL has a poor performance below standard Q-learning as it can not appropriately approximate the reward functions with a linear model.
In difference CMF ξ -learning outperforms QL.
[Figure]
by Barreto et al.
[Figure]
Mann–Whitney U tests between the agents.
50k 100k 150k 200k
[Table]
[Table] [Table]
SFQL
[Table]
Xi
[Table]
T o t a l
R
[Table]
t u
[Table]
n
[Table]
50k
[Table]
[Figure] [Figure]