Constrained GPI for Zero-Shot Transfer in Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress
Abstract
For zero-shot transfer in reinforcement learning where the reward function variesbetween different tasks, the successor features framework has been one of thepopular approaches.
However, in this framework, the transfer to new target taskswith generalized policy improvement (GPI) relies on only the source successorfeatures [5] or additional successor features obtained solely with the functionapproximators’ generalization to novel inputs [10].
The goal of this work isto improve the transfer by further bounding the value approximation errors ofsuccessor features on the new target tasks.
To this end, we ﬁrst present lower andupper bounds of optimal values for the novel tasks that are expressible as linearcombinations of source tasks given the reward decomposition structure.
Based onthe bounds, we then propose constrained GPI as a simple test-time approach thatcan improve transfer by constraining action-value approximation errors on newtarget tasks.
With experiments in the Scavenger and Reacher environments, weshow that the proposed constrained GPI signiﬁcantly outperforms the prior GPI’stransfer performance.
Introduction
In the ﬁeld of sequential decision making, deep reinforcement learning (RL) has been shown tobe effective for various types of problems, including games [26] and robotics [18, 20].
With thesesuccesses, interest in multi-task RL has also been surged, where its goal is to train a single RL agentthat can efﬁciently solve multiple varying tasks.
In multi-task RL, we especially focus on the transferlearning setting, where the agent learns a shared structural knowledge from a set of source tasksduring training and exploits and generalizes them in new, unseen target tasks at test time.
One popular approach to transfer in reinforcement learning is to leverage the successor features(SFs) framework [1, 5, 6, 10, 21], which can transfer policies learned on source tasks to target tasks,where the tasks share the same environment dynamics but differ in their reward functions.
Successorfeatures build a representation of value functions decoupled from reward functions, and enable theirtransfer to tasks with arbitrary reward functions by taking an inner product with corresponding taskvectors.
They utilize generalized policy improvement (GPI) [5], which is a generalized version ofpolicy improvement by combining multiple policies, and it provides the performance lower boundsfor GPI policies.
However, GPI does not take into account any information from the smoothness of action-valuefunctions with respect to task vectors.
Tackling this issue, Borsa et al.
[10] propose universalsuccessor features approximators (USFAs), which can estimate the optimal successor features forgiven novel task vectors.
Nevertheless, the function approximators can make high approximationerrors on the task vectors, especially when the new task vectors are distant from the source taskvectors.
For instance, when USFAs are trained with source tasks to get close to given goals, they may not generalize well to the target tasks where the agent should get away from the given goals.
That is,if we test on target task vectors whose elements have the opposite signs from the source task vectors,
USFAs could output successor features with high approximation errors.
Therefore, we aim at bounding value approximation errors on the new target tasks to improve thesuccessor features approximation of USFAs for the new tasks.
We ﬁrst introduce a new theoremon bounding the optimal values for the tasks that are expressible as linear combinations of sourcetasks, generalizing the conical combination condition used by the prior theorem by Nemecek and Parr[21].
Using the bounds that this theorem suggests as constraints, we can train the successor featuresapproximators for bounding their action-value approximation errors on novel tasks.
We extend thisidea further so that we can accomplish a similar effect with no additional training; as a result, wepropose constrained GPI as a test-time approach to bounding the approximation error.
Despite itssimplicity and no need for any modiﬁcation to the training procedure, we empirically attain largeperformance improvements using the constrained GPI compared to the original GPI in the Scavenger[7, 8] and Reacher [11] environments.
Our main contributions can be summarized as follows:
• We present a novel theorem on lower- and upper-bounding optimal values for novel tasks
that can be expressed as linear combinations of source tasks.
It extends and generalizes the
previous theorem for conical combinations by Nemecek and Parr [21], to enable a broader
application of the bounds.
• Based on our new theorem, we propose constrained GPI as a simple test-time approach thatcan improve transfer to novel tasks by constraining action-value approximation errors on
new target tasks, without any modiﬁcation to the training procedure.
• We empirically show that our approach can improve the performance over the GPI baselines
by a large margin in the Scavenger and Reacher environments.
We also provide analyses for
a better understanding of the results of our method.
Preliminaries
We describe the problem setting and background on successor features and universal successorfeatures approximators.
Please refer to Appendix for a more in-depth discussion of other relatedwork.
2.1 The Zero-Shot Transfer Problem in RL
We deﬁne a Markov Decision Process (MDP) as M ≡ ( S , A , P, R, γ ) .
S and A are the state and 67 action spaces, and P is the transition probability where P ( ·| s, a ) deﬁnes the probability distribution 68 of the next states given s ∈ S and a ∈ A .
R is the reward function where R ( s, a, s (cid:48) ) is the reward for 69 taking action a at state s resulting in s (cid:48) , and γ ∈ (0 , 1] is the discount factor.
We assume that rewardsare bounded.
We consider the zero-shot transfer problem in reinforcement learning; as in [5], each task is deﬁnedby its task vector w ∈ R d , and only the reward functions differ across tasks, being decomposed as
[Equation]
where φ ( s, a, s (cid:48) )
∈ R
d is the features of ( s, a, s (cid:48) ) .
We denote the set of source task vectors as T ,which is used for training.
We employ the assumption that the features φ ( s, a, s (cid:48) ) are available to theagent [5, 10], but our approach is orthogonal to the availability of φ and also applicable to the casewhere φ is learned from samples.
At test time, we evaluate the transferred policy on each target taskw (cid:48) / ∈ T without any additional update of pre-trained components.
2.2 Successor Features and Universal Successor Features Approximators
We now review successor features [5] and how they are transferred to different tasks.
Equation (1)allows expressing the action-value function for policy π on task w as
[Equation]
where φ t = φ ( s t , a t , s t +1 ) ∈ R d .
Here, ψ π ( s, a
) ∈ R d is called the successor features for policy πat ( s, a ) , and taking its inner product with an arbitrary task w results in the action-value for π on w ;i.e ., Q π w (s, a).
Thanks to the analogy between (rewards r , action-value functions Q ) and (features φ ,successor features ψ ), the Bellman equation applies to the successor features and thus they can betrained similarly as action-value functions are learned; e.g ., Q-learning.
The policy improvement theorem [9] states that a new policy that takes a greedy action according to agiven policy’s value function at each state performs at least as good as the original policy.
Generalizedpolicy improvement (GPI) [5] extends policy improvement to the case where the value functions ofmultiple policies are used.
Given a task w (cid:48) , a set of policies π 1 , . . .
, π n , their action-value functions
Q π 1 w (cid:48) , . .
, Q π n w (cid:48) and their approximations ˜ Q π 1 w (cid:48) , . .
˜ Q π n w (cid:48) , the GPI policy is deﬁned as
[Equation]
and Barreto et al.
[5] suggest that Q π GPI w
[Equation]
s, a ) ≥ max i
[Equation]
s, a ) − 21 − γ
[Equation]
Q
[Equation]
˜ Q
[Equation]
w
[Equation]
(cid:13)(cid:13)(cid:13) ∞ .
Theyalso provide the upper bound on the suboptimality of the GPI policy as
[Equation]
where each π i is an optimal policy for w i .
While the GPI theorem allows transfer of learned successor features to arbitrary tasks with sharedenvironment dynamics, it is limited in the following aspect; GPI uses the action-values for sourcetasks on target tasks based on the reward decomposition assumption (Equation (1))
i.e .,
˜ Q
π i w (cid:48) ( s, a ) =
˜
ψ π i ( s, a ) (cid:62) w (cid:48) for each i , but does not take any advantage of the smoothness of the optimal actionvalue functions with respect to different task vectors [10].
To overcome this limitation, Borsa et al.
[10] introduce universal successor features approximators(USFAs).
Inspired by universal value functions (UVFs) [25], they extend the original successorfeatures with policy vectors z ∈ R l as input to their approximators.
More speciﬁcally, universalsuccessor features (USFs) are deﬁned to satisfy
[Equation]
[Equation]
where z is a policy vector for the policy π z , and USFAs ˜ ψ are the learned approximators of USFs ψ .
Naturally, the value functions are expressed as
[Equation]
Each reward function induces optimal policies, which can be encoded using the corresponding taskvectors.
That is, one can simply choose to deﬁne the policy vector space to be the same as the taskvector space ( l = d ) and let z = w be a policy vector of an optimal policy for task w .
Then, π w
and
Q π w w denote an optimal policy for w and its action-value function, respectively.
The training of USFAs is similar to that of successor features, except for that it additionally involvessampling of policy vectors given task vectors.
The update of USFAs at the k -th iteration is˜
[Equation]
(7) for a (cid:48) =
argmax b ˜ ψ ( k ) ( s (cid:48) , b, z ) (cid:62) z .
D z ( ·| w ) is the policy vector distribution; for instance, N ( w , σI )can be used for better training with diversiﬁed inputs µ transition buffer
USFAs provide a beneﬁt that they allow a GPI policy to use an arbitrary set of policies { π z } z ∈C asπ GPI ( s ) ∈ argmax a max z ∈C ˜ Q π z w (cid:48)
( s, a ) .
However, the generalization of USFAs to new policy vectorsdepends on a function approximator ψ , and thus if C contains policy vector(s) distant from sourcevectors, a GPI policy with C may have high approximation errors and perform poorly or even worsethan a GPI policy with only source vectors [10], as will be demonstrated later in our experiments.
Constrained GPI for Improved Zero-Shot Transfer of Successor Features
To mitigate the aforementioned issue of the possibly unlimited approximation errors of USFAs, wepropose a simple yet effective method that improves the transfer of successor features by furtherleveraging the reward decomposition structure in Equation (1).
We ﬁrst present, under a morerelaxed condition, lower and upper bounds on the optimal values for novel task vectors, which canbe expressed as linear combinations of source task vectors (Section 3.1).
Then, we propose a novelapproach called constrained GPI , which effectively conﬁnes the approximated action-values insidethe computed lower and upper bounds (Section 3.2).
3.1 Bounding Optimal Values for New Tasks
[Equation]
[Table]
/qVaxLtDdV/
JurMSNSsrnTRhn85Ws+LDDMIWq+DF04IwmOD1Ku7EKYS/hRP8qVUPHe/p2HtNd0Om7GTUo2t0k2HBJNMsHPq3US78DNj8WgSW2oX07xYnwRf4FYlLD+jZmFHT/IEF9W5jBFMzkQwGRDh2KPp076qHHRa0YvWxseNxtbb8uwXwWPwFKyCCLwEW+AD2AX7gIAh+AF+gl+1pPal9rX2bRw6P1diHoGK1L7/BWVotDE=</latexit> 2 ○
While the conical combinations of wand w 2 covers only area 1 (cid:13) , the linear combinations covers both area 1 (cid:13) – 2 (cid:13) .
Theorem 1 of [21] provides the lower and upper bounds on the value ofan optimal policy for a new task, whose vector w (cid:48) is a positive conicalcombination of source task vectors i.e ., w (cid:48)
= (cid:80) w ∈T
α w w such that α w ≥0 , ∀ w ∈ T and (cid:80) w ∈T α w > 0 1 .
However, for broad applications of such 132 bounds, the positive conical combination condition could be too restrictive,since the resulting bounds only apply to the task vectors that appear insidethe conical hull of source task vectors.
Therefore, we suggest a more relaxed theorem, which holds for an arbitrarytask vector w (cid:48) that is expressed as a linear combination of the source taskvectors i.e ., w (cid:48) =
(cid:80) w ∈T α w w for α w ∈ R , ∀ w ∈ T .
Figure 1 compares 138 an example of the task space coverage of conical [21] and our linear combinations.
With our extended task space coverage, we can use the bounds forimproving transfer to more general target tasks outside of the conical hull,which will be further discussed in the next section.
We deﬁne (cid:15) π w 1 w 2 to be an upper bound on the approximation error of ˜ Q π w 1 w 2 forarbitrary tasks w 1 , w 2 such that
[Equation]
and we present our theorem as follows.
Theorem 1.
Given a task vector w (cid:48) =
(cid:80) w ∈T α w w for α w ∈ R , ∀ w ∈ T , for all state-action pairs( s, a ) ∈ S ×A , the action-value of π w (cid:48) for an optimal policy for task w (cid:48) is lower- and upper-boundedas L
[Equation] [Equation]
[Equation]
T , α ( s, a ) for
[Equation]
where r min w is the minimum reward on w i.e ., r min w = min ( s,a ) ∈S×A R w ( s, a ) and α = { α w } w ∈T .
Proof.
For the derivation of the lower bound L w (cid:48) , T ( s, a ) , since Q π w (cid:48) w (cid:48) is the optimal action-valuefunction for task w (cid:48) and Q π w (cid:48) w (cid:48)
( s, a ) ≥ Q π w w (cid:48)
( s, a ) for arbitrary task w and state-action pair ( s, a ) ,
[Equation]
For the upper bound U w (cid:48) , T , α ( s, a ) , we use that Q π w (cid:48) w ( s, a ) ≤ Q π w w ( s, a ) and Q π w (cid:48)
w ( s, a ) ≥ 11 − γ r min wfor arbitrary task w and state-action pair ( s, a ) , which leads to
[Equation]
Intuitively, this theorem states the condition that the optimal action-value of an arbitrary target taskmust satisfy, for the given optimal action-values of the source tasks.
Thus, as will be shown in thenext section, this can be employed as a constraint on the action-values for novel target tasks, for theguidance of transfer.
Also, note that our Theorem 1 recovers Theorem 1 of [21] when w (cid:48) is a conicalcombination of w ’s from T i.e ., α w ≥ 0 , ∀ w ∈ T .
In Sections 4.1 and 4.2, we will empirically show 159 that the application of our Theorem 1 can result in signiﬁcant performance improvement in the casewhere target tasks are outside the conical hull of source tasks.
3.2 Constrained Training and Constrained GPI
As described in Section 2.2, the universal successor features approximators (USFAs) [10] improvethe original successor features so that arbitrary policy vectors, including the ones for target tasks,can be used for GPI.
However, the use of arbitrary policy vectors with USFAs solely relies on thegeneralization power of the approximators ( e.g . neural networks).
Thus, the obtained successorfeatures on novel tasks might contain high approximation errors, which could make the GPI policyperform poorly.
Our high-level idea to tackle the issue is to exploit the reward decomposition structure in Equation (1)even for obtaining successor features for new tasks, instead of solely relying on the approximators.
We employ the lower and upper bounds on optimal values from Theorem 1 to enforce the boundson the approximate successor features.
As a result, the approximation errors can be reduced byrestricting the estimated optimal values to be inside those bounds around the optimal values, whichcan prevent the use of erroneous values during the transfer to unseen tasks.
For now, we will ﬁrst introduce how to train the successor features approximators to output thesuccessor features that satisfy the bounds on novel tasks.
Then, we will point out that an analogouseffect can be accomplished by modifying only the inference algorithm, and propose constrained GPIas a simple yet effective test-time approach to improving zero-shot transfer to novel tasks.
Constrained training of SF approximators .
In the original training of USFAs, the approximators arelearned with a set of source tasks T in Equation (7).
We propose to guide the training by employing
Theorem 1; we impose constraints for the approximators using the lower and upper bounds on theoptimal values for arbitrary linear combinations of source tasks.
Speciﬁcally, for the training of
USFAs, we use Equation (7) but with the following constraints:
[Equation]
where ( s, a ) is the same sample as the main objective of Equation (7).
W is a set of task vectorsfor the constraints, which can be independent of the source task set T , and ξ ( · ) determines the coefﬁcients α given a target task w (cid:48) and T .
We will explain later how to determine ξ ( · ) .
W can beany subset of the linear span of source task vectors, but practically, we randomly sample a number ofvectors from the span at each update.
Since the targets of the constraints are not ﬁxed with respect toboth w (cid:48) and ( s, a ) throughout the training, we use penalty terms (or soft constraints) that linearlypenalize the constraint violations as
[Equation]
where { x } + denotes max { x, 0 } .
The constrained training suggested above can make the approximators comply with the bounds forany tasks without requiring any additional interactions with the environment.
However, it has somedownsides.
First, since it is a new training procedure, existing pre-trained models cannot be used.
It requires some additional computational cost compared to the naive training of successor featuresapproximators.
Second, the enforcement of the constraints for training can introduce additionalhyperparameters ( e.g .
the weight coefﬁcient of the penalty terms).
Thus, their suboptimal choice mayintroduce instability in training or decrease performance.
Test-time constrained GPI .
Our idea starts from the observation that in the constrained training,the learned successor features from source tasks are considered the “trustworthy” features for theconstraints, because the USFAs are trained on the source tasks.
Besides, only the source successorfeatures are used for the computation of the constraints for all the other tasks.
It also implies thatit is better that the learning of the source successor features is not affected by other criteria andmore accurate source successor features would produce better constraints for other tasks with smallererrors.
Based on the implication, we propose constrained GPI , which can not only overcome the limitationof USFA as done by the aforementioned constrained training but also have two additional practicalmerits: (i) it is computationally simpler, and (ii) it is a test-time approach with no training.
Simplyput, we propose replacing the usual GPI policy with the constrained GPI policy as
[Equation]
where the target task w (cid:48) is expressible as a linear combination of the source tasks and ξ ( · ) againoutputs α given w (cid:48) and T as in Equation (18).
C is a set of policies that we can freely choose whenapplying the constrained GPI.
The constrained GPI policy selects the actions that maximize the maximum action-values as theoriginal GPI policy does but also caps the action-values by the lower and upper bound constraintsderived from the source successor features.
The upper bound constraint ﬁxes the overestimation ofvalues computed with approximate successor features for either the target task w (cid:48) or any other tasksused for constrained GPI.
The lower bound constraint ensures that action-values on the target taskused for the greedy action selection are at least as close to the optimal target action-values as thelower bounds.
The approximation error terms in the lower and upper bounds i.e ., (cid:15) π w w (cid:48) ( s, a ) and (cid:15) π w w ( s, a ) in
Theorem 1 could be ignored in practice, as long as the approximation errors of the source successorfeatures themselves can be considered to be small.
Also, we can obtain the tightest possible upperbound by deﬁning ξ ( · ) as
[Equation]
subject to w
[Equation]
w ∈T α w w .
The objective U w (cid:48) , T , { α w } w ∈T ( s, a ) is the sum of the piecewise linear functions, and thus Equa- 224 tion (20) can be solved with linear programming.
We observe that using the lower bound constraint with L w (cid:48) , T ( s, a ) is equivalent to including thesuccessor features of source tasks in the input to the constrained GPI; i.e ., T ⊆ C .
Also, since 227 L w (cid:48) , T ( s, a ) ≤ U w (cid:48) , T ,ξ ( w (cid:48) , T ,s,a ) ( s, a ) , there would be no difference between GPI and constrained
GPI when C = T .
Thus, in our experiments, we mainly use C = { w (cid:48) } , which is equivalent to using
C = T ∪ { w (cid:48) } .
[Figure]
<latexit sha1_base64="YJEsX3Uv5R7urvMZoEmxZ4YbCKo=">AAAE6HicjVNLbxMxEHbbACW8Wjhw4GKIKiVRG2UjXhWqVB4HLlWL1JcUR5XX67Ru/FjZXpLW2v4GbogrXOEH8FP4N3g3G9RNG4mREs3OzDef5xXGnBnbbv+Zm1+o3Lh5a/F29c7de/cfLC0/3Dcq0YTuEcWVPgyxoZxJumeZ5fQw1hSLkNODcPA+8x98ptowJXftWUx7Ah9L1mcEW286WnqMtLLY0lCN3Ho7dXXcgG85T
[Figure]
/kxbm1ssIh4UxZyL0kQLbEzPty4zX+bqJ7b/uOSbjxFJJxkT9hEOrYNYOGDFNieVnXsFEM
[Table]
/yDHMIWuveC6cEvlgv1fIu+Kn4P8WjbCkVz9zvqV97Tbf9ZuzEVGOrdNMhwSQT7Iymbqqdp3Nj8XgaW2jns7xYn3if51YFLDujZm5HT7IE5+W5TBBMzkUw6RH+2IPZ076sHHRawcvW8w+d2ubb4uyXwGPwFNRBAF6BTbAFdsE+IOAz+AF+gl8VXvlS+Vr5Ngm9tlBgHoKSVL7/BY6Etuk=</latexit
[Figure]
(a) “All” is the evaluation on the entire set of target task vectors from {− 1 , 1 } d , whereas (b) “Harsh” denotes the evaluation on a subset consisting of ‘harsh’ tasks, whose number of − 1 ’s is no less than that of 1 ’s ( e.g ., w = ( − 1 , 1) ).
Per each method, eight models trained for 1 M steps are evaluatedtimes for each target task with a ﬁxed set of 10 random seeds.
Scores for each task are normalized by the minimum and maximum scores across all the agents’ evaluation episodes on that task.
Experiments
4.1 Scavenger Experiments
We start our experiments in the Scavenger environment [7, 8], to assess the effectiveness of ourapproach with as minimal inﬂuence from external causes as possible.
In Scavenger, the agent ispositioned at one of the cells in a G × G grid, and the goal is to maximize the return by collectingobjects.
Not only the agent but also objects are spawned at random locations, and there are d classesof objects where the class determines the value of the reward for each object.
The state space is
S = { 0 , 1 } G × G × ( d +1) , where the ﬁrst d channels describe the current locations of the objects on themap and the last channel speciﬁes the walls where the agent cannot go and objects do not appear.
There are four actions available: A = { UP , DOWN , LEFT , RIGHT } , and the agent picks up an objectautomatically by visiting the cell of the object, which makes a new object of a random class appearat a random location.
Each element of the feature φ ( s, a, s (cid:48) )
∈
R d represents whether the agent ispicking up an object of that type or not with 1 or 0 , and the task vector w ∈ R d determines rewardvalues for the d different classes of objects.
Please see Barreto et al.
[8] for the full details.
We evaluate the zero-shot transfer performance of agents with different approaches i.e ., we measurethe performance of the same set of trained USFAs with GPI and constrained GPI policies on targettasks without any further policy updates.
We perform the evaluation with G = 11 , 20 objects intotal and different numbers of classes; d = 2 and d = 4 , and with d = 4 , we also include USFAsagents learned with the constrained training for a comparison.
We use the standard basis vectors of 249 R d as the set of source tasks similarly to [10], and evaluate agents on the set of target tasks deﬁnedby {− 1 , 1 } d .
Therefore, all the target tasks except for the all-ones vector, 1 , are not covered by theconical hull of source tasks, which requires Theorem 1 for bounding of values.
[Table]
We train eight USFAs agents for 1 M environmentsteps, and evaluate them on each target vector 10times with a ﬁxed set of 10 random seeds.
To evaluate them regardless of the reward scale differencebetween different tasks, we normalize the scores (orreturns) from the environment by the minimum andmaximum scores with respect to all the agents’ evaluation episodes on each task.
Figures 2 and 3 compare the performance of the USFAs agents with GPI and constrained GPI for exploitation, following the evaluation scheme suggestedby [2].
The results show that although they use the same set of trained USFAs, using constrained
GPI brings a notable performance improvement in comparison with the original GPI.
Also, Figure
[Figure]
[Table]
[Figure]
[Table]
[Figure]
suggests that constrained GPI, the test-time method, can match or even outperform the agents learnedwith the constrained training.
One possible explanation is that the constrained training might experience some instability in the learning depending on the choice of the hyperparameters, as described in
Section 3.2.
Table 1 presents the portion of the action-values after taking the maximum and resulting greedyactions that are changed by constrained GPI’s lower and upper bounds.
It implies that USFAs i.e ., thefunction approximators themselves may not satisfy the value bounds in Theorem 1 and applying thebounds could meaningfully improve the performance.
4.2 Robotic Locomotion Experiments
To evaluate our approach in a more realistic and physical domain, following [5, 21], we employ
Reacher, a MuJoCo’s robotic locomotion environment [27] from OpenAI Gym [11].
Reachersimulates a robotic arm with two joints, and its state space S is 11 -dimensional.
Its original actionspace is continuous and two-dimensional, which represents torques at the two hinge joints, and wediscretize each action dimension into three values resulting in nine discrete actions in total, as in [5].
For its use in the zero-shot transfer problem, we ﬁrst set four ﬁxed goal locations at(0 . 1 , 0 . 0) , (0 . 0 , 0 . 1) , ( − 0 .
1 , 0 . 0) , (0 . 0 , − 0 .
and deﬁne φ ( s, a, s (cid:48) )
∈ R 4 to be a vector whose elements are the negative distances between the agent’s ﬁngertip and the four goals.
USFAs agents aretrained with the four standard basis vectors as source tasks, learning to reach or get close to one ofthe four goals on each of the four source tasks.
However, for the evaluation, we deﬁne {− 1 , 1 } d tobe the set of target tasks.
A negative value at each dimension implies not only that the target vectoris outside the conical hull of the source tasks but also that the agent would obtain higher rewardswith respect to that dimension by getting away from the corresponding goal, instead of reaching thegoal.
This can make this zero-shot transfer problem challenging, as it requires the agent to do verydifferent behaviors suddenly at test time.
For better understanding, Figure 5 shows a rendered sceneof Reacher, where the four red dots represent the goal locations.
In our experiments, we train 16 USFAs agents for 1 M environment steps to obtain statistically moremeaningful results.
We evaluate each of the trained agents with 10 episodes for each target vector,again with a set of 10 environment random seeds.
Similarly to the Scavenger experiments in theprevious section, to take into account the difference in reward scales between different target tasks,for each of the target tasks, we normalize the returns i.e ., scores using the minimum and maximumscores that any of the agents achieve during the evaluation.
Figure 4 provides the comparison of GPI and constrained GPI in terms of the normalized scores.
As it shows, the inference with constrained GPI signiﬁcantly outperforms the ones with GPI on thezero-shot transfer problem.
Especially, the gap becomes even larger when they are evaluated on the
Harsh set of target task vectors, which implies that constrained GPI can be helpful for transferring todifferent types of target tasks outside the conical hull of source tasks.
On the other hand, Figure 7presents the performance proﬁles [2, 13] of the approaches with the matching evaluation categories
[Figure]
<latexit sha1_base64="YJEsX3Uv5R7urvMZoEmxZ4YbCKo=">AAAE6HicjVNLbxMxEHbbACW8Wjhw4GKIKiVRG2UjXhWqVB4HLlWL1JcUR5XX67Ru/FjZXpLW2v4GbogrXOEH8FP4N3g3G9RNG4mREs3OzDef5xXGnBnbbv+Zm1+o3Lh5a/F29c7de/cfLC0/3Dcq0YTuEcWVPgyxoZxJumeZ5fQw1hSLkNODcPA+8x98ptowJXftWUx7Ah9L1mcEW286WnqMtLLY0lCN3Ho7dXXcgG85T
[Figure]
Wau1WOxd4VQkKpQYK2TlaXviNIkUSQaUlHBvTDdqx7TmsLSOcplWUGBpjMsDHtOtViQU1PZdXkMIVb4lgX2n/kxbm1ssIh4UxZyL0kQLbEzPty4zX+bqJ7b/uOSbjxFJJxkT9hEOrYNYOGDFNieVnXsFEM
[Table]
fpqtXJNobqf9M1JmTqIlkIgq0H1Pehn85Wi/yDHMIWuveC6cEvlgv1fIu+Kn4P8WjbCkVz9zvqV97Tbf9ZuzEVGOrdNMhwSQT7Iymbqqdp3Nj8XgaW2jns7xYn3if51YFLDujZm5HT7IE5+W5TBBMzkUw6RH+2IPZ076sHHRawcvW8w+d2ubb4uyXwGPwFNRBAF6BTbAFdsE+IOAz+AF+gl8VXvlS+Vr5Ngm9tlBgHoKSVL7/BY6Etuk=</latexit
[Figure]
We refer the reader to Figure 2 for the full description.
[Figure]
and color mapping.
It qualitatively suggests that constrained GPI is more likely to achieve higherscores than GPI.
Conclusion and Discussion
We presented constrained GPI, a simple yet effective test-time approach for transfer with approximatesuccessor features.
We ﬁrst focused on the issue that although universal successor features approximators (USFAs) exploit the smoothness of optimal values across different tasks, their approximationerrors on novel target tasks could be large especially when those tasks are quite distant from sourcetasks.
Thus, we presented a theorem about lower and upper bounds on the optimal values for noveltasks which are expressible as linear combinations of source tasks, relaxing the conical combinationcondition used by the theorem by Nemecek and Parr [21].
We proposed the constrained trainingscheme making use of those bounds for reducing the action-value approximation errors of the learnedapproximators on novel tasks.
We then suggested constrained GPI and using the bounds only at testtime to achieve an analogous effect, allowing the use of existing trained models, and empiricallyshowed that this test-time approach can improve the zero-shot transfer performance by a large margin.
Limitations and future directions .
There may be cases where the minimum rewards for source tasksi.e .
, r min w ’s are overly small, which could lead to less changes of action-values and thus behaviorsinduced by the upper-bounding in Theorem 1.
Also, as our approach makes use of the successorfeatures framework, it needs φ to be observable or learned with a function approximator.
On the otherhand, extending our constrained GPI to other non-linear forms of reward or value decompositionscould be an intriguing direction for future research.
Making transfer with successor features andconstrained GPI compatible with large-scale approaches for generalization such as [24] might also bean interesting future direction.
We don’t see direct negative societal impacts of this work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will includein the supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We will include in the supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] We will include in thesupplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
N/ A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]