Constrained GPI for Zero-Shot Transfer in Reinforcement Learning
Abstract
For zero-shot transfer in reinforcement learning where the reward function varies between different tasks, the successor features framework has been one of the popular approaches.
However, in this framework, the transfer to new target tasks with generalized policy improvement (GPI) relies on only the source successor features [5] or additional successor features obtained from the function approximators’ generalization to novel inputs [11].
The goal of this work is to improve the transfer by more tightly bounding the value approximation errors of successor features on the new target tasks.
Given a set of source tasks with their successor features, we present lower and upper bounds on the optimal values for novel task vectors that are expressible as linear combinations of source task vectors.
Based on the bounds, we propose constrained GPI as a simple test-time approach that can improve transfer by constraining action-value approximation errors on new target tasks.
Through experiments in the Scavenger and Reacher environment with state observations as well as the DeepMind Lab environment with visual observations, we show that the proposed constrained GPI signiﬁcantly outperforms the prior
GPI’s transfer performance.
Our code and additional information are available at https://jaekyeom.github.io/projects/cgpi/ .
Introduction
For sequential decision making, deep reinforcement learning (RL) has been shown to be effective for various types of problems including games [29] and robotics [19, 21].
With such great successes, interest in multi-task RL has also been surged, where its goal is to train a single agent that can efﬁciently solve multiple varying tasks.
In multi-task RL, we focus on the transfer learning setting, where the agent learns shared structural knowledge from a set of source tasks during training and exploits and generalizes them in new, unseen target tasks at test time.
One popular approach to transfer in RL is to leverage the successor features (SFs) framework [1, 5, 6, 11, 24], which transfers policies learned on source tasks to target tasks, where the tasks share the same environment dynamics but differ in their reward functions.
Successor features build a representation of value functions decoupled from reward functions, and transfer to the tasks with arbitrary reward functions by taking an inner product with corresponding task vectors.
They utilize generalized policy improvement (GPI) [5], which is a generalized version of policy improvement by combining multiple policies, and provides the performance lower bounds for GPI policies.
However, GPI does not take into account any information from the smoothness of optimal actionvalue functions with respect to task vectors.
Tackling this issue, Borsa et al.
[11] propose universal successor features approximators (USFAs), which can estimate the optimal successor features for novel task vectors.
Nevertheless, the function approximator can make high approximation errors on the task vectors, especially when the new task vectors are distant from the source task vectors.
For instance, when USFAs are trained with source tasks to get close to given goals, they may not generalize well to the target tasks where the agent should get away from the given goals.
That is, if the elements of target task vectors have the opposite signs from the source task vectors, USFAs could output successor features with high approximation errors.
To improve the successor features approximation of USFAs for the new tasks, we aim at bounding value approximation errors on the new target tasks.
We ﬁrst introduce a new theorem on bounding the optimal values for the tasks that are expressible as linear combinations of source tasks.
Our theorem generalizes the conical combination condition used by the prior theorem by Nemecek and
Parr [24].
Using our new bounds as constraints, we can train the successor features approximators whose action-value approximation errors on novel tasks are bounded.
We extend this idea so that we accomplish a similar effect with no additional training; as a result, we propose constrained GPI as a test-time approach to bounding the approximation errors.
Despite its simplicity and no need for modiﬁcation to the training procedure, we empirically show that constrained GPI attains large performance improvements compared to the original GPI in multiple environments, including the
Scavenger [7, 8] and Reacher [12] environments with state observations and the DeepMind Lab [6, 9, 11] environment with ﬁrst-person visual observations.
Our main contributions can be summarized as follows:
• We present a novel theorem on lower- and upper-bounding optimal values for novel tasks that can be expressed as linear combinations of source tasks.
It extends and generalizes the previous theorem for conical combinations by Nemecek and Parr [24], to enable a broader application of the bounds.
• Based on our new theorem, we propose constrained GPI as a simple test-time approach that can improve transfer to novel tasks by constraining action-value approximation errors on new target tasks, with no modiﬁcation to the training procedure.
• We empirically show that our approach can improve the performance over the GPI baselines by large margins in the Scavenger, Reacher and DeepMind Lab environments.
We also provide analyses for better understanding of the results of our method.
Preliminaries
We describe the problem setting and background on successor features and universal successor features approximators.
We refer the reader to Appendix for in-depth discussion of related work.
2.1 The Zero-Shot Transfer Problem in RL
We deﬁne a Markov Decision Process (MDP) as M ≡ ( S , A , P, R, γ ) .
S and A are the state and action spaces, respectively.
P ( ·| s, a ) deﬁnes the transition probability distribution of the next states given s ∈ S and a ∈ A .
R ( s, a, s (cid:48) ) is the reward for taking action a at state s resulting in s (cid:48) , and γ ∈ (0 , 1] is the discount factor.
We assume that rewards are bounded.
We consider the zero-shot transfer problem; as in [5], each task is deﬁned by its task vector w ∈ R d , and only the reward functions differ across tasks, being decomposed as
[Equation]
where φ ( s, a, s (cid:48) )
∈ R
d is the features of ( s, a, s (cid:48) ) .
We denote the set of source task vectors as T , which is used for training.
At test time, we evaluate the transferred policy on each target task w (cid:48) / ∈ T with no additional update of pre-trained components.
We examine both the possible scenarios: (i) the features φ ( s, a, s (cid:48) ) are available to the agent [5, 11] and (ii) no pre-deﬁned features are available and the agent needs to construct its own features and task vectors.
We ﬁrst introduce the formulation for (i) in Section 2.2 and then its variant for (ii) in Section 2.3.
2.2 Successor Features and Universal Successor Features Approximators
We now review successor features (SFs)
[5] and how they are transferred to different tasks.
Equation (1) allows expressing the action-value function for policy π on task w as
Q
[Equation]
where φ t = φ ( s t , a t , s t +1 ) ∈ R d .
Here, ψ π ( s, a
) ∈ R d is called the SFs for policy π at ( s, a ) , and taking its inner product with an arbitrary task w results in the action-value for π on w ; i.e .,
Q π w (s, a).
Thanks to the analogy between (rewards r , action-value functions Q ) and (features φ , successor features ψ ), the Bellman equation applies to the SFs and thus they can be trained similarly as action-value functions are learned; e.g ., Q-learning.
The policy improvement theorem [10] states that a new policy that takes a greedy action according to a given policy’s value function at each state performs at least as good as the original policy.
Generalized policy improvement (GPI) [5] extends policy improvement to the case where the value functions of multiple policies are available.
Given a task w (cid:48) , a set of policies π 1 , . . .
, π n , their action-value functions Q π 1 w (cid:48) , . . .
, Q π n w (cid:48) and their approximations ˜ Q π 1 w (cid:48) , . .
˜ Q π n w (cid:48) , the GPI policy is deﬁned as
[Equation]
Barreto et al.
[5] suggest that Q
[Equation]
∞ .
They also provide the upper bound on the suboptimality of the GPI policy as
[Equation]
where each π i is an optimal policy for w i .
While the GPI theorem allows the transfer of learned successor features to arbitrary tasks that share the same environment dynamics, it is limited in the following aspect.
GPI uses the action-values for source tasks on target tasks based on the reward decomposition assumption (Equation (1))
i.e ., ˜
Q π i w (cid:48) ( s, a ) =
˜
ψ π i ( s, a ) (cid:62) w (cid:48) for each i .
However, it does not take any advantage of the smoothness of the optimal action-value functions with respect to different task vectors [11].
To overcome this limitation, Borsa et al.
[11] introduce universal successor features approximators (USFAs).
Inspired by universal value functions (UVFs) [28], they extend the original successor features with policy vectors z ∈ R l as input to their approximators.
More speciﬁcally, universal successor features (USFs) are deﬁned to satisfy
[Equation]
where z is a policy vector for the policy π z , and USFAs ˜ ψ are the learned approximators of USFs ψ .
Naturally, the value functions are expressed as
[Equation]
Each reward function induces optimal policies, which can be encoded using the corresponding task vectors.
That is, one can simply choose to deﬁne the policy vector space to be the same as the task vector space ( l = d ) and let z = w be a policy vector of an optimal policy for task w .
Then, π w
and
Q π w w denote an optimal policy for w and its action-value function, respectively.
The training of USFAs is similar to that of SFs, except for that it additionally involves sampling of policy vectors given task vectors.
The update of USFAs at the k -th iteration is ˜ ψ ( k +1) ← argmin
[Equation]
(7) for a (cid:48) =
argmax b ˜ ψ ( k ) ( s (cid:48) , b, z ) (cid:62) z .
D z ( ·| w ) is the policy vector distribution; for instance, N ( w , σI ) can be used for better training with diversiﬁed inputs.
µ is the transition sampling distribution, which involves the GPI policy of the samples from D z ( ·| w ) or a replay buffer.
We use gradient descent to update the parameters.
USFAs provide a beneﬁt that they allow a GPI policy to use an arbitrary set of policies { π z } z ∈C as π GPI ( s ) ∈ argmax a max z ∈C ˜ Q π z w (cid:48) ( s, a ) .
However, the generalization of USFAs to new policy vectors depends on a function approximator ψ , and thus if C contains policy vector(s) distant from source vectors, a GPI policy with C may have high approximation errors and perform poorly or even worse than a GPI policy with only source vectors [11], as will be demonstrated later in our experiments.
2.3 Universal Successor Features Approximators with Learned φ
For the scenario where features φ ’s are not provided to the agent 1 , we adopt the problem formulation from Ma et al.
[22] where for each task the task information g ∈ G is given to the agent.
Although the task information g , unlike a task vector, cannot be directly combined with successor features for transfer to a novel task, zero-shot inference could still be possible by leveraging the information about the task.
Speciﬁcally, we not only perform the original learning of ˜ ψ
but using the task information as the policy vector input instead, but also train ˜ φ and ˜ w to approximate the reward decomposition with transition samples.
As done in [22], we update ˜ ψ , ˜ φ and ˜ w using gradient descent to minimize E g ∼T g , z ∼D gz (
[Equation]
L ψ + L Q (cid:3) for
[Equation]
[Equation]
, a (cid:48) , z ) (cid:62) ˜ w ( k ) ( z ) − ˜ ψ ( s, a, z ) (cid:62) ˜ w ( z ) (cid:111)(9) and a (cid:48)
=
argmax b ˜ ψ ( k ) ( s (cid:48) , b, z ) (cid:62) ˜ w ( k ) ( z ) at the k -th iteration.
The superscript ( k ) denotes the target, T g is the source task information set, D gz ( ·| g ) is the policy vector distribution conditioned on the task information and µ is the sampling distribution.
Constrained GPI for Improved Zero-Shot Transfer of Successor Features
[Table]
+AD2AX7gIAh+AF+gl+1pPal9rX2bRw6P1diHoGK1L7/BWVotDE=</latexit> 2 ○
While the conical combinations of wand w 2 covers only area 1 (cid:13) , the linear combinations covers both area 1 (cid:13) – 2 (cid:13) .
To mitigate the aforementioned issue of the possibly unlimited approximation errors of USFAs, we propose a simple yet effective method that improves the transfer of successor features by further leveraging the reward decomposition structure in Equation (1).
We ﬁrst present, under a more relaxed condition, lower and upper bounds on the optimal values for novel task vectors that are expressed as linear combinations of source task vectors (Section 3.1).
Then, we propose a novel approach called constrained GPI , which effectively conﬁnes the approximated action-values inside the computed lower and upper bounds (Section 3.2).
3.1 Bounding Optimal Values for New Tasks
Theorem 1 of [24] provides the lower and upper bounds on the value of an optimal policy for a new task, whose vector w (cid:48) is a positive conical combination of source task vectors i.e ., w (cid:48)
= (cid:80) w ∈T
α w w such that α w ≥ 0 , ∀ w ∈ T and (cid:80) w ∈T α w > 0 2 .
However, for a broad application of such bounds, the positive conical combination condition can be too restrictive, since the resulting bounds only apply to the task vectors that appear inside the conical hull of source task vectors.
Therefore, we suggest a more relaxed theorem, which holds for an arbitrary task vector w (cid:48) that is expressed as a linear combination of the source task vectors i.e ., w (cid:48) =
(cid:80) w ∈T α w w for α w ∈ R , ∀ w ∈ T .
Figure 1 compares an example of the task space coverage of conical [24] and our linear combinations.
With our extended task space coverage, we can apply the bounds to more general target tasks outside of the conical hull, which will be further discussed in the next section.
We deﬁne (cid:15) π w 1 w 2 to be an upper bound on the approximation error of ˜ Q π w 1 w 2 for arbitrary tasks w 1 , wsuch that
[Equation]
and we present our theorem as follows.
Theorem 1.
Given a task vector w (cid:48) =
(cid:80) w ∈T α w w for α w ∈ R , ∀ w ∈ T , for all state-action pairs ( s, a ) ∈ S × A , the action-value of π w (cid:48) , which is an optimal policy for task w (cid:48) , on task w (cid:48) is lowerand upper-bounded as L w (cid:48) , T ( s, a ) ≤ Q π w (cid:48) w (cid:48)
( s, a ) ≤ U w
[Equation]
, T , α ( s, a ) for
[Equation]
where r min w is the minimum reward on w i.e ., r min w = min ( s,a ) ∈S×A R w ( s, a ) and α = { α w } w ∈T .
Proof.
For the derivation of the lower bound L w (cid:48) , T ( s, a ) , since Q π w (cid:48) w (cid:48) is the optimal action-value function for task w (cid:48) and Q π w (cid:48) w (cid:48)
( s, a ) ≥ Q π w w (cid:48)
( s, a ) for arbitrary task w and state-action pair ( s, a ) ,
[Equation]
For the upper bound U w (cid:48) , T , α ( s, a ) , we use that Q π w (cid:48) w ( s, a ) ≤ Q π w w ( s, a ) and Q π w (cid:48)
w ( s, a ) ≥ min π Q π w ( s, a ) ≥ 11 − γ r min w for arbitrary task w and state-action pair ( s, a ) , which leads to
Q
[Equation]
w
[Equation]
w
[Equation]
In Equation (19), for each w ∈ T , the sign of α w determines which of the two terms is used given to the max operator.
If α w ≥ 0 , the max operator selects the ﬁrst term, whereas a negative α w makes the second term used.
Note that our Theorem 1 recovers Theorem 1 of [24] when w (cid:48) is a conical combination of w ’s from T i.e ., α w ≥ 0 , ∀ w ∈ T .
Intuitively, this theorem states the condition that the optimal action-value for an arbitrary target task must satisfy, given the optimal action-values for the source tasks.
The theorem is applicable to the problems whenever bounding of optimal values is useful.
One example is policy cache construction, where the agent should decide whether to reuse existing policies in the cache set or learn a new one [24].
As will be shown in the next section, we employ the bounding as a constraint on the actionvalues for novel target tasks, for the guidance of transfer.
In Sections 4.1 and 4.2 and appendix B, we empirically show that the application of our Theorem 1 can signiﬁcantly improve the performance in the cases where target tasks are outside the conical hull of source tasks.
3.2 Constrained Training and Constrained GPI
As described in Section 2.2, the universal successor features approximators (USFAs)
[11] improve the original successor features so that arbitrary policy vectors, including the ones for target tasks, can be used for GPI.
However, the use of arbitrary policy vectors with USFAs solely relies on the generalization power of the approximators ( e.g ., neural networks).
Thus, the obtained successor features on novel tasks might contain high approximation errors, which could make the GPI policy perform poorly.
Our high-level idea to tackle the issue is to exploit the reward decomposition structure in Equation (1) even for obtaining SFs for new tasks, instead of solely relying on the approximators.
We employ the lower and upper bounds on optimal values from Theorem 1 to enforce the bounds on the approximate successor features.
As a result, the approximation errors can be reduced by restricting the estimated optimal values to be inside those bounds around the optimal values, which can prevent the use of erroneous values during the transfer to unseen tasks.
For now, we will ﬁrst introduce how to train the successor features approximators to output the successor features that satisfy the bounds on novel tasks.
Then, we will point out that an analogous effect can be accomplished by modifying only the inference algorithm, and propose constrained GPI as a simple yet effective test-time approach to improving zero-shot transfer to novel tasks.
Constrained training of SF approximators .
In the original training of USFAs, the approximators are learned with a set of source tasks T in Equation (7).
We propose to guide the training by employing
Theorem 1; we impose constraints for the approximators using the lower and upper bounds on the optimal values for arbitrary linear combinations of source tasks.
Speciﬁcally, for the training of
USFAs, we use Equation (7) but with the following constraints:
[Equation]
where ( s, a ) is the same sample as the main objective of Equation (7).
W is a set of task vectors for the constraints, which can be independent of the source task set T , and ξ ( · ) determines the coefﬁcients α given a target task w (cid:48) and T .
We will explain later how to determine ξ ( · ) .
W can be any subset of the linear span of source task vectors, but practically, we randomly sample a number of vectors from the span at each update.
Since the targets of the constraints are not ﬁxed with respect to both w (cid:48) and ( s, a ) throughout the training, we use penalty terms (or soft constraints) that linearly penalize the constraint violations as
[Equation]
where { x } + denotes max { x, 0 } .
The constrained training suggested above can make the approximators comply with the bounds for any tasks without requiring any additional interactions with the environment.
However, it has some downsides.
First, since it is a new training procedure, existing pre-trained models cannot be used.
It requires some additional computational cost compared to the naive training of successor features approximators.
Second, the enforcement of the constraints for training can introduce additional hyperparameters ( e.g .
the weight coefﬁcient of the penalty terms).
Thus, their suboptimal choice may introduce either instability in training or decrease performance.
Test-time constrained GPI .
Our idea starts from the observation that in the constrained training, the learned successor features from source tasks are considered the “trustworthy” features for the constraints, because the USFAs are trained on the source tasks.
Besides, only the source successor features are used for computing the constraints for all the other tasks.
It implies that the learning of the source successor features is not affected by other criteria, and more accurate source successor features would produce better constraints for other tasks with smaller errors.
Based on the implication, we propose constrained GPI , which can not only overcome the limitation of USFA as done by the aforementioned constrained training but also have two additional practical merits: (i) it is computationally simpler, and (ii) it is a test-time approach with no training.
Simply put, we propose replacing the usual GPI policy with the constrained GPI policy as
[Equation]
where the target task w (cid:48) is expressible as a linear combination of the source tasks and ξ ( · ) again outputs α given w (cid:48) and T as in Equation (20).
C is a set of policies that we can freely choose when applying the constrained GPI.
The constrained GPI policy selects the actions that maximize the maximum action-values as the original GPI policy does but also caps the action-values by the lower and upper bound constraints derived from the source successor features.
The upper bound constraint ﬁxes the overestimation of values computed with approximate successor features for either the target task w (cid:48) or any other tasks used for constrained GPI.
The lower bound constraint ensures that action-values on the target task used for the greedy action selection are at least as close to the optimal target action-values as the lower bounds.
The approximation error terms in the lower and upper bounds i.e ., (cid:15) π w w (cid:48) ( s, a ) and (cid:15) π w w ( s, a ) in
Theorem 1 could be ignored in practice, as long as the approximation errors of the source successor features are sufﬁciently small.
Also, we can obtain the tightest upper bound by deﬁning ξ ( · ) as
[Equation]
) subject to w (cid:48)
[Equation]
The objective U w (cid:48) , T , { α w } w ∈T ( s, a ) is the sum of the piecewise linear functions.
Thus, Equation (22) can be solved with linear programming.
We observe that using the lower bound constraint with L w (cid:48) , T ( s, a ) is equivalent to including the successor features of source tasks in the input to the constrained GPI; i.e ., T ⊆ C .
Also, since L
[Equation]
( s, a ) , there would be no difference between GPI and constrained
GPI when C = T .
Thus, in our experiments, we mainly use C = { w (cid:48) } , which is equivalent to using
C = T ∪ { w (cid:48) } .
Experiments 4.1 Scavenger Experiments
We start our experiments in the Scavenger environment [7, 8], which can assess our approach with as minimal inﬂuence from external causes as possible.
In Scavenger, the agent is positioned at one of the cells in a G × G grid, and the goal is to maximize the return by collecting objects.
Both the agent and objects are spawned at random locations, and there are d classes of objects where the class determines the value of the reward.
The state space is S
= { 0 , 1 } G × G × ( d +1) , where the ﬁrst d channels describe the current locations of the objects on the map and the last channel speciﬁes the walls where the agent cannot go and objects do not appear.
There are four actions available:
A = { UP , DOWN , LEFT , RIGHT } , and the agent picks up an object by visiting the cell of the object, which makes a new object of a random class of object appear at a random location.
The feature φ ( s, a, s (cid:48) ) ∈ { 0 , 1 } d is a one-hot vector whose element represents whether the agent is picking up an object of that type or not.
The task vector w ∈ R d determines reward values for the d different classes of objects.
Please see Barreto et al.
[8] for the full details.
We evaluate the zero-shot transfer performance of the agents with different approaches i.e ., we ﬁrst train the same set of USFAs as proposed in [11], and measure the performance between GPI and constrained GPI policies on target tasks with no further policy updates.
We test with G = 11 , andobjects in total with the different numbers of classes; d = 2 and d = 4 .
With d = 4 , we also include
USFAs agents learned with the constrained training for a comparison.
We use the standard basis vectors of R d as the set of source tasks similarly to [11], and evaluate agents on the set of target tasks deﬁned by {− 1 , 1 } d .
Therefore, all the target tasks except for the all-ones vector, 1 , are not covered by the conical hull of source tasks, which requires Theorem 1 for bounding of values.
We train eight USFAs agents for 1 M steps, and evaluate them on each target vector 10 times with a ﬁxed set of 10 random seeds.
To be invariant to the reward scale difference between different tasks, we normalize the scores (or returns) from the environment by the minimum and maximum scores with respect to all the agents’ evaluation episodes on each task.
Figures 2 and 3 compare the performance of the USFAs agents with GPI and constrained GPI for exploitation, following the evaluation scheme suggested by [2].
The results show that although they use the same set of trained USFAs, using constrained GPI brings a notable performance improvement
[Figure] [Figure]
in comparison with original GPI.
Also, Figure 3 suggests that constrained GPI, the test-time method, can match or even outperform the agents learned with the constrained training.
One possible explanation is that the constrained training might experience some instability in learning depending on the choice of the hyperparameters, as described in Section 3.2.
[Table]
d =23 .
11% d =5 .
Table 1 presents the portion of the action-values after taking the maximum and resulting greedy actions that are changed by the lower and upper bounds for constrained GPI, measured for the evaluation on Scavenger.
It implies that USFAs i.e ., the function approximators of successor features themselves may not satisfy the optimal value bounds presented in Theorem 1, and applying the bounds could meaningfully improve the performance.
[Figure]
4.2 DeepMind Lab Experiments with Learned φ
For evaluation of our approach in a more complex and realistic setting, we employ DeepMind Lab [6, 9, 11] and conduct experiments in a ﬁrst-person view 3D environment.
In a single room, a goal object is randomly placed, and the objective is to reach the goal before the episode ends where its location changes between tasks.
Figure 4 shows an example scene that the agent sees, where the goal object is shown.
The object is the goal.
At each time step, the agent observes an 84 × 84 × 3 image from the environment and outputs one of 45 possible actions, where there are 5 , 3 and 3 choices for LOOK_LEFT_RIGHT_PIXELS_PER_FRAME ,
STRAFE_LEFT_RIGHT and MOVE_BACK_FORWARD controls, respectively.
As observations are in the ﬁrst-person view, the goal object may not be seen by the agent, which makes transfer given the task information g critical to the success of the tasks.
In each task, the goal object is placed in one of equally sized cells treating the room as an 11 ×grid, and the task information g is a two-dimensional vector that contains the coordinate of the goal in the grid.
Starting at the center of the room, the agent receives a reward of one if it reaches the goal within the episode horizon or no rewards otherwise.
Therefore, the reward functions are sparse.
For these experiments where the agent observes rendered images rather than the underlying states, it may not be viable to deﬁne features φ ’s and task vectors w ’s that linearly decompose reward functions.
Therefore, we train agents with the learning of ˜ φ and ˜ w from samples from the source tasks with d = 2 as described in Section 2.3.
Inspired by Hong et al.
[17], we examine zero-shot transfer with GPI and constrained GPI using two transfer settings: “left-to-right” and “near-to-far”.
In the “left-to-right” setting, the agent is trained on the source tasks whose goals are sampled from the left half of the room and is tested on the target tasks with goals from the right half.
In the “near-to-far” setting, the source tasks have the goals within a L ∞ = 2 distance from the center of the room, and target tasks set the goals farther than L ∞ .
For each setting, we train eight USFAs agents with different seeds for 3 M environment steps on the source tasks and test them on the target tasks.
Figure 5 presents the comparison of GPI with different
C ’s and constrained GPI.
As the results suggest, by leveraging the same set of trained USFAs with learned ˜ φ and ˜ w , constrained GPI outperforms GPI with the three C ’s in both settings by a notable margin.
Another observation is that the trained USFAs agents seem to overﬁt more to the source tasks in the “near-to-far” setting compared to the “left-to-right” setting.
It makes the performance of the agent on the test tasks much worse.
On ther other hand, the presented results show that constrained
GPI is still helpful in such overﬁtting situations.
Conclusion and Discussion
We presented constrained GPI, a simple yet effective test-time approach for transfer with approximate successor features.
We ﬁrst focused on the issue that although universal successor features approximators (USFAs) exploit the smoothness of optimal values across different tasks, their approximation errors on novel target tasks could be large especially when those tasks are quite distant from source tasks.
Thus, we introduced a theorem about lower and upper bounds on the optimal values for novel task vectors that belong to the task vector space linearly spanned by the set of source task vectors, relaxing the conical combination condition used for the theorem by Nemecek and Parr [24].
We proposed a constrained training scheme making use of those bounds for reducing the action-value errors of the learned approximators on novel tasks.
We then suggested constrained GPI that uses the bounds at test time to achieve an analogous effect, allowing the use of previously trained models.
We empirically showed that this test-time approach can improve the zero-shot transfer performance by a large margin in multiple environments.
Limitations and future directions .
There may be some cases where the minimum rewards for source tasks i.e ., r min w ’s are overly small, which could lead to less changes of both action-values and behaviors induced by the upper-bounding in Theorem 1.
An interesting direction to tackle the issue is to learn the minimum action-value function during the training and to use the approximate minimum value at each state-action pair instead of 11 − γ r min w for deriving the upper bound in Theorem 1.
It may allow computing upper bounds adaptively for different state-action pairs.
Also, if the learned successor features approximators have large errors even on source tasks, not only GPI but also constrained GPI’s bounding may not be meaningfully helpful.
One idea to mitigate the issue is to take the uncertainty in the approximators and the approximation error term that appears in Theoreminto account, e.g ., by using ensemble models.
As an intriguing direction for future research, we could extend our constrained GPI to other non-linear forms of reward or value decompositions.
It may be also interesting to make transfer with successor features and constrained GPI compatible with large-scale approaches for generalization such as [27].
We do not see direct negative societal impacts of this work.
Acknowledgements
We thank the anonymous reviewers for their valuable comments.
This work was supported by Samsung Advanced Institute of Technology, and Institute of Information & communications Technology
Planning & Evaluation (IITP) grants funded by the Korea government (MSIT), including (No.20190-01082, SW StarLab), (No.2022-0-00156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation), and (No.2021-0-01343,
Artiﬁcial Intelligence Graduate School Program (Seoul National University)).
Jaekyeom Kim was partly supported by Google PhD Fellowship.
Gunhee Kim is the corresponding author.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will include in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We will include in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We will include in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Related Work
Transfer in reinforcement learning aims at solving a new target task with no additional learning or sample-efﬁciently by exploiting agents and information obtained from source tasks.
We review a line of research with relevant approaches.
Transfer by reusing policies .
This group of approaches reuses policies learned on source tasks for target tasks.
Fernández and Veloso [16] suggest an exploration strategy for the learning of a new policy given a new task and learned source policies, where the gain of using each policy is estimated together on-line and one of the policies in the set is selected probabilistically at each step, based on the gain, but they focus on aiding the training of the target policy with samples from the target task rather than improving the zero-shot transfer performance.
On the other hand, Dayan [13] introduce successor representations (SRs), state space occupancy representations disentangled from rewards, which allow linear decomposition of value functions.
Barreto et al.
[5] propose successor features (SFs) and using SFs as an extension of SRs.
Especially, in addition to SFs, which can be combined with arbitrary task vectors for obtaining the corresponding values, Barreto et al.
[5] also suggest generalized policy improvement (GPI), allowing composition of multiple source policies on a single task where the resulting GPI policy performs at least as well as any of the source policies.
The transfer with SFs and GPI and a number of connected methods [6, 8] combine source policies using the reward decomposition structure but are limited in that they do not further make use of the smoothness of the optimal value functions with respect to different tasks i.e ., given two similar tasks, their values are likely to be close to each other.
Nemecek and Parr [24] maintain a set of policies and determine whether to learn a new one for a given task or exploit existing ones based on their optimal value bounds, which is a different problem from ours, and they also target tasks expressible as conical combinations of source tasks, whereas we target linear combinations.
Alver and Precup [4] suggest sets of assumptions and conditions under which a group of basis policies can induce GPI policies that maximizes undiscounted returns on novel tasks and an iterative algorithm for constructing the basis, but thus they tackle a different problem, many of whose requirements do not apply to more general settings or our problem, due to e.g ., (non-binary) continuous reward features φ , no guarantee on possible trajectories, stochastic transition dynamics, etc.
Differently from the problem we are tackling, in the maximum entropy setting Hunt et al.
[18] aim at compositing source policies for target tasks optimally by estimating and correcting the divergence between source policies, but they consider target tasks whose rewards are convex combinations of source task rewards and the divergence estimation becomes signiﬁcantly harder when there are more than two source tasks [18].
Transfer with function approximation .
There is also a series of studies that directly exploits the smoothness of optimal values across tasks with function approximators.
Schaul et al.
[28] propose universal value functions and their approximators, which incorporate goals into their input, and use the approximators for generalization to novel goals.
Inspired by [28], Borsa et al.
[11] suggest universal successor features approximators (USFAs), which allow the use of GPI with arbitrary approximate policies by extending the original successor features approximators to take policy vectors as their inputs.
However, the generalization to novel or distant policy vectors with the function approximators could result in successor features with large approximation errors, and the resulting performance may be low.
In this work, we tackle this problem by leveraging the reward decomposition structure for bounding estimated values around optimal values and thus their errors.
On the other hand, Hong et al.
[17] propose bilinear value networks (BVN) with a bilinear decomposition of value functions into goal-agnostic and goal-speciﬁc components for better sample efﬁciency in multi-goal reinforcement learning.
They provide an alternative formulation of value function approximators differently from the decomposition used in the successor features framework, and thus our approach of bounding approximate values at test time is orthogonal to BVN.
Compared to the existing work mentioned above in general, our constrained GPI is a simple test-time approach for target tasks which are expressible as linear combinations of source tasks.
B Robotic Locomotion Experiments
To evaluate our approach in a physical domain, following [5, 24], we employ Reacher, a MuJoCo’s robotic locomotion environment [30] from OpenAI Gym [12].
Reacher simulates a robotic arm with two joints, and its state space S is 11 -dimensional.
Its original action space is continuous and
[Figure] [Figure]
two-dimensional, which represents torques at the two hinge joints, and we discretize each action dimension into three values resulting in nine discrete actions in total, as in [5].
For its use in the zero-shot transfer problem, we ﬁrst set four ﬁxed goal locations at (0 . 1 , 0 . 0) , (0 . 0 , 0 . 1) , ( − 0 .
1 , 0 . 0) , (0 . 0 , − 0 .
and deﬁne φ ( s, a, s (cid:48) )
∈ R 4 to be a vector whose elements are the negative distances between the agent’s ﬁngertip and the four goals.
USFAs agents are trained with the four standard basis vectors as source tasks, learning to reach or get close to one of the four goals on each of the four source tasks.
However, for the evaluation, we deﬁne {− 1 , 1 } d to be the set of target tasks.
A negative value at each dimension implies not only that the target vector is outside the conical hull of the source tasks but also that the agent would obtain higher rewards with respect to that dimension by getting away from the corresponding goal, instead of reaching the goal.
This can make this zero-shot transfer problem challenging, as it requires the agent to do very different behaviors suddenly at test time.
For better understanding, Figure 7 shows a rendered scene of Reacher, where the four red dots represent the goal locations.
In our experiments, we train 16 USFAs agents for 1 M environment steps to obtain statistically more meaningful results.
We evaluate each of the trained agents with 10 episodes for each target vector, again with a ﬁxed set of 10 environment random seeds.
Similarly to the Scavenger experiments in the previous section, to take into account the difference in reward scales between different target tasks,
− 0 .0 .0 .0 .0 .
( a ) S c
o r e ga i n v s .
G P
I - S T
GPI-S GPI-T
GPI-ST CGPI-T (ours) ( - 1 . 0 , - 1 . 0 , - 1 . 0 , - 1 . 0 ) ( - 1 . 0 , - 1 . 0 , - 1 . 0 , 1 . 0 ) ( - 1 . 0 , - 1 . 0 , 1 . 0 , - 1 . 0 ) ( - 1 . 0 , - 1 . 0 , 1 . 0 , 1 . 0 ) ( - 1 . 0 , 1 . 0 , - 1 . 0 , - 1 . 0 ) ( - 1 . 0 , 1 . 0 , - 1 . 0 , 1 . 0 ) ( - 1 . 0 , 1 . 0 , 1 . 0 , - 1 . 0 ) ( - 1 . 0 , 1 . 0 , 1 . 0 , 1 . 0 )
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
i o
[Figure]
denote using the set of source task vectors, the target task vector and both as C , respectively.
(a) presents the interquartile means (IQMs) of their normalized scores, visualized as performance gains (or differences) compared to GPI-ST , where the error bars are the 95% conﬁdence intervals.
Action change in (b) shows the average portion of the ﬁnal actions changed by constrained GPI for each target task, during the evaluation.
Lower-bounding and Upper-bounding in (b) denote the average ratios of the action-values after taking the maximums changed by constrained GPI’s lower and upper bounds, respectively.
for each of the target tasks, we normalize the returns i.e ., scores using the minimum and maximum scores that any of the agents achieve during the evaluation.
Figure 6 provides the comparison of GPI and constrained GPI in terms of the normalized scores.
As it shows, the inference with constrained GPI signiﬁcantly outperforms the ones with GPI on the zero-shot transfer problem.
Especially, the gap becomes even larger when they are evaluated on the “Harsh” set of target task vectors, which implies that constrained GPI can be helpful for transferring to different types of target tasks outside the conical hull of source tasks.
On the other hand, Figurepresents the performance proﬁles [2, 14] of the approaches with the matching evaluation categories and color mapping.
It qualitatively suggests that constrained GPI is more likely to achieve higher scores than GPI.
C Further Empirical Analyses
We provide further analyses of the experimental results for a better understanding of our approach.
Figure 10 visualizes four quantities with respect to the target task vectors, in the Reacher environment.
Figure 10a reports the performance gains of the approaches compared to GPI-ST in terms of the interquartile means (IQMs) of the normalized scores on each target task.
In Figure 10b, Action change shows the average ratio of the action changes due to constrained GPI, for each of the target tasks.
Lower-bounding and Upper-bounding are the average portions of the maximum actionvalues for the inference that are bounded and changed by constrained GPI’s lower and upper bounds, respectively.
We restate that the USFAs are trained with the four standard basis task vectors i.e ., a (positive) one-hot vector for each of the four dimensions of the task vector space, R 4 .
Our ﬁrst observation is that while the transferred agents perform comparably on some tasks, constrained GPI makes signiﬁcant differences on the others, especially more on the “Harsh” target tasks with many − 1 ’s as elements in their task vectors.
It implies that the USFAs, the function approximators, might work reasonably on tasks to which the approximations could be extrapolated similarly as in the source tasks, but they could perform poorly when there is little or no knowledge easily transferable from the source tasks only with the function approximators.
It also coincides with our second observation: examining the two plots in Figure 10 together, there is a tendency that more action changes by constrained GPI usually result in greater performance gains.
This suggests that bounding the action-values at test time with constrained GPI for reducing the value approximation errors often has a meaningful effect on the resulting performance, and the highest increases in both the performance gains and the action change ratios are observed on the “Harsh” target tasks.
Thus, we infer that the bounds from Theorem 1 could be effective for constraining the approximation errors of USFAs, especially including target tasks outside the conical combinations of source tasks.
Additionally, Lower-bounding and Upper-bounding in Figure 10b indicate that a large portion of the action changes and thus the performance gains are related to the upper-bounding in the Reacher environment, and the reduction of the USFAs’ overestimation with constrained GPI is an important factor of the performance gains, in this case.
Depending on the underlying environment and tasks, there can be target tasks where USFAs underestimate the corresponding action-values and the lowerbounding, which is equivalent to performing GPI with the source task set T included in the input policy set C , could help improve the resulting performance, as well.
D Experimental Details D.1 The Scavenger Environment
For the Scavenger experiments, we use the environment from [7, 8].
In addition to the environment settings explained in Section 4.1, we use a maximum episode horizon of 50 and a discount factor of 0 . 9 .
D.2 The Reacher Environment
Regarding the Reacher experiments, we employ the environment from OpenAI Gym [12].
Besides the environment conﬁgurations described in Appendix B,we set the maximum episode horizon asand the discount factor as 0 . 9 .
Since original states deﬁned for Reacher contain information about target coordinates, we set those coordinates to zeros for our experiments with different tasks.
D.3 The DeepMind Lab Environment
For the DeepMind Lab experiments, additionally to the experimental setup introduced in Section 4.2, we render 10 frames per second and use an episode horizon of 50 and a discount factor of 0 . 99 .
As the tasks we use are sparse-reward tasks, we sample each episode with one source task vector during the training.
D.4 Universal Successor Features Approximators and Constrained GPI
We employ the autonomous learning library [25] and PyTorch [26] for the implementation of USFAs.
The following selection of hyperparameters was done based on the performance of the USFAs on source tasks, because we have no access to target tasks during the training in the transfer problem and the learning on the source tasks is important for the transfer with both GPI and constrained GPI.
For both of the Scavenger and Reacher environments, we use an MLP with the ReLU nonlinearity and two hidden layers whose sizes are (128 , 256) , which is chosen over (64 , 128) and (256 , 256) , as USFAs.
We set the number of output heads of the networks to the respective number of discrete actions; four for Scavenger and nine for Reacher.
For the training of the USFAs, we make use of the
Adam optimizer [20] with a learning rate of 1 e − 4 , which is selected out of { 3 e − 4 , 1 e − 4 , 3 e − 5 } .
The target update frequency is set to 100 for Scavenger and 500 for Reacher, where we consider { 100 } for Scavenger and { 100 , 500 } for Reacher, and the update frequency is conﬁgured as 1 for the two environments.
On Scavenger, we use 256 -sized replay buffers (chosen out of { 1 , 128 , 256 } ) and mini-batches with a size of 1 after testing { 1 , 4 , 8 , 16 } .
For Reacher, we make replay buffers and mini-batches 2048 -sized and 32 -sized, which are chosen from { 2048 } and { 1 , 8 , 32 } .
For the exploration, we use the (cid:15) -greedy with its value of 0 . 1 .
For the DeepMind Lab environment, we employ the frame stacking strategy and the network architecture for our shared feature extractor for ˜ ψ and ˜ φ from Mnih et al.
[23], where the latent feature dimensionality is 64 .
˜ w has one hidden layer with 16 units.
A learning rate of 1 e − 5 for the Adam optimizer is chosen out of { 1 e − 4 , 1 e − 5 } .
The target update frequency and the update frequency are 500 and 1 , respectively.
We use a replay buffer whose size is 65536 and sample 16 -sized mini-batches, which is chosen from { 16 , 64 } .
Speciﬁcally for USFAs, for the Scavenger and Reacher environments, we sample ﬁve policies from D z ( ·| w ) at each step, where each dimension of a policy vector is sampled from a Gaussian distribution with a standard deviation of 0 . 1 and a mean of the corresponding element of w .
We use the same number of policy vectors and policy vector distribution but conditioned on the task information instead of task vector, for the DeepMind Lab environment.
For the constrained training of USFAs, we use the same set of hyperparameters, as well as a weight coefﬁcient for the constraints of 0 .
1 (chosen from { 0 . 1 , 1 . 0 , 5 . 0 } ), where one task is uniformly randomly sampled from the bounded linear span of source tasks at each step.
Also, for a more stable learning, we load the model checkpoints of USFAs at the 0 .
5 M-th step and train for the remaining 0 .
5 M steps.
We universal employ cvxpylayers [3] as a general LP solver for the upper bound with the default solver conﬁgurations, for both constrained GPI and the constrained training.
For the Scavenger and Reacher experiments, we conduct the experiments with our CPU machines, where majority of the CPUs are Intel Xeon Gold 6130 or Intel Xeon E5-2695.
For each training run, we use two CPU cores without any GPUs for about 6-12 hours.
For the DeepMind Lab experiments, we employ our GPU machines and run experiments for about 48-72 hours.