Original: C ONSTRUCTING A G OOD B EHAVIOR B ASIS FOR T RANSFER USING G ENERALIZED P OLICY U PDATES
Revised: C ONSTRUCTING A G OOD B EHAVIOR B ASIS FOR T RANSFER USING G ENERALIZED P OLICY U PDATES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of learning a good set of policies so that when combined together , they can solve a wide variety of unseen reinforcement learning tasks with no or very little new data
Revised: We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks with no or very little new data.

Original: Speciﬁcally, we consider the framework of generalized policy evaluation and improvement, in which the rewards for all tasks of interest are assumed to be expressible as a linear combination of a ﬁxed set of features.
Revised: Speciﬁcally, we consider the framework of generalized policy evaluation and improvement, in which the rewards for all tasks of interest are assumed to be expressible as a linear combination of a ﬁxed set of features.

Original: We show theoretically that, under certain assumptions, having access to a speciﬁc set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance on all possible downstreamtasks, although these tasks  are typically more complex than the ones on which the agent was trained
Revised: We show theoretically that, under certain assumptions, having access to a speciﬁc set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance on all possible downstream tasks which are typically more complex than the ones on which the agent was trained.

Original: Based on this theoretical analysis, we propose a simple algorithm that iteratively constructs this set of policies.
Revised: Based on this theoretical analysis, we propose a simple algorithm that iteratively constructs this set of policies.

Original: In addition to empirically validating our theoretical results, we compare our approach with recently proposed diverse policy set construction methods and show that, while others fail, our approach is able to build a behavior basis that enables instantaneous transfer to all possible downstream tasks.
Revised: In addition to empirically validating our theoretical results, we compare our approach with recently proposed diverse policy set construction methods and show that, while others fail, our approach is able to build a behavior basis that enables instantaneous transfer to all possible downstream tasks.

Original: We also show empirically that having access to a set of independent policies can better bootstrap the learning process on downstream tasks where the new reward function cannot be described as a linear combination of the features.
Revised: We also show empirically that having access to a set of independent policies can better bootstrap the learning process on downstream tasks where the new reward function cannot be described as a linear combination of the features.

Original: Finally, we demonstrate that this policy set can be useful in a realistic lifelong reinforcement learning setting
Revised: Finally, we demonstrate that this policy set can be useful in a realistic lifelong reinforcement learning setting

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) studies the problem of building rational decision-making agents that maximize long term cumulative reward through trial-and-error interaction with a given environment.
Revised: Reinforcement learning (RL) studies the problem of building rational decision-making agents that maximize long term cumulative reward through trial-and-error interaction with a given environment.

Original: In recent years, RL algorithms combined with powerful function approximators such as deep neural networks have achieved signiﬁcant successes in a wide range of challenging domains, see e.g. Mnih et al.
Revised: In recent years, RL algorithms combined with powerful function approximators such as deep neural networks have achieved signiﬁcant successes in a wide range of challenging domains (see e.g. Mnih et al.

Original: ; Vinyals et al.
Revised: ; Vinyals et al.

Original: ; Silver et al.
Revised: ; Silver et al.

Original: However, these algorithms require substantial amounts of data for performing very narrowly-deﬁned tasks.
Revised: However, these algorithms require substantial amounts of data for performing very narrowly-deﬁned tasks.

Original: In addition to being datahungry, they are also very brittle to changes in the environment, such as changes in the tasks over time.
Revised: In addition to being datahungry, they are also very brittle to changes in the environment, such as changes in the tasks over time.

Revised: The most important reasons behind these two shortcomings is that RL algorithms usually learn to perform a task from scratch, without leveraging any form of prior knowledge, and they are trained to optimize performance on only a single task.

Original: A promising approach to tackle both of these shortcomings is to learn multiple ways of behaving, i.e., multiple policies that optimize different reward functions, and to reuse them as needed.
Revised: A promising approach to tackle both of these shortcomings is to learn multiple ways of behaving, i.e., multiple policies that optimize different reward functions, and to reuse them as needed.

Original: Having access to multiple previously-learned policies can allow an agent to quickly solve reoccurring tasks in a lifelong RL setting.
Revised: Having access to multiple pre-learned policies can allow an agent to quickly solve reoccurring tasks in a lifelong RL setting.

Original: It can also allow for learning to combine the existing policies via a meta-policy to quickly learn new tasks, as in hierarchical RL.
Revised: It can also allow for learning to combine the existing policies via a meta-policy to quickly learn new tasks, as in hierarchical RL.

Original: Recently, Barreto et al.
Revised: Recently, Barreto et al.

Original: (2020) have proposed the “generalized policy updates” framework, which generalizes the classical policy evaluation and policy improvement operations that underlie many of today’s RL algorithms.
Revised: (2020) have proposed the “generalized policy updates” framework, which generalizes the classical policy evaluation and policy improvement operations that underlie many of today’s RL algorithms.

Original: Its goal is to allow reusing policies resulting from previously learned tasks in order to perform well on downstream tasks, while also being data-efﬁcient.
Revised: Its goal is to allow reusing policies resulting from previously learned tasks in order to perform well on downstream tasks, while also being data-efﬁcient.

Original: More precisely, they learn policies for different tasks, and corresponding successor features, which model the expected value of discounted cumulative feature vectors encountered when executing these policies. Using this information, they were able to instantaneously “synthesize”, in a zero-shot manner, new policies to solverelated downstream tasks, via generalized policy improvement.
Revised: More precisely, after learning the successor features of several policies in a policy set, also referred to as a behavior basis, they were able to instantaneously “synthesize”, in a zero-shot manner, new policies to solve downstream tasks, via generalized policy improvement.

Original: However, this work leaves open two important questions: (1) what set of policies should the agent learn so that its instantaneous performance on all possible downstream tasks is guaranteed to be good, and (2) under what conditions does such a set of policies exist.
Revised: However, this work leaves open two important questions: (1) what set of policies should the agent learn so that its instantaneous performance on all possible downstream tasks is guaranteed to be good, and (2) under what conditions does such a set of policies exist.

Original: In this paper, we provide answers to the questions above by proving that under certain assumptions about the environment dynamics and features, learning a diverse set of policies, which we call a set of independent policies , indeed guarantees good instantaneous performance on all possible downstream tasks.
Revised: In this paper, we provide answers to the questions above by proving that under certain assumptions about the environment dynamics and features, learning a diverse set of policies, which we call a set of independent policies , indeed guarantees good instantaneous performance on all possible downstream tasks.

Original: We provide an iterative algorithm for the construction of this set,and we perform several experiments using this algorithm. The results validate our theoretical ﬁndings.
Revised: After providing an iterative algorithm for the construction of this set, we perform several experiments that validate our theoretical ﬁndings.

Original: In addition to these validation experiments, we compare this algorithm with recently proposed diverse policy set construction methods (Eysenbach et al., 2018; Zahavy et al., 2020; 2021) and show that, unlike these methods, our approach is able to construct a behavior basis that enables instantaneous transfer to all possible tasks.
Revised: In addition to the validation experiments, we compare this algorithm with recently proposed diverse policy set construction methods (Eysenbach et al., 2018; Zahavy et al., 2020; 2021) and show that, unlike these methods, our approach is able to construct a behavior basis that enables instantaneous transfer to all possible tasks.

Original: We also show empirically that learning a set of independent policies can better bootstrap the learning process on downstream tasks where the reward function cannot be described by asingle linear combination of the features.
Revised: We also show empirically that learning a set of independent policies can better bootstrap the learning process on downstream tasks where the reward function cannot be described by a linear combination of the features.

Original: Finally, we demonstrate the usefulness of this set in a realistic lifelong RL scenario, in which the agent faces different tasks over its lifetime.
Revised: Finally, we demonstrate the usefulness of this set in a realistic lifelong RL scenario, in which the agent faces different tasks over its lifetime.

Original: We hope that our study will bring the community a step closer to building lifelong RL agents that are able to perform multiple tasks and a to instantaneously/quickly adapt to new ones.
Revised: We hope that our study will bring the community a step closer to building lifelong RL agents that are able to perform multiple tasks and are able to instantaneously/quickly adapt to new ones during its lifetime.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: In RL (Sutton & Barto, 2018), an agent interacts with its environment by choosing actions to get as much as cumulative long-term reward.
Revised: In RL (Sutton & Barto, 2018), an agent interacts with its environment by choosing actions to get as much as cumulative long-term reward.

Original: The interaction between the agent and its environment is usually modeled as a Markov Decision Process (MDP).
Revised: The interaction between the agent and its environment is usually modeled as a Markov Decision Process (MDP).

Original: An MDP is a tuple M ≡ ( S , A , P, r, d 0 , γ ) , where S is the (ﬁnite) set of states, A is the (ﬁnite) set of actions,
Revised: An MDP is a tuple M ≡ ( S , A , P, r, d 0 , γ ) , where S is the (ﬁnite) set of states, A is the (ﬁnite) set of actions,

Original: P : S ×
Revised: P : S ×

Original: A × S → [0 , 1] is the transition distribution, r : S ×
Revised: A × S → [0 , 1] is the transition distribution, r : S ×

Original: A × S → R is the reward function, which speciﬁes the task of interest, d 0 : S → [0 , 1] is the initial state distribution and γ ∈ [0 , 1) is the discount factor.
Revised: A × S → R is the reward function, which speciﬁes the task of interest, d 0 : S → [0 , 1] is the initial state distribution and γ ∈ [0 , 1) is the discount factor.

Original: In RL, typically the agent does not have any knowledge about P and r beforehand, and its goal is to ﬁnd, through pure interaction, a policy π : S
Revised: In RL, typically the agent does not have any knowledge about P and r beforehand, and its goal is to ﬁnd, through pure interaction, a policy π : S

Original: →
Revised: →

Original: A that maximizes the expected sum of discounted rewards E π,P [ (cid:80) ∞ t =0 γ t r ( S t , A t , S t +1 ) | S 0 ∼ d 0 ] , where E π,P [ · ] denotes the expectation over trajectories induced by π and P .
Revised: A that maximizes the expected sum of discounted rewards E π,P [ (cid:80) ∞ t =0 γ t r ( S t , A t , S t +1 ) | S 0 ∼ d 0 ] , where E π,P [ · ] denotes the expectation over trajectories induced by π and P .

Original: Successor Features.
Revised: Successor Features.

Original: The successor representation for a state s under a policy π allows s to be represented by the (discounted) distribution of states encountered when following π from s (Dayan, 1993).
Revised: The successor representation for a state s under a policy π allows s to be represented by the (discounted) distribution of states encountered when following π from s (Dayan, 1993).

Original: Given a policy, successor features (SF, Barreto et al., 2017) are a generalization of the idea of successor representations from the tabular setting to function approximation.
Revised: Given a policy, successor features (SF, Barreto et al., 2017) are a generalization of the idea of successor representations from the tabular setting to function approximation.

Original: Following Barreto et al.
Revised: Following Barreto et al.

Original: , we deﬁne SFs of a policy π for state-action ( s, a ) as:
Revised: , we deﬁne SFs of a policy π for state-action ( s, a ) as:

Original: [Equation]
Revised: [Equation]

Original: where the i th component of ψ π gives the expected discounted sum of the i th component of the feature vector, φ i , when following policy π , starting from the state-action pair ( s, a ) .
Revised: where the i th component of ψ π gives the expected discounted sum of the i th component of the feature vector, φ i , when following policy π , starting from the state-action pair ( s, a ) .

Original: Successor features allow a decoupling between the reward function and the environment dynamics.
Revised: Successor features allow a decoupling between the reward function and the environment dynamics.

Original: More concretely, if the reward function for a task can be represented as a linear combination of a feature vector φ ( s, a, s (cid:48) )
Revised: More concretely, if the reward function for a task can be represented as a linear combination of a feature vector φ ( s, a, s (cid:48) )

Original: ∈ R n :
Revised: ∈ R n :

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: ) where w ∈ R n , then the state-action value function Q πr w can be computed immediately as the dotproduct of ψ π and w .
Revised: (cid:62) [Equation] (2) where w ∈ R n , then, as we will detail below, the state-action value function Q πr w can be computed immediately as the dot-product of ψ π and w .

Original: The elements of w , w i , can be viewed as indicating a “preference” towards each of the features.
Revised: The elements of w , w i , can be viewed as indicating a “preference” towards each of the features.

Original: Thus, we refer to w interchangeably as either the preference vector or the task.
Revised: Thus, we refer to w interchangeably as either the preference vector or the task.

Original: Intuitively, the elements φ i of the feature vector φ can be viewed as salient events that maybe desirable or undesirable to the agent, such as picking up or leaving objects of certain type, and reaching and/or avoiding certain states.
Revised: Intuitively, the elements φ i of the feature vector φ can be viewed as salient events that maybe desirable or undesirable to the agent, such as picking up or leaving objects of certain type, and reaching and/or avoiding certain states.

Original: Generalized Policy Evaluation and Improvement.
Revised: Generalized Policy Evaluation and Improvement.

Original: Generalized Policy Evaluation (GPE) and
Revised: Generalized Policy Evaluation (GPE) and

Original: Generalized Policy Improvement (GPI), together referred to as Generalized Policy Updates, are generalizations of the well-known policy evaluation and policy improvement operations in standard dynamic programming to a set of tasks and a set of policies (Barreto et al., 2020).
Revised: Generalized Policy Improvement (GPI), together referred to as Generalized Policy Updates, are generalizations of the well-known policy evaluation and policy improvement operations in standard dynamic programming to a set of tasks and a set of policies (Barreto et al., 2020).

Original: They are used as a transfer mechanism in RL to quickly construct a solution for a newly given task.
Revised: They are used as a transfer mechanism in RL to quickly construct a solution for a newly given task.

Original: One particularly efﬁcient instantiation of GPE & GPI is through the use of SFs and value-based action selection.
Revised: One particularly efﬁcient instantiation of GPE & GPI is through the use of SFs and value-based action selection.

Original: More concretely, given a set of MDPs having the following form:
Revised: More concretely, given a set of MDPs having the following form:

Original: [Equation]
Revised: [Equation]

Original: and given SFs ψ π ( s, a ) of a policy π , an efﬁcient form of GPE on the task r w for policy π can be performed as follows:
Revised: and given SFs ψ π ( s, a ) of a policy π , an efﬁcient form of GPE on the task r w for policy π can be performed as follows:

Original: [Equation]
Revised: [Equation]

Original: where Q πr w ( s, a ) is the state-action value function of π on the task r w .
Revised: where Q πr w ( s, a ) is the state-action value function of π on the task r w .

Original: And, after performing GPE for all the policies π in a ﬁnite policy set Π , following Barreto et al.
Revised: And, after performing GPE for all the policies π in a ﬁnite policy set Π , following Barreto et al.

Original: (2017), an efﬁcient form of GPI can be performed as follows:
Revised: (2017), an efﬁcient form of GPI can be performed as follows:

Original: [Equation]
Revised: [Equation]

Original: We will refer to this speciﬁc use of SFs and value-based action selection for performing GPE & GPI as simply GPE & GPI throughout the rest of this study.
Revised: We will refer to this speciﬁc use of SFs and value-based action selection for performing GPE & GPI as simply GPE & GPI throughout the rest of this study.

Original: Note that π GPI will in general outperform all the policies in Π , and that the actions selected by π GPI on a state may not coincide with any of the actions selected by π ∈ Π on that state.
Revised: Note that π GPI will in general outperform all the policies in Π , and that the actions selected by π GPI on a state may not coincide with any of the actions selected by π ∈ Π on that state.

Original: Hence, the policy space that can be attained by GPI can in principle be a lot larger than, e.g., the space accessible by callingsequentially policies  from the original set.
Revised: Hence, the policy space that can be attained by GPI can in principle be a lot larger than, e.g., the space accessible by calling policies sequentially from the original set.

Original:  P ROBLEM F ORMULATION AND T HEORETICAL A NALYSIS
Revised:  P ROBLEM F ORMULATION AND T HEORETICAL A NALYSIS

Original: GPE & GPI provide a guarantee that, for any reward function linear in the features, π GPI is at least as good as any of the policies π from the “base set” which was used to construct it.
Revised: GPE & GPI provide a guarantee that, for any reward function linear in the features, π GPI is at least as good as any of the policies π from the “base set” which was used to construct it.

Original: While this is an appealing guarantee of monotonic improvement, it does not say much, for two reasons.
Revised: While this is an appealing guarantee of monotonic improvement, it does not say much, for two reasons.

Original: First, it is not clear how big an improvement can be expected for different tasks.
Revised: First, it is not clear how big an improvement can be expected for different tasks.

Original: More importantly, it leave open the question of how one should choose base policies in order to ensure as much improvement as possible.
Revised: More importantly, it leaves open the question of how one should choose base policies in order to ensure as much improvement as possible.

Original: After all, if we had a weak set of policies and we simply matched their value with π GPI , this may not be very useful.
Revised: After all, if we had a weak set of policies and we simply matched their value with π GPI , this may not be very useful.

Original: We will now show that, under certain assumptions, having access to a speciﬁc set of diverse policies, which we call a set of independent policies, can allow  instantaneously achieving high-level performance on all possible downstream tasks.
Revised: We will now show that, under certain assumptions, having access to a speciﬁc set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance on all possible downstream tasks.

Original: Let us start by assuming that we are interested in a set of MDPs M φ , as deﬁned in ( 3 ), with deterministic transition functions (the reason for the determinism assumption will become clear by the end  this section).
Revised: Let us start by assuming that we are interested in a set of MDPs M φ , as deﬁned in ( 3 ), with deterministic transition functions (the reason for the determinism assumption will become clear by the end of this section).

Original: For convenience, we also restrict the possible w values from R n to W , where W is the surface of the (cid:96) 2 n -dimensional ball.
Revised: For convenience, we also restrict the possible w values from R n to W , where W is the surface of the (cid:96) 2 n -dimensional ball.

Original: Note that this choice does not alter the optimal policies of the MDPs in M φ , as an optimal policy is invariant with respect to the scale of the reward and
Revised: Note that this choice does not alter the optimal policies of the MDPs in M φ , as an optimal policy is invariant with respect to the scale of the reward and

Original: W contains all possible directions.
Revised: W contains all possible directions.

Original: Next, we assume that the features φ i that make up the feature vectors φ form a set of independent features  , deﬁned as follows:
Revised: Next, we assume that the features φ i that make up the feature vectors φ form a set of independent features (SIF) , deﬁned as follows:

Original: Deﬁnition 1 (Set of Independent Features) .
Revised: Deﬁnition 1 (SIF) .

Original: A set  features Φ = { φ i | φ i : S ×
Revised: A set of features Φ = { φ i | φ i : S ×

Original: A × S → { 0 , C } , C ∈ R + } ni =1 is called independent if, for any feature φ i ∈ Φ and any initial state s 0 ∼ d 0 , we have: (1) φ i ( s 0 , a 0 , s 1 ) =0 , where a 0 =
Revised: A × S → { 0 , C } , C ∈ R + } ni =1 is called independent if, for any feature φ i ∈ Φ and any initial state s 0 ∼ d 0 , we have: (1) φ i ( s 0 , a 0 , s 1 ) =

Original: π
Revised: ∀

Original:  ( s 0  )  and (2) there exists at least one trajectory, starting from s 0 , in which all the states associated with φ i ( s t , a t , s t +1 ) =
Revised: a 0 ∈ A and ∀ s 1 ∼ P ( s 0 , a 0 , · ) , and (2) there exists at least one trajectory, starting from s 0 , in which all the states associated with φ i ( s t , a t , s t +1 ) =

Original: C are visited, while the states associated with φ j ( s t , a t , s t +1 )
Revised: C are visited, while the states associated with φ j ( s t , a t , s t +1 )

Original: =
Revised: =

Original: C , for all j (cid:54) =
Revised: C , ∀ j (cid:54) =

Original: i , are not visited.
Revised: i , are not visited.

Original: It should be noted that a speciﬁc instantiation of this deﬁnition is the case where each feature is set to a positive constant at certain independently reachable state/states and zero elsewhere, which is the most common instantiation of feature vectors used in previous related work (Barreto et al., 2017; 2020).
Revised: It should be noted that a speciﬁc instantiation of this deﬁnition is the case where each feature is set to a positive constant at certain independently reachable state/states and zero elsewhere, which is the most common instantiation of feature vectors used in previous related work (Barreto et al., 2017; 2020).

Original: We deﬁne the performance of an arbitrary policy π on a task r w as:
Revised: We deﬁne the performance of an arbitrary policy π on a task r w as:

Original: [Equation]
Revised: [Equation]

Original: (6) where Q πr w ,γ =1 is the undiscounted state-action value function of policy π on task r w .
Revised: where E π,P [ · ] denotes the expectation over trajectories induced by π and P .

Original: Note that J πr w is a scalar corresponding to the expected undiscounted return of policy π under the initial state distribution d 0 , which is the expected total reward obtained by π when starting from s 0 ∼ d 0 .
Revised: Note that J π r w is a scalar corresponding to the expected undiscounted return of policy π under the initial state distribution d 0 , which is the expected total reward obtained by π when starting from s 0 ∼ d 0 .

Original: We are now ready to formalize the problem wee want to tackle.
Revised: We are now ready to formalize the problem we want to tackle:

Original: Problem Formulation.
Revised: Problem Formulation.

Original: Given a set of MDPs M φ with deterministic transition functions and a set of independent features, we want to construct a set of n policies Π n = { π i } ni =1 , such that the performance of the policy π GPI Π n obtained by performing GPI starting from that set will maximize (6) for all rewards r w , where w ∈ W .
Revised: Given a set of MDPs M φ with deterministic transition functions and a SIF, we want to construct a set of n policies Π n = { π i } n i =1 , such that the performance of the policy π GPI Π n obtained by performing GPI on that set will maximize (6) for all rewards r w , where w ∈ W .

Original: That is, we want to solve the following problem:
Revised: That is, we want to solve the following problem:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It should be noted that the performance measure provided in (6) only measures the expected total reward and thus cannot capture the optimality of the GPI policy.
Revised: It should be noted that the performance measure provided in (6) only measures the expected total reward and thus cannot capture the optimality of the GPI policy.

Original: For instance, this measure cannot distinguish between two policies that achieve the same expected total reward in a different number of time steps.
Revised: For instance, this measure cannot distinguish between two policies that achieve the same expected total reward in a different number of time steps.

Original: However, Theorem 2 in Barreto et al.
Revised: However, Theorem 2 in Barreto et al.

Original: (2017) implies that, in general, the only way to guarantee the optimality of the GPI policy is to construct a behavior basis that contains all possible policies induced by all w ∈ W .
Revised: (2017) implies that, in general, the only way to guarantee the optimality of the GPI policy is to construct a behavior basis that contains all possible policies induced by all w ∈ W .

Original: Since there are inﬁnitely many w values, this is impractical.
Revised: Since there are inﬁnitely many w values, this is impractical.

Original: Thus, in this study, we only consider GPI policies that maximize the expected total reward (6).
Revised: Thus, in this study, we only consider GPI policies that maximize the expected total reward (6).

Original: As a solution candidate to the problem in (7), we now focus on a speciﬁc set of deterministic policies, called set of independent policies  , that are able to obtain features independently of each other:
Revised: As a solution candidate to the problem in (7), we now focus on a speciﬁc set of deterministic policies, called set of independent policies (SIP) , that are able to obtain features independently of each other:

Original: Deﬁnition 2 (Set of Independent Policies) .
Revised: Deﬁnition 2 (SIP) .

Original: Let Φ = { φ i } ni =1 be a set of independent features and let Π = { π i } ni =1 be a set of deterministic policies that are induced by each of the features in Φ .
Revised: Let Φ = { φ i } ni =1 be a SIF and let Π = { π i } ni =1 be a set of deterministic policies that are induced by each of the features in Φ .

Original: Π is deﬁned to be a set of independent policies if its elements, π i , satisfy:
Revised: Π is deﬁned to be a SIP if its elements, π i , satisfy:

Original: [Equation]
Revised: [Equation]

Original: where T is the horizon in episodic environments and T → ∞ in non-episodic ones, a 

Revised: where T is the horizon in episodic environments and T → ∞ in non-episodic ones, a

Original: = π
Revised: = π

Original: i ( s 0 ) and ( s t , a t , s t +1 )
Revised: i ( s 0 ) and ( s t , a t , s t +1 )

Original: Tt =1 is the sequence of state-action-state triples generated by π i ’s interaction with the environment.
Revised: Tt =1 is the sequence of state-action-state triples generated by π i ’s interaction with the environment.

Original: In general, having a set of independent policies in a set of MDPs with stochastic transition functions is not possible, as the stochasticity can prevent (8) from holding.
Revised: In general, having a SIP in a set of MDPs with stochastic transition functions is not possible, as the stochasticity can prevent (8) from holding.

Original: Thus, the assumption of a set of MDPs with deterministic transition functions is critical to our analysis.
Revised: Thus, the assumption of a set of MDPs with deterministic transition functions is critical to our analysis.

Original: An immediate consequence of having this set of policies is that the corresponding SFs can be expressed in a simpler form, as follows:
Revised: An immediate consequence of having this set of policies is that the corresponding SFs can be expressed in a simpler form, as follows:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let φ j be a member of a set of independent features Φ and let π i be a policy that is induced by  feature φ i  and is a member of a set of independent policies Π .
Revised: Let Φ be a SIF and let π i be a policy that is induced by the feature φ i ∈ Φ and is a member of a SIP Π .

Original: Then, the entries of the SF ψ π i of policy π i has the following form:
Revised: Then, the entries of the SF ψ π i of policy π i has the following form:

Original: [Equation]
Revised: [Equation]

Original: where φ j ( s, a ) =

Original: E P [ φ j ( S, A, S (cid:48) )

Original: | S = s, A = a ] .

Original: Due to the space constraints, we provide all the proofs in the supplementary material.
Revised: Due to the space constraints, we provide all the proofs in the supp. material.

Original: Lemmaimplies that once we have a set of independent policies, the SFs take the much simpler form (1), which can allow for the GPI policy to maximize performance on all possible tasks according to the measure in (6), solving the optimization objective  (7):
Revised: Lemma 1 implies that once we have a SIP, the SFs take the much simpler form (9), which can allow for the GPI policy to maximize performance on all possible tasks according to the measure in (6), solving the optimization objective in (7):

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Φ be a set of independent features and let Π be a set of independent policies induced by each of the features in Φ .
Revised: Let Φ be a SIF and let Π be a SIP induced by each of the features in Φ .

Original: Then, the GPI policy π GPI Π is a solution to the optimization problem deﬁned in (7).
Revised: Then, the GPI policy π GPI Π is a solution to the optimization problem deﬁned in (7).

Original: Theorem 1 implies that having access to a set of independent policies which consists of only n policies, where n is the dimensionality of φ , and applying the policy composition operator GPI on top, allows for instantaneously achieving maximum performance across all possible downstream tasks.
Revised: Theorem 1 implies that having access to a SIP which consists of only n policies, where n is the dimensionality of φ , and applying the policy composition operator GPI on top, allows for instantaneously achieving maximum performance across all possible downstream tasks.

Original: Considering the fact that there are inﬁnitely many such tasks, this provides a signiﬁcant gain in the number of tasks that need to be solved.
Revised: Considering the fact that there are inﬁnitely many such tasks, this provides a signiﬁcant gain in the number of policies that are required to be learned for full downstream task coverage.

Original:  C ONSTRUCTING A S ET OF I NDEPENDENT P OLICIES
Revised:  C ONSTRUCTING A S ET OF I NDEPENDENT P OLICIES

Original: Based on our theoretical analysis in Section 3, we now propose a simple algorithm that iteratively constructs a set of independent policies, given a set of independent features.
Revised: Based on our theoretical analysis in Section 3, we now propose a simple algorithm that iteratively constructs a SIP, given a SIF.

Original: Since independent policies can attain certain features without affecting the others, a set  can be constructed by simply learning policies for each of the appropriate tasks and then by adding these learned policies one-byone to an initially empty set.
Revised: Since independent policies can attain certain features without affecting the others, a set of them can be constructed by simply learning policies for each of the appropriate tasks and then by adding these learned policies one-by-one to an initially empty set.

Original: In particular, by sequentially learning policies for each of the tasks in W = { w i } ni =1 ⊂ W , where w i is an n -dimensional vector with a positive number in the i th coordinate and negative numbers elsewhere, and the adding these policies to a policy set, one can construct a set of independent policies.
Revised: In particular, by sequentially learning policies for each of the tasks in W = { w i } n i =1 ⊂ W , where w i is an n -dimensional vector with a positive number in the i th coordinate and negative numbers elsewhere, and the adding these policies to a policy set, one can construct a SIP.

Original: Algorithm 1 provides a step-by-step description of this[Figure] construction process.
Revised: Algorithm 1 provides a stepby-step description of this construction process.

Original: Note that the algorithm does not depend on the particular values of w i ∈ W as long as the corresponding coordinate has a positive value and all others are negative.
Revised: Note that the algorithm does not depend on the particular values of w i ∈ W as long as the corresponding coordinate has a positive value and all others are negative.

Original: Another thing to note is that the algorithm runs for only n iterations, where n is the dimensionality of the feature vector φ . 
Revised: Another thing to note is that the algorithm runs for only n iterations, where n is the dimensionality of the feature vector φ . [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original:  Initialize: Π 0 ← {} , t ← 1 , W ← { w i } ni =1 ; while t ≤ n do w t ← W [ t ] ; π t ← solution of the task r w t using RL ; Π t ← Π t − 1 + { π t } ; t ← t + 1 ; endreturn: Π n
Revised: Algorithm 1: Constructing a SIP Require: a SIF Φ ; Initialize: Π 0 ← {} , t ← 1 , W ← { w i } ni =1 ; while t ≤ n do w t ← W [ t ] ; π t ← solution of the task r w t using RL ; Π t ← Π t − 1 + { π t } ; t ← t + 1 ; endreturn: Π n

Original: We start this section by performing experiments to illustrate the theoretical results derived in Section 3.
Revised: We start this section by performing experiments to illustrate the theoretical results derived in Section 3.

Original: Then, we compare the performance of the policy construction approach presented in Algorithm 1 with recently proposed diverse policy set construction methods (Eysenbach et al., 2018; Zahavy et al., 2020; 2021).
Revised: Then, we compare the performance of the policy construction approach presented in Algorithm 1 with recently proposed diverse policy set construction methods (Eysenbach et al., 2018; Zahavy et al., 2020; 2021).

Original: Next, we show that a set of independent policies can better bootstrap learning on new tasks where the reward function cannot in fact be expressed as a linear combination of the features, a case that lies outside of our assumptions.
Revised: Next, we show that a SIP can better bootstrap learning on new tasks where the reward function cannot in fact be expressed as a linear combination of the features, a case that lies outside of our assumptions.

Original: Finally, we demonstrate that the set of policies produced by our approach can be useful in a realistic lifelong RL setting.
Revised: Finally, we demonstrate that the set of policies produced by our approach can be useful in a realistic lifelong RL setting.

Original: The experimental details together with more detailed results can be found in the supplementary material.
Revised: The experimental details together with more detailed results can be found in the supp. material.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: Throughout this section, we perform experiments on the 2D item collection environment proposed in Barreto et al.
Revised: Throughout this section, we perform experiments on the 2D item collection environment proposed in Barreto et al.

Original: (2020) (see Fig. 1a), as it is a prototypical environment where
Revised: (2020) (see Fig. 1a), as it is a prototypical environment where

Original: GPE & GPI is useful and it allows for easy visualization of performance across all downstream tasks.
Revised: GPE & GPI is useful and it allows for easy visualization of performance across all downstream tasks.

Original: Here, the agent, depicted by the yellow circle, starts randomly in one of the cells and has to obtain/avoid 5 randomly placed red and green items, which are of different type.
Revised: Here, the agent, depicted by the yellow circle, starts randomly in one of the cells and has to obtain/avoid 5 randomly placed red and green items, which are of different type.

Original: At each time step, the agent receives an image that contains its own position and the position and type of each item.
Revised: At each time step, the agent receives an image that contains its own position and the position and type of each item.

Original: Based on this, the agent selects an action that deterministically moves it to one of its four neighboring cells.
Revised: Based on this, the agent selects an action that deterministically moves it to one of its four neighboring cells (except if the agent is adjacent to the boundaries of the grid, it remains on the same cell).

Original: By moving to the cells occupied by an item, the agent picks up that item and gets a reward deﬁned by that item type.
Revised: By moving to the cells occupied by an item, the agent picks up that item and gets a reward deﬁned by that item type.

Original: The goal of the agent is to pick up the “good” items and avoid the “bad” ones, depending on the preference vector w .
Revised: The goal of the agent is to pick up the “good” items and avoid the “bad” ones, depending on the preference vector w .

Original: The agent-environment interaction lasts for 50 time steps, after which the agent receives a “done” signal, marking the end of the episode.
Revised: The agent-environment interaction lasts for 50 time steps, after which the agent receives a “done” signal, marking the end of the episode.

Original: Note that despite its small size, the cardinality of this environment’s state space is on the order of 10and thus requires the use of function approximation.
Revised: Note that despite its small size, the cardinality of this environment’s state space is on the order of 10and thus requires the use of function approximation.

Original: Moredetails on the  environment  can be found in Barreto et al.
Revised: More on the implementation details of the environment and results on its stochastic version can be found in the supp. material.

Original: In order to use GPE & GPI in this environment, we must ﬁrst deﬁne: (1) the features φ and (2) a set of policies Π .
Revised: In order to use GPE & GPI in this environment, we must ﬁrst deﬁne: (1) the features φ and (2) a set of policies Π .

Original: Following Barreto et al.
Revised: Following Barreto et al.

Original: (2020), we deﬁne each feature φ i as an indicator function signalling whether an item of type i has been picked up by the agent.
Revised: (2020), we deﬁne each feature φ i as an indicator function signalling whether an item of type i has been picked up by the agent.

Original: That is, φ i ( s, a, s (cid:48) )
Revised: That is, φ i ( s, a, s (cid:48) )

Original: = 1 if taking action a in state s results in picking up an item of type i , and φ i ( s, a, s (cid:48) )
Revised: = 1 if taking action a in state s results in picking up an item of type i , and φ i ( s, a, s (cid:48) )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: Note wwwww0. 1.0 Normalized sum of rewards 3 (a) Disjoint sets wwwww0.
Revised: Note that these form a SIF, as there exist trajectories which start as φ ( s 0 , a 0 , s 1 )

Original: [-1.0 Normalized sum of rewards 
 [Figure] that these form a set of independent features, as there exist trajectories which start as φ ( s 0 , a 0 , s 1 )-] = 0 and in which all the features associated with a certain item type can be obtained without affecting the others
Revised: [-1.0 Normalized sum of rewards 
 [Figure] that these form a set of independent features, as there exist trajectories which start as φ ( s 0 , a 0 , s 1 )-] = 0 and in which all the features associated with a certain item type can be obtained without affecting the others.

Original: We now turn to the question on how to determine the set of policies Π .
Revised: We now turn to the question on how to determine the set of policies Π .

Original: We restrict the policies in Π to be solutions of the tasks w ∈ W .
Revised: We restrict the policies in Π to be solutions of the tasks w ∈ W .

Original: With Algorithm 1, we have already provided one way of constructing this set.
Revised: With Algorithm 1, we have already provided one way of constructing wwwww0. 1.0 Normalized sum of rewards 3 (a) Disjoint sets wwwww0. 1.0 Normalized sum of rewards 151523( [Figure]

Revised: growing sets this set.

Original: Throughout the rest of this section we will compare Algorithm 1 with alternative ways of obtaining policies.
Revised: Throughout the rest of this section we will compare Algorithm 1 with alternative policy set construction methods.

Original: Following Barreto et al.
Revised: Following Barreto et al.

Original: (2020), we use an algorithm analogous to Q-learning to approximate the
Revised: (2020), we use an algorithm analogous to Q-learning to approximate the

Original: SFs induced by φ and Π .
Revised: SFs induced by φ and Π .

Original: Similarly, we represent the SFs using multilayer perceptrons with two hidden layers.
Revised: Similarly, we represent the SFs using multilayer perceptrons with two hidden layers.

Original: More details can be found in Barreto et al.
Revised: More details can be found in Barreto et al.

Original: 5. I LLUSTRATIVE E XPERIMENTS
Revised: 5. I LLUSTRATIVE E XPERIMENTS

Original: Question 1.
Revised: Question 1.

Original: Does the theoretical results, derived in Section 3, also hold empirically?
Revised: Do the theoretical results, derived in Section 3, also hold empirically?

Original: In order to answer this question, we constructed a policy set using Algorithm 1 and evaluated it using the evaluation scheme proposed by Barreto et al.
Revised: In order to answer this question, we constructed a policy set using Algorithm 1 and evaluated it using the evaluation scheme proposed by Barreto et al.

Original: Speciﬁcally, after initializing W = { w 1 , w 5 } and running Algorithm 1, we obtained the independent policy set Π 15 = { π 1 , π 5 } , whose elements are solutions to the tasks w 1 and w 5 , respectively (see Fig. 1b).
Revised: Speciﬁcally, after initializing W = { w 1 , w 5 } and running Algorithm 1, we obtained the SIP Π 15 = { π 1 , π 5 } , whose elements are solutions to the tasks w 1 and w 5 , respectively (see Fig. 1b).

Original: Then, we ran an evaluation scheme that evaluates the GPI policy π GPI Π 15 , over 17 evenly spread tasks over the nonnegative quadrants of the unit circle in Fig.
Revised: Then, we ran an evaluation scheme that evaluates the GPI policy π GPI Π 15 , over 17 evenly spread tasks over the nonnegative quadrants of the unit circle in Fig.

Original: 1b.
Revised: 1b.

Original: We did not perform evaluation in the negative quadrant, as the tasks there are not interesting.
Revised: We did not perform evaluations in the negative quadrant, as the tasks there are not interesting.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 2a.
Revised: 2a.

Original: As can be seen, π GPI Π 15 performs well across all the downstream tasks, as expected based on the result in Theorem 1.
Revised: As can be seen, π GPI Π 15 performs well across all the downstream tasks, empirically verifying Theorem 1.

Original: Question 2.
Revised: Question 2.

Original: Is having a set of independent policies essential for full downstream task coverage? 
Revised: Is having a SIP essential for full downstream task coverage? I t e m s R G

Revised: I t e m s

Revised: R

Original:  According to Theorem 1, having a set of independent policies is a requirement for guaranteeing good performance across all downstream tasks.
Revised: G P o li c i e s V a l u e According to Theorem 1, having a SIP is a requirement for guaranteeing good performance across all downstream tasks.

Original: However, in order to see how well the GPI policy performs when this is not the case, similar to Barreto et al.
Revised: However, in order to see how well the GPI policy performs when this is not the case, similar to Barreto et al.

Original: , we compared Π 15 with two other possible policy sets: Π 24 = { π 2 , π 4 } and Π 3 = { π 3 } , whose policies correspond to the tasks w 2 , w 4 and w 3 , respectively.
Revised: , we compared Π 15 with two other possible policy sets: Π 24 = { π 2 , π 4 } and Π 3 = { π 3 } , whose policies correspond to the tasks w 2 , w 4 and w 3 , respectively.

Original: It should be noted that Π 24 is a default policy set used in prior studies on SFs (Barreto et al., 2017; 2018; 2020; Borsa et al., 2018).
Revised: It should be noted that Π 24 is a default policy set used in prior studies on SFs (Barreto et al., 2017; 2018; 2020; Borsa et al., 2018).

Original: The results, in Fig.
Revised: The results, in Fig.

Original: 2a, show that π GPI Πand π GPI Π 3 are not able to perform well across all downstream tasks, speciﬁcally failing on quadrants 
Revised: 2a, show that π GPI Π 24 and π GPI Π 3 are not able to perform well across all downstream tasks, speciﬁcally failing on quadrants II and

Original: II and IV.
Revised: IV.

Original: This justiﬁes the requirement of learning a set of independent policies for full downstream task coverage.
Revised: This justiﬁes the requirement of learning a SIP for full downstream task coverage.

Original: Question 3.
Revised: Question 3.

Original: Do the SFs of the policies in a set of independent policies have a simple form?
Revised: Do the SFs of the policies in a SIP have a simple form?

Original: According to Lemma 1, the SFs of the policies in a set of independent policies should have a simple form.
Revised: According to Lemma 1, the SFs of the policies in a SIP should have a simple form.

Original: To empirically verify this, in Fig.
Revised: To empirically verify this, in Fig.

Original: 3, we visualize the SFs of the policies in the sets Π 15 and Π 24 for the simpliﬁed environment depicted in Fig. 1c.
Revised: 3, we visualize the SFs of the policies in the sets Π 15 and Π 24 for the simpliﬁed environment depicted in Fig. 1c.

Original: We can see that the SFs of the policies in Π 15 indeed have a simple form, in which the off-diagonal entries have values that are very close to zero; this is not the case for Π 24 .I t e m s R
Revised: We can see that the SFs of the policies in Π 15 indeed have a simple form, in which the off-diagonal entries have values that are very close to zero, while this is not the case for Π 24 .

Original: G

Original: I t e m s

Original: R

Original: G

Original: P o li c i e s

Original: V a l u e

Original: Question 4.
Revised: Question 4.

Original: Does adding more policies to a set of independent policies have any effect on its downstream task coverage?
Revised: Does adding more policies to a SIP have any effect on its downstream task coverage?

Original: Theorem 1 implies that having a set of independent policies is enough to guarantee good performance across all possible tasks.
Revised: Theorem 1 implies that, given a SIF, having a SIP is enough to guarantee good performance across all possible tasks.

Original: To test empirically whether adding more policies to the set has any effect on the performance across all tasks, we compare the performance of the policy set Π 15 with policy sets Π 152 , Π 1523 and Π 15234 , formed by adding one-by-one the policies π 2 , π 3 and πto Π 15 .
Revised: To test empirically whether adding more policies to a SIP has any effect on its performance across all tasks, we compare the performance of the policy set Π 15 with policy sets Π 152 , Π 1523 and Π 15234 , formed by adding one-by-one the policies π 2 , π 3 and π 4 to Π 15 .

Original: Results are shown in Fig. 2b.
Revised: Results are shown in Fig. 2b.

Original: As expected, adding more policies to the set of independent policies Π 15 has no effect on  downstream task performance.
Revised: As expected, adding more policies to the SIP Π 15 has no effect on its downstream task coverage.

Original: 5. C OMPARATIVE AND L IFELONG L EARNING E XPERIMENTS
Revised: 5. C OMPARATIVE AND L IFELONG L EARNING E XPERIMENTS

Original: Question 5.
Revised: Question 5.

Original: How does Algorithm 1 compare to prior diverse policy set construction methods?
Revised: How does Algorithm 1 compare to prior diverse policy set construction methods?

Original: [Figure]
Revised: [Figure]

Original: Since the policy sets constructed by the prior methods depend on their particular initialization, their plots are obtained by running each of the constructed policy sets for 5 runs and then averaging over their results.

Original: For each task, the agent was evaluated on 1000 episodes.

Original: The plot for Πis obtained in a similar way as in Fig.

Original: Prior work has shown that having access to a diverse set of policies can help transfer to downstream tasks.

Original: Algorithm 1 can also be seen as a diverse policy set construction method, as it constructs a policy set that is diverse in the feature visitation proﬁle.
Revised: Algorithm 1 can also be seen as a diverse policy set construction method, as it constructs a policy set that is diverse in the feature visitation proﬁle.

Original: In order to see how well it compares to prior work, we compare
Revised: In order to see how well it compares to prior work, we compare

Original: Algorithm 1 with three recently proposed diverse policy set construction methods: (1) DIAYN 1 (Eysenbach et al., 2018) which constructs a policy set by collectively training policies with informationtheoretic reward functions that encourage discriminablity among the policies in the set based on their state visitation proﬁle, (2) SMP (Zahavy et al., 2020) which constructs a policy set by iteratively adding policies trained on the worst-case linear reward with respect to the previous policies in the set, and (3)
Revised: Algorithm 1 with three recently proposed diverse policy set construction methods: (1) DIAYN 1 (Eysenbach et al., 2018) which constructs a policy set by collectively training policies with informationtheoretic reward functions that encourage discriminablity among the policies in the set based on their state visitation proﬁle, (2) SMP (Zahavy et al., 2020) which constructs a policy set by iteratively adding policies trained on the worst-case linear reward with respect to the previous policies in the set, and (3)

Original: DSP (Zahavy et al., 2021) which constructs a policy set by iteratively adding policies that are trained on tasks w n = − (1 /n ) (cid:80) n k =

Revised: DSP (Zahavy et al., 2021) which constructs a policy set by iteratively adding policies that are trained on tasks w n = − (1 /n ) (cid:80) nk =

Original: ψ k , where n is the number of policies in the current policy set.
Revised: ψ k , where n is the number of policies in the current policy set.

Original: More on the speciﬁc implementation details of these methods can be found in the supplementary material.
Revised: More on the speciﬁc implementation details of these methods can be found in the supp. material.

Original: The comparison results, shown in Fig.
Revised: The comparison results, shown in Fig.

Original: 4, indicate that none of the prior diverse policy set construction methods are able to consistently construct a policy set that enables full downstream task coverage.
Revised: 4, indicate that none of the prior diverse policy set construction methods are able to consistently construct a policy set that enables full downstream task coverage.

Original: Question 6.
Revised: Question 6.

Original: Is learning a set of independent policies also effective when the downstream task’s reward function is not a single linear combination of the features?
Revised: Is learning a SIP also effective when the downstream task’s reward function cannot be expressed a single linear combination of the features?

Original: So far, we have considered the scenario where the GPI policy is evaluated on tasks whose reward functions were obtained by linearly combining the features φ using a single, ﬁxed w .
Revised: So far, we have considered the scenario where the GPI policy is evaluated on tasks whose reward functions were obtained by linearly combining the features φ using a single, ﬁxed w .

Original: We now consider downstream tasks that do now satisfy this assumption, e.g. where w changes as a function over the environment’s state.
Revised: We now consider downstream tasks that do not satisfy this assumption, e.g. where w changes as a function over the environment’s state.

Original:  Theorem 1 has nothing to say in this case.
Revised: Note that Theorem 1 has nothing to say in this case.

Original: Nevertheless, in order to test whether if learning a set of independent policies can still be useful in this scenario, we consider a transfer setting in which a meta-policy is trained to orchestrate the policies in this set when faced with a downstream task.
Revised: Nevertheless, in order to test whether if learning a SIP can still be useful in this scenario, we consider a transfer setting in which a meta-policy is trained to orchestrate the policies in this set when faced with a downstream task.

Original: Speciﬁcally, after constructing the policy set, we learn a function that maps states to preference vectors, ω : S → W (cid:48) , whose output is then used by GPE & GPI  synthesize a policy for the downstream task of interest (see the “Preferences as Actions” section in Barreto et al.
Revised: Speciﬁcally, after constructing the policy set, we learn a function that maps states to preference vectors, ω : S → W (cid:48) , whose output is then used by GPE & GPI to synthesize a policy for the downstream task of interest (see the “Preferences as Actions” section in Barreto et al.

Original: (2020) and the Option Keyboard framework (Barreto et al., 2019) for more details).
Revised: (2020) and the Option Keyboard framework (Barreto et al., 2019) for more details).

Original: We deﬁne
Revised: We deﬁne

Original: W (cid:48)
Revised: W (cid:48)

Original: = { w 1 , w 2 , w 3 , w 4 , w 5 } and use Q-learning to learn ω .
Revised: = { w 1 , w 2 , w 3 , w 4 , w 5 } and use Q-learning to learn ω .

Original: [Figure] As an instantiation of the scenario discussed above [Figure] we consider two different downstream tasks, depicted in Fig.
Revised: As an instantiation of the scenario discussed above, we consider two different downstream tasks, depicted in Fig.

Original: 1a: (1) sequential reward collection, in which the agent must ﬁrst pick up items of a certain type and then collect the items of the other type, and (2) balanced reward collection,  in which the agent must pick up the type of item that is more abundant in the environment.
Revised: 1a: (1) sequential reward collection, in which the agent must ﬁrst pick up items of a certain type and then collect the items of the other type, and (2) balanced reward collection, [Figure] in which the agent must pick up the type of item that is more abundant in the environment.

Original: We compared the performance of the set of independent policies Π 15 with the policy set Π 24 . 
Revised: We compared the performance of the SIP Π 15 with the policy set Π 24 . As a reference to the maximum reachable performance

Revised: [Figure]

Original: As a reference to the maximum reachable performance, we also provided the learning curve of DQN (Mnih et al., 2015), whose speciﬁc implementation details can be found in the supplementary material.
Revised: we also provided the learning curve of DQN (Mnih et al., 2015) [Figure] whose speciﬁc implementation details can be found in the supp. material.

Original: The results, shown in Fig.
Revised: The results, shown in Fig.

Original: 5a and 5b, show that Π 15 outperforms Π 24 both in terms of the learning speed and asymptotic performance, reaching the asymptotic performance of DQN.
Revised: 5a and 5b [Figure] show that Π 15 outperforms Π 24 both in terms of the learning speed and asymptotic performance, reaching the asymptotic performance of DQN.

Original: These results suggest that having access to a set of independent policies can also be effective in scenarios where the downstream task’s reward function cannot be described as a single linear combination of the features.
Revised: These results suggest that having access to a SIP can also be effective in scenarios where the downstream task’s reward function cannot be described as a single linear combination of the features [Figure]

Original: Question 7.
Revised: Question 7.

Original: How can learning a set of independent policies be useful in a lifelong RL setting?
Revised: How can learning a SIP be useful in a lifelong RL setting?

Original: [Figure]
Revised: [Figure]

Original: {+d s u m o f r e w a r d s wwwwwww DQN MaxQInit 
+} Except for the last question, we have considered an idealistic scenario in which, during transfer, the agent was provided with a single preference vector w describing a stationary downstream task of interest.
Revised: {+d s u m o f r e w a r d s wwwwwww DQN MaxQInit 
+} Except for the last question, we have considered an idealistic scenario in which, during transfer, the agent was provided with a single preference vector w describing a stationary downstream task of interest.

Original: We now consider a lifelong RL setting, in which the agent has to infer the current w from data and has to quickly adapt to the changing w ’s during its lifetime.
Revised: We now consider a lifelong RL setting, in which the agent has to infer the current w from data and has to quickly adapt to the changing w ’s during its lifetime.

Original: Concretely, we consider the setting in which the agent infers w by following the regression procedure provided in Barreto et al.
Revised: Concretely, we consider the setting in which the agent infers w by following the regression procedure provided in Barreto et al.

Original: and the tasks change from w 1 to w 5 one-by-one in a clockwise fashion every 5 × 10 5 time steps.
Revised: and the tasks change from w 1 to w 5 one-by-one in a clockwise fashion every 5 × 10 5 time steps.

Original: After the end of task w 5 , the task resets back to w 1 and this task cycle goes on forever.
Revised: After the end of task w 5 , the task resets back to w 1 and this task cycle goes on forever.

Original: In Fig.
Revised: In Fig.

Original: 6, we compare the performance of Π 15 with the policy set Π 24  and with DQN.
Revised: 6, we compare the performance of Π 15 with the policy set Π 24 , with DQN, and with a lifelong RL method MaxQInit (Abel et al., 2018) whose implementation details can be found in the supp. material.

Original: We see that the policy set Π 15 allows for better transfer to changing tasks compared to Π 24 .
Revised: We see that the policy set Π 15 allows for better transfer to changing tasks compared to Π 24 .

Original: We also see that DQN fails in transferring useful knowledge between changing task, sometimes catastrophically failing on tasks that it has already encountered before (see the drastic drop in performance when the task transitions from w 5 to w 1 ).
Revised: We also see that, although MaxQInit outperforms DQN, they both fail in instantaneously adapting to the changing tasks, and DQN sometimes catastrophically fails on tasks that it has already encountered before (see the drastic drop in performance when the task transitions from w 5 to w 1 ).

Original: These results illustrate how a set of independent policies can also be useful in a realistic lifelong RL scenario.
Revised: These results illustrate how a SIP can also be useful in a realistic lifelong RL scenario.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Successor Features & Generalized Policy Improvement.
Revised: Successor Features & GPI.

Original: In this study, we focused on relevant work that uses a set of SFs together with GPI for performing transfer in RL. These studies start with the seminal work of Barreto et al. and continue with Barreto et al. Borsa et al.
Revised: In this study, we focused on relevant work that uses a set of SFs together with GPI for performing transfer in RL (Barreto et al., 2017; 2018; 2019; 2020; Borsa et al., 2018;

Original: Hansen et al.
Revised: Hansen et al., 2019).

Original: Among them, the study of Barreto et al.
Revised: Among these studies, the study of Barreto et al.

Original: (2020) is the closest to our work.
Revised: (2020) is the closest to our work.

Original: We built on top of their approach and tried to answer the following questions: (1) what behavior basis should be learned by the agent so that its instantaneous performance on all possible downstream tasks is guaranteed to be good, and (2) what are the conditions required for this?
Revised: We built on top of their approach and tried to answer the following questions: (1) what behavior basis should be learned by the agent so that its instantaneous performance on all possible downstream tasks is guaranteed to be good, and (2) what are the conditions required for this?

Original: In addition, we provided empirical results illustrating how a good behavior basis can be useful in realistic scenarios.
Revised: In addition, we provided empirical results illustrating how a good behavior basis can be useful in realistic scenarios.

Original: Another closely related work is the study of Grimm et al.
Revised: Another closely related work is the study of Grimm et al.

Original: (2019) which shows that having access to a set of policies that obtain certain disentangled features can allow for achieving high level performance on exponentially many downstream goal-reaching tasks.
Revised: (2019) which shows that having access to a set of policies that obtain certain disentangled features can allow for achieving high level performance on exponentially many downstream goal-reaching tasks.

Original: However, this study requires having access to disentangled features, which have a very speciﬁc form; in contrast, we require a set of independent features, which are more general and commonly used (Barreto et al., 2017; 2020).
Revised: However, this study requires having access to disentangled features, which have a very speciﬁc form; in contrast, we require a set of independent features, which are more general and commonly used (Barreto et al., 2017; 2020).

Original: Additionally, we provide a more formal treatment of the instantaneous transfer problem and our experiments address a broader range of questions.
Revised: Additionally, we provide a more formal treatment of the instantaneous transfer problem and our experiments address a broader range of questions.

Original: Diverse Skill Discovery with Information Theory.
Revised: Diverse Skill Discovery.

Original: Another related line of research is the information theoretic diverse skill discovery literature (Gregor et al., 2016;
Revised: Another related line of research is the information theoretic diverse skill discovery literature (Gregor et al., 2016;

Original: Eysenbach et al., 2018; Achiam et al., 2018), in which information theoretic objectives are used to construct a diverse policy set that enables better exploration and transfer when used in downstream tasks.
Revised: Eysenbach et al., 2018; Achiam et al., 2018), in which information theoretic objectives are used to construct a diverse policy set that enables better exploration and transfer on downstream tasks.

Original: One problem with learning thesediverse set of skills is that most of the discovered policies end up being uninteresting, in the sense that they do not come to be useful in solving downstream tasks of interest, ﬁlling up the set with not so interesting policies.
Revised: An important problem with learning these skills is that most of the discovered policies end up being uninteresting, in the sense that they do not come to be useful in solving downstream tasks of interest, ﬁlling up the set with not so interesting policies.

Original: Our work prevents this by constructing a policy set in which every policy is trained to achieve only a particular task so that when combined can solve more complex downstream tasks of interest.
Revised: Our work prevents this by constructing a policy set in which every policy is trained to achieve only a particular task so that when combined, can solve more complex downstream tasks of interest.

Original: A second important problem with these approaches is that they provide no instantaneous policy composition operators as GPI and thus come with no theoretical guarantees on the downstream task coverage. In contrast, we build on top of the GPE & GPI framework which enables for deriving theoretical guarantees. Diverse Skill Discovery with Successor Features. Other related literature includes the use of SFs to build a diverse skill set either to guarantee good performance on the worst-case downstream task or to achieve a single downstream task in multiple different ways (Zahavy et al., 2020; 2021).
Revised: Other related literature includes the use of SFs to build a diverse skill set.

Original: Through using theformal criterion of robustness, Zahavy et al.
Revised: Through using the criterion of robustness, Zahavy et al.

Original: (2020) proposes a method that constructs a set of policies that do as well as possible in the worst-case scenario and shows that this set naturally becomes diverse.
Revised: (2020) proposes a method that constructs a set of policies that do as well as possible in the worst-case scenario, and shows that this set naturally becomes diverse.

Original: However, a problem with this policy construction method is that it is sensitive to the initialization of the preference vector and thus leads to policy sets that do not have full downstream task coverage.
Revised: However, a problem with this policy construction method is that it is sensitive to the initialization of the preference vector and thus leads to policy sets that do not have full downstream task coverage.

Original: By using explicit diversity rewards, Zahavy et al.
Revised: By using explicit diversity rewards, Zahavy et al.

Original: (2021) proposes another method that aims to minimize the correlation between the SFs of the policies in the constructed set.
Revised: (2021) proposes another method that aims to minimize the correlation between the SFs of the policies in the constructed set.

Original: However, since this method depends on the evolution of the policy set, it also fails in building a policy set with full downstream tasks coverage; by contrast, Algorithm 1 does exactly this.
Revised: However, since this method depends on the evolution of the policy set, it also fails in building a policy set with full downstream tasks coverage; by contrast, Algorithm 1 does exactly this.

Original: C ONCLUSION AND FUTURE WORK
Revised: Policy Reuse for Transfer and Lifelong RL. Although there have been prior studies on policy reuse for faster learning on downstream tasks (Pickett & Barto, 2002; Fern ´ andez & Veloso, 2006; Machado et al., 2018; Barreto et al., 2018) and for achieving lifelong RL (Abel et al., 2018), none of them tackle the problem of learning a set of policies for full downstream task coverage. Recently, by building on top of Barreto et al.

Revised: (2017), Nemecek & Parr (2021) proposes a policy set construction method that starts with a set of policies and gradually adds new ones based on a pre-deﬁned threshold.

Revised: However, they also do not consider the full downstream task coverage problem.

Revised: The most closely related work however is the study of Tasse et al.

Revised: (2020) which, by improving on Van Niekerk et al.

Revised: (2019), proposes the Boolean Task Algebra framework for super-exponential downstream task coverage with a pre-learned set of base policies.

Revised: Despite the similarity in the motivation, there are important differences compared to our work: (1) while their approach requires extensions to the reward and value function deﬁnitions, our approach builds on top of the widely adopted GPE & GPI framework, (2) while their framework assumes the undiscounted setting, GPE & GPI assumes the widely used discounted one, and (3) while their approach only considers goal-based tasks, where upon reaching a goal the episode terminates, our framework handles both these tasks and the ones in which the agent has to achieve multiple goals within a single ﬁxed-length episode.

Revised:  C ONCLUSION AND F UTURE W ORK

Original: To summarize, in this study, we provided a theoretical analysis elucidating what counts as a good behavior basis for performing GPE & GPI and showed that, under certain assumptions, having access to a set of independent policies allows for instantaneously achieving high level performance on all  tasks.
Revised: To summarize, in this study, we provided a theoretical analysis elucidating what counts as a good behavior basis for performing GPE & GPI and showed that, under certain assumptions, having access to a set of independent policies allows for instantaneously achieving high level performance on all downstream tasks.

Original: We proposed a simple algorithm that iteratively constructs this policy set.
Revised: Based on this analysis, we proposed a simple algorithm that iteratively constructs this policy set.

Original: Our empirical results illustrate our theoretical results,  show that the proposed algorithm compares favourably to prior methods, and  demonstrate that a set of independent policies can be useful in  lifelong RL.
Revised: Our empirical results (1) validate our theoretical results, (2) show that the proposed algorithm compares favourably to prior methods and (3) demonstrate that a set of independent policies can be useful in scenarios where the downstream task of interest cannot be expressed with a single preference vector, which includes lifelong RL scenarios.

Original: Note that our approach relies on the existence of an independent set of features, which are maximized to obtain independent policies.
Revised: Note that our approach relies on the existence of an independent set of features, which are maximized to obtain independent policies.

Original: In general, an orthogonalization procedure may be needed as a preprocessing step to obtain such features.
Revised: In general, a feature transformation procedure may be needed as a preprocessing step to obtain such features.

Original: We  also hope to ﬁnd useful generalizations of these ideas to the stochastic case.
Revised: We hope to tackle this problem in future work. We also hope to ﬁnd useful generalizations of our theoretical results to MDPs with stochastic transition functions

Original: A P ROOFS
Revised: A P ROOFS

Original: Lemma 1.
Revised: Lemma 1.

Original: Let φ j be a member of a set of independent features Φ and let π i be a policy that is induced by  feature φ i  and is a member of a set of independent policies Π .
Revised: Let Φ be a SIF and let π i be a policy that is induced by the feature φ i ∈ Φ and is a member of a SIP Π .

Original: Then, the entries of the SF ψ π i of policy π i has the following form:
Revised: Then, the entries of the SF ψ π i of policy π i has the following form:

Original: [Equation]
Revised: [Equation]

Original: where φ j ( s, a ) =

Original: E P [ φ j ( S, A, S (cid:48) )

Original: | S = s, A = a ] .

Original: Proof.
Revised: Proof.

Original: The proof follows directly from the deﬁnition of a set of independent policies.
Revised: The proof follows directly from the deﬁnition of a SIP.

Original: Remember that for a policy π i that is a member of a set of independent policies, we have that:[Equation] ∀ t ∈ [Equation]
Revised: Remember that for a policy π i that is a member of a SIP, we have that:

Original: [Equation]

Original: T }

Original: [Equation]
Revised: [Equation]

Original: where T is the horizon, a 0 =
Revised: where T is the horizon, a 0 =

Original: π i ( s 0 ) and ( s t , a t , s t +1 )
Revised: π i ( s 0 ) and ( s t , a t , s t +1 )

Original: Tt =1 is the sequence of state-action-state triples that are generated by π i ’s interaction with the environment.
Revised: Tt =1 is the sequence of state-action-state triples that are generated by π i ’s interaction with the environment.

Original: Thus, for j (cid:54) =
Revised: Thus, for j (cid:54) =

Original: i , we have:
Revised: i , we have:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Φ be a set of independent features and let Π be a set of independent policies induced by each of the features in Φ .
Revised: Let Φ be a SIF and let Π be a SIP induced by each of the features in Φ .

Original: Then, the GPI policy π GPI Π is a solution to the optimization problem deﬁned in (7).
Revised: Then, the GPI policy π GPI Π is a solution to the optimization problem deﬁned in (7).

Original: Proof.
Revised: Proof.

Original: Remember that the GPI policy for a downstream task w is obtained as:
Revised: Remember that the GPI policy for a downstream task w is obtained as:

Original: [Equation]
Revised: [Equation]

Original: where Q max r w ( s, a ) =
Revised: where Q max r w ( s, a ) =

Original: max π i ∈
Revised: max π i ∈

Original: Π Q π i r w ( s, a ) .
Revised: Π Q π i r w ( s, a ) .

Original: By expressing Q π i r w ( s, a ) as a weighted sum of the SFs of policy π
Revised: By expressing Q π i r w ( s, a ) as a weighted sum of the SFs of policy π

Original: i , we have:
Revised: i , we have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition [Equation] [Equation] (by Deﬁnition 1)
Revised: Lemma 1)

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: which implies that the GPI policy π GPI Π will obtain all the features associated with a positive w i (in an order that depends on the product w i ψ π i i ( s, a ) ), ignore the ones with a zero w i , and avoid the ones with a negative w i .
Revised: which implies that the GPI policy π GPI Π will obtain all the features associated with a positive w i (in an order that depends on the product w i ψ π i i ( s, a ) ), ignore the ones with a zero w i , and avoid the ones with a negative w i .

Original: This in turn implies that the GPI policy π GPI Π will solve the  problem in (7).
Revised: This in turn implies that the GPI policy π GPI Π will solve the optimization problem in (7).

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: In this section, we provide the  experimental details of our policy construction method together with the prior methods that we have used for comparison.
Revised: In this section, we provide the implementation details of the 2D item collection environment and the experimental details of our policy construction method together with the prior methods that we have used for comparison.

Revised: B. I MPLEMENTATION D ETAILS OF THE 2D I TEM C OLLECTION E NVIRONMENT

Revised: As described in the supplementary material of Barreto et al.

Revised: (2020), at each step the agent receives an 11 × 11 × ( n +1) tensor (an 11 × 11 image with ( n +1) channels) representing the conﬁguration of the environment, where n is the number of items in the environment.

Original: B. O UR M ETHOD AND DQN
Revised: The channels are used to identify the items and the walls. Speciﬁcally, there is one channel for each of the n items and one channel for the impassable walls around the edges of the grid. The observations are “egocentric” in the sense that images are shifted so that the agent is always at the top-left cell of the grid. Barreto et al.

Revised: (2020) found that this representation helps the learning process as each action taken by the agent results in larger changes in the observations.

Revised: The observations are also toroidal in the sense that the images wrap around the edges so that the agent always observes the environment, even though the walls prevent it from crossing one side to the other.

Original: For our experiments with GPE & GPI and DQN (Mnih et al., 2015), we have used the publicly available code 2 of Barreto et al.
Revised: For our experiments with GPE & GPI and DQN (Mnih et al., 2015), we have used the publicly available code 2 of Barreto et al.

Original: Thus, implementation details as the neural network architectures that are used, the hyperparameters (replay buffer sizes, learning rates etc.) and the way the
Revised: Thus, implementation details as the neural network architectures that are used, the hyperparameters (replay buffer sizes, learning rates etc.) and the way the

Original: SFs and state-action value functions are learned can all be found both in the provided link and in the supplementary material of Barreto et al.
Revised: SFs and state-action value functions are learned can all be found both in the provided link and in the supplementary material of Barreto et al.

Original: We have also followed the same experimental protocol provided in the supplementary material of Barreto et al.[Figure]
Revised: We have also followed the same experimental protocol provided in the supplementary material of Barreto et al.

Original: ( − √ [Equation] [Equation] [Equation]
Revised: B. P RIOR D IVERSE P OLICY S ET C ONSTRUCTION M ETHODS

Original: [Equation]

Original: [Equation] = [Equation] Before moving on to the implementation details of the prior diverse policy set construction methods, we start this section by deﬁning the concept of reward equivalent policies  .
Revised: Before moving on to the implementation details of the prior diverse policy set construction methods, we start this section by deﬁning the concept of reward equivalent policies (REP) .

Original: This deﬁnition allows for a simple representation of the policy sets that are constructed by these prior methods in terms of the policy sets that can be induced by the nine preference vectors in Fig.
Revised: This deﬁnition allows for a simple representation of the policy sets that are constructed by these prior methods in terms of the policy sets that can be induced by the nine preference vectors in Fig.

Original: Deﬁnition 3 (Reward Equivalent Policies) .
Revised: Deﬁnition 3 (REP) .

Original: Let π i and π
Revised: Let π i and π

Original: j be two policies that are induced by tasks w i and w j , respectively.
Revised: j be two policies that are induced by tasks w i and w j , respectively.

Original: π i and π j are deﬁned to be reward equivalent if they achieve the same expected total reward, i.e. if:
Revised: π i and π j are deﬁned to be REP if they achieve the same expected total reward on an arbitrary task w , i.e. if:

Original: [Equation]
Revised: [Equation]

Original: It should be noted that even if two policies are induced by two different tasks, they can still be reward equivalent.
Revised: It should be noted that even if two policies are induced by two different tasks, they can still be REP.

Original: For instance, the policies induced by the tasks (0 . 6 0 .
Revised: For instance, the policies induced by the tasks (0 . 6 0 .

Original: 8) and (0 . 8 0 . 6) are reward equivalent as they achieve the same expected total reward by collecting all the items in the environment depicted in Fig.
Revised: 8) and (0 . 8 0 . 6) are reward equivalent as they achieve the same expected total reward by collecting all the items in the environment depicted in Fig.

Original: 1a.
Revised: 1a.

Original: Note that Deﬁnition 3 only takes into account what the policies achieve and not how they exactly achieve it. 
Revised: Note that Deﬁnition 3 only takes into account what the policies achieve and not how they exactly achieve it. keyboard

Original:  Importantly, Deﬁnition 3 allows for deﬁning equivalences between policy sets.
Revised: [Equation] Importantly, Deﬁnition 3 allows for deﬁning equivalences between policy sets .

Original: For instance, the policy set induced by tasks (0 . 6 0 .
Revised: For instance, the policy set induced by tasks (0 . 6 0 .

Original: 8) and (0 .
Revised: 8) and (0 .

Original: 6 − 0 .
Revised: 6 − 0 .

Original: 8) contains a policy that collects both of the items (reward equivalent to π 3 ) and a policy that collects the ﬁrst item while avoiding the second onekeyboard (reward equivalent to π 5 ).
Revised: 8) contains a policy that collects both of the items (reward equivalent to π 3 ) and a policy that collects the ﬁrst item while avoiding the second one (reward equivalent to π 5 ).

Original: The policy set Π 35 also contains policies that achieve the same tasks.
Revised: The policy set Π 35 also contains policies that achieve the same tasks.

Original: Thus, the former policy set can be considered to be reward equivalent to the policy set Π 35 .
Revised: Thus, the former policy set can be considered to be reward equivalent to the policy set Π 35 .

Original: Note that another policy set induced by tasks (0 . 8 0 . 6) and (0 . 8 − 0 . 6) can also be considered to be reward equivalent to the policy set Π 35 .
Revised: Note that another policy set induced by tasks (0 . 8 0 . 6) and (0 . 8 − 0 . 6) can also be considered to be reward equivalent to the policy set Π 35 .

Original: In the rest of this section, we will use this reward equivalence relation for a simple representation of the policy sets constructed by prior methods (in terms of the policy sets that can be induced by the preference vectors in Fig. 7).
Revised: In the rest of this section, we will use this reward equivalence relation for a simple representation of the policy sets constructed by prior methods (in terms of the policy sets that can be induced by the preference vectors in Fig. 7).

Original: DIAYN (Eysenbach et al., 2018).
Revised: DIAYN (Eysenbach et al., 2018).

Original: Among the prior diverse policy set construction methods that we have used, all except DIAYN are compatible with the GPE & GPI framework.
Revised: Among the prior diverse policy set construction methods that we have used, all except DIAYN are compatible with the GPE & GPI framework.

Original: Thus, in order to also make DIAYN compatible, while discovering a set of diverse policies, we have concurrently learned the SFs of the policies in this set so that GPE & GPI can be performed later on.
Revised: Thus, in order to also make DIAYN compatible, while discovering a set of diverse policies, we have concurrently learned the SFs of the policies in this set so that GPE & GPI can be performed later on.

Original: Importantly, rather than encouraging diversity in the state visitation proﬁle, we have encouraged diversity in the feature visitation proﬁle by feeding the visited feature vectors φ as input to the discriminator (as opposed to feeding the visited states as in regular DIAYN).
Revised: Importantly, rather than encouraging diversity in the state visitation proﬁle, we have encouraged diversity in the feature visitation proﬁle by feeding the visited feature vectors φ as input to the discriminator (as opposed to feeding the visited states as in regular DIAYN).

Original: This is because we are interested in policies that are able to obtain certain features, as opposed to ones that just push the agent to different parts of the state space.
Revised: This is because we are interested in policies that are able to obtain certain features, as opposed to ones that just push the agent to different parts of the state space.

Original: For the RL algorithm, we have used a vanilla policy gradient algorithm with an additional entropy loss term, for preventing the policy from becoming deterministic in the early stages of training.
Revised: For the RL algorithm, we have used a vanilla policy gradient algorithm with an additional entropy loss term, for preventing the policy from becoming deterministic in the early stages of training.

Original: After setting the number of policies to be discovered to 3 (as there are 3 possible feature vectors in the environment depicted in Fig.
Revised: After setting the number of policies to be discovered to 3 (as there are 3 possible feature vectors in the environment depicted in Fig.

Original: 1a: (1 0) , (0 1) and (0 0) ), learning the SFs of these policies and running the GPE & GPI compatible version of DIAYN described above, we have observed that the discovered policies either end up being policies that obtain both of the items (reward equivalent to π 3 ) or policies that obtain none (reward equivalent to π 7 ), corresponding to a policy set that is reward equivalent to the policy set Π 37 (see Fig. 7).
Revised: 1a: (1 0) , (0 1) and (0 0) ), learning the SFs of these policies and running the GPE & GPI compatible version of DIAYN described above, we have observed that the discovered policies either end up being policies that obtain both of the items (reward equivalent to π 3 ) or policies that obtain none (reward equivalent to π 7 ), corresponding to a policy set that is reward equivalent to the policy set Π 37 (see Fig. 7).

Original: Thus, the results presented in Fig.
Revised: Thus, the results presented in Fig.

Original: 4 corresponds to the results obtained with the policy set Π 37 .
Revised: 4 corresponds to the results obtained with the policy set Π 37 .

Original: We have also experimented with larger numbers of policies to be discovered, however, we observed no difference in the qualitative behavior of the policies that were discovered.
Revised: We have also experimented with larger numbers of policies to be discovered, however, we observed no difference in the qualitative behavior of the policies that were discovered.

Original: In our experiments, for the representation of the policy we have used the same neural network architecture that was provided in Barreto et al.
Revised: In our experiments, for the representation of the policy we have used the same neural network architecture that was provided in Barreto et al.

Original: The hyperparameters of our implementation are provided in Table 1.
Revised: The hyperparameters of our implementation are provided in Table 1.

Original: [Table]
Revised: [Table]

Original: SMP (Zahavy et al., 2020).
Revised: SMP (Zahavy et al., 2020).

Original: After initializing the preference vector to an arbitrary vector on the surface of the (cid:96) 2 2-dimensional ball (see Fig. 7) and running the SMP algorithm till termination, we have observed that for each initialization of preference vector, different policy sets were constructed.
Revised: After initializing the preference vector to an arbitrary vector on the surface of the (cid:96) 2 2-dimensional ball (see Fig. 7) and running the SMP algorithm till termination, we have observed that for each initialization of preference vector, different policy sets were constructed.

Original: The resulting policy sets end up being reward equivalent to one of the following sets: Π 37 , Π 17 , Π 157 , Π 57 , Π 517 , Π 27 , Π 47 , Π 7 , Π 87 , Π 857 , Π 8517 , Π 617 , Π 6157 and Π 67 .
Revised: The resulting policy sets end up being reward equivalent to one of the following sets: Π 37 , Π 17 , Π 157 , Π 57 , Π 517 , Π 27 , Π 47 , Π 7 , Π 87 , Π 857 , Π 8517 , Π 617 , Π 6157 and Π 67 .

Original: Thus, the results presented in Fig.
Revised: Thus, the results presented in Fig.

Original: 4 correspond to the average results obtained by these policy sets.
Revised: 4 correspond to the average results obtained by these policy sets.

Original: DSP (Zahavy et al., 2021).
Revised: DSP (Zahavy et al., 2021).

Original: After initializing the preference vector to an arbitrary vector on the surface of the (cid:96) 2 2-dimensional ball (see Fig. 7), setting T = 2 (see Algorithm 1 in Zahavy et al.
Revised: After initializing the preference vector to an arbitrary vector on the surface of the (cid:96) 2 2-dimensional ball (see Fig. 7), setting T = 2 (see Algorithm 1 in Zahavy et al.

Original: (2021) for the details) and running the DSP algorithm, we have observed that for each initialization of preference vector, different policy sets were constructed.
Revised: (2021) for the details) and running the DSP algorithm, we have observed that for each initialization of preference vector, different policy sets were constructed.

Original: The resulting policy sets end up being reward equivalent to one of the following sets: Π 37 , Π 58 , Π 16 , Π 27 , Π 47 , Π 79 , Π 68 and Π 86 .
Revised: The resulting policy sets end up being reward equivalent to one of the following sets: Π 37 , Π 58 , Π 16 , Π 27 , Π 47 , Π 79 , Π 68 and Π 86 .

Original: Thus, the results presented in Fig.
Revised: Thus, the results presented in Fig.

Original: 4 correspond to the average results obtained by these policy sets.
Revised: 4 correspond to the average results obtained by these policy sets.

Original: We have also experimented with larger T values, however, after T = 2 the only policy to be added to the set is a policy that is reward equivalent to π 7 .
Revised: We have also experimented with larger T values, however, after T = 2 the only policy to be added to the set is a policy that is reward equivalent to π 7 .

Original: For instance, when T = 3 , the resulting policy sets end up being reward equivalent to one of the following sets: Π 377 , Π 587 , Π 167 , Π 277 , Π 477 , Π 797 and Π 867 .
Revised: For instance, when T = 3 , the resulting policy sets end up being reward equivalent to one of the following sets: Π 377 , Π 587 , Π 167 , Π 277 , Π 477 , Π 797 and Π 867 .

Original: Since π 7 is a policy that avoids the items in the environment depicted in Fig.
Revised: Since π 7 is a policy that avoids the items in the environment depicted in Fig.

Original: 1a, it is not a useful policy for the tasks considered in this study.
Revised: 1a, it is not a useful policy for the tasks considered in this study.

Original: Thus, we reported results only for the case of T = 2 .
Revised: Thus, we reported results only for the case of T = 2 .

Original: C U NNORMALIZED R ESULTS
Revised: B. M

Revised: [Equation] QI [Equation]

Revised: MaxQInit (Abel et al., 2018) is a lifelong RL approach in which the value function of an agent is initialized to the best possible value that minimizes the learning time in a new downstream tasks, while at the same time preserving PAC guarantees.

Revised: More speciﬁcally, before starting a new task, the value function is initialized as follows:

Revised: [Equation]

Revised: 10) where ˆ M is the set of MDPs that the agent has sampled so far, and Q M is the value function that the agent learned from interacting with each MDP.

Revised: However, in order to allow for a fair comparison with Algorithm 1, we have allowed for ˆ M to be equal to M , where M is the set of all MDPs (see (3)).

Revised: In the GPE & GPI framework, (10) corresponds to initializing the value functions using to the values of the task w 3 = (+ √ 1 / 2 + √ 1 / 2) (see Fig.

Revised: 7) as it is with this task that Q M is the maximized.

Revised: Thus, the results presented in Fig.

Revised: 6 correspond to the results obtained by initializing the value function of DQN (Mnih et al., 2015) to the value function of the task w 3 right at the beginning of each new task.

Revised: C U NNORMALIZED R ESULTS

Original: In this section, we provide the unnormalized results for the experiments in Section 5.
Revised: In this section, we provide the unnormalized results for the experiments in Section 5.

Original: The r- and y-axis of the plots now indicates the sum of rewards as it is, without any normalization. 
Revised: The r- and y-axis of the plots now indicates the sum of rewards as it is, without any normalization. [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: Even though we have developed our theoretical results by assuming MDPs with deterministic transition functions, in order to test the applicability of our results with stochastic transition functions, we also performed experiments in the stochastic version of the 2D item collection environment with “slip” probabilities 0 . 125 and 0 .

Original: [Figure]
Revised: Results are shown in Fig. 12 and 13. As can be seen, even in stochastic settings, π GPI Π 15 is able to perform better across all downstream tasks compared to π GPI Π 24 and π GPI Π 3 . It can also be seen that adding more policies to the independent policy set π GPI Π 15 again does not result in a difference in the downstream task performance.

Original: [Figure] 
Revised: [Figure] [Figure]