Original: Building a Subspace of Policies for Scalable Continual Learning
Revised: Building a Subspace of Policies for Scalable Continual Learning

Original: Abstract
Revised: Abstract

Original: The ability to continuously acquire new knowledge and skills is crucial for autonomous agents.
Revised: The ability to continuously acquire new knowledge and skills is crucial for autonomous agents.

Original: However, existing methods are typically based on either fixedsize models that cannot capture many diverse behaviors, or growing-size modelsthat scale poorly with the number of tasks.
Revised: However, existing methods are typically based on either fixed-size models that cannot capture many diverse behaviors, or growing-size models that scale poorly with the number of tasks.

Original: In this paper, we introduceContinual
Revised: In this paper, we introduce

Original:  Subspace of Policies (CSP), a method that iteratively learns a subspace of policiesin the continual reinforcement learning setting where tasks are presented sequentially.
Revised: Continual Subspace of Policies (CSP), a method that iteratively learns a subspace of policies in the continual reinforcement learning setting where tasks are presented sequentially.

Original: The subspace’s high expressivity allows our method to strike a good balancebetween stability ( i.e. not forgetting prior tasks) and plasticity ( i.e. learning newtasks), while the number of parameters grows sublinearly with the number of tasks.
Revised: The subspace’s high expressivity allows our method to strike a good balance between stability ( i.e. not forgetting prior tasks) and plasticity ( i.e. learning new tasks), while the number of parameters grows sublinearly with the number of tasks.

Original: In addition, CSP displays good transfer , being able to quickly adapt to new tasksincluding combinations of previously seen ones without additional training.
Revised: In addition, CSP displays good transfer , being able to quickly adapt to new tasks including combinations of previously seen ones without additional training.

Original: Finally, CSP outperforms state-of-the-art methods on a wide range of scenarios in twodifferent domains.
Revised: Finally, CSP outperforms state-of-the-art methods on a wide range of scenarios in two different domains.

Original: An interactive visualization of the subspace can be found athttps://share.streamlit.io/continual-subspace/policies/main .
Revised: An interactive visualization of the subspace can be found at https://continual-subspace-policies-streamlit-app-gofujp. com/ .

Original:  Introduction
Revised:  Introduction

Original: Developing autonomous agents that can continuously acquire new knowledge and skills is a key openchallenge in AI.
Revised: Developing autonomous agents that can continuously acquire new knowledge and skills is a key open challenge in AI.

Original: This problem is referred to as continual reinforcement learning (CRL) and solving itis crucial for large-scale deployment of autonomous agents in non-stationary domains such as roboticsor dialogue systems.
Revised: This problem is referred to as continual reinforcement learning (CRL) and solving it is crucial for large-scale deployment of autonomous agents in non-stationary domains such as robotics or dialogue systems.

Original: In recent years, there has been a growing interest in this problem [38, 22, 40, 36].
Revised: In recent years, there has been a growing interest in this problem [38, 22, 40, 36]

Original: As suggested in [40], the balance between stability ( i e. no forgetting of prior tasks), plasticity( i e. positive transfer to new tasks including combinations of previously seen ones and the ability toexpress many diverse behaviors), and scalability ( i.e. memory increases sublinearly with the numberof tasks) is crucial for designing effective CRL methods.
Revised: As suggested in [40] the balance between stability ( i.e. no forgetting of prior tasks), plasticity ( i.e. positive transfer to new tasks including combinations of previously seen ones and the ability to express many diverse behaviors), and scalability ( i e. memory increases sublinearly with the number of tasks) is crucial for designing effective CRL methods.

Original: While current methods perform well along some of these dimensions, they tend to suffer alongothers [32, 49].
Revised: While current methods perform well along some of these dimensions, they tend to suffer along others [32, 49].

Original: One class of methods aims to alleviate forgetting by storing previously trainedmodels [38, 9, 48] or maintaining replay buffers with trajectories from prior tasks [24, 18, 34, 36].
Revised: One class of methods aims to alleviate forgetting by storing previously trained models [38, 9, 48] or maintaining replay buffers with trajectories from prior tasks [24, 18, 34, 36].

Original: However, these methods scale poorly with memory and compute, especially as the number andcomplexity of the tasks increase, rendering them unfeasible for real-world applications.
Revised: However, these methods scale poorly with memory and compute, especially as the number and complexity of the tasks increase, rendering them unfeasible for real-world applications.

Original: Anotherbody of work focuses on improving transfer to new tasks [17, 37, 23, 5, 20, 46], but these methodstend to forget previously learned skills.
Revised: Another body of work focuses on improving transfer to new tasks [17, 37, 23, 5, 20, 46], but these methods tend to forget previously learned skills.

Original: A few approaches attempt to find a better balance betweentransfer and forgetting [22, 40].
Revised: A few approaches attempt to find a better balance between transfer and forgetting [22, 40].

Original: However, because they train a single model, these methods struggleto express a wide range of diverse behaviors.
Revised: However, because they train a single model, these methods struggle to express a wide range of diverse behaviors.

Original: In this paper, we take inspiration from the mode connectivity literature [13] to develop a novel CRLmethod by iteratively learning a subspace of policies [14].
Revised: In this paper, we take inspiration from the mode connectivity literature [13] to develop a novel CRL method by iteratively learning a subspace of policies [14].

Original: Our method, Continual Subspace of
Revised: Our method, Continual Subspace of

Original: Policies or CSP for short, aims to strike a good balance between stability, plasticity, and scalability.
Revised: Policies or CSP for short, aims to strike a good balance between stability, plasticity, and scalability.

Original: Instead of learning a single policy, CSP maintains an entire subspace of policies defined as a convex hull in parameter space.
Revised: Instead of learning a single policy, CSP maintains an entire subspace of policies defined as a convex hull in parameter space.

Original: The vertices of this convex hull are called anchors, with each anchorrepresenting the parameters of a policy.
Revised: The vertices of this convex hull are called anchors, with each anchor representing the parameters of a policy.

Original: This subspace captures a large number of diverse behaviorswhich enables efficient training on a wide range of tasks.
Revised: This subspace captures a large number of diverse behaviors which enables efficient training on a wide range of tasks.

Original: At every stage of the CRL process, thebest found policy for a previously seen task is represented as a single point in the current subspace( i.e. unique convex combination of the anchors).
Revised: At every stage of the CRL process, the best found policy for a previously seen task is represented as a single point in the current subspace ( i.e. unique convex combination of the anchors).

Original: This enables cheap storage and retrieval of priorsolutions.
Revised: This enables cheap storage and retrieval of prior solutions.

Original: If a new task is a combination of previously seen ones, a good policy can be found in thecurrent subspace without increasing the number of parameters.
Revised: If a new task is a combination of previously seen ones, a good policy can be found in the current subspace without increasing the number of parameters.

Original: On the other hand, if a new task isvery different from previously seen ones, CSP extends the current subspace by adding another anchor,and learns a new policy in the extended subspace.
Revised: On the other hand, if a new task is very different from previously seen ones, CSP extends the current subspace by adding another anchor, and learns a new policy in the extended subspace.

Original: In this case, the expressivity of the subspace willincrease allowing our model to deal with more diverse tasks in the future, and thus ensuring goodplasticity.
Revised: In this case, the expressivity of the subspace will increase allowing our model to deal with more diverse tasks in the future, and thus ensuring good plasticity.

Original: See Figure 1 for an overview of our method.
Revised: See Figure 1 for an overview of our method.

Original: To summarize, in this paper we make the following contributions: (i) we introduce CSP, a new
Revised: To summarize, in this paper we make the following contributions: (i) we introduce CSP, a new

Original: CRL method based on adaptively building a subspace of policies which allows to cheaply store andretrieve solutions to prior tasks, provides positive transfer and grows sublinearly w.r.t the number oftasks, (ii) we design a number of continuous control environments to evaluate specific abilities of
Revised: CRL method based on adaptively building a subspace of policies which allows to cheaply store and retrieve solutions to prior tasks, provides positive transfer and grows sublinearly w.r.t the number of tasks, (ii) we design a number of continuous control environments to evaluate specific abilities of

Original: CRL agents, (iii) we illustrate our method’s ability to learn diverse task sequences and transfer tocompositional tasks, (iv) and we show that CSP outperforms state-of-the-art CRL methods on a suiteof continuous control scenarios in both Brax [12] and Continual World [49].
Revised: CRL agents, (iii) we illustrate our method’s ability to learn diverse task sequences and transfer to compositional tasks, (iv) and we show that CSP outperforms state-of-the-art CRL methods on a suite of continuous control scenarios in both Brax [12] and Continual World [49].

Original:  Problem Setting
Revised:  Problem Setting

Original: A continual learning problem is defined by a sequence of N tasks denoted t 1 ,...,t N .
Revised: A continual learning problem is defined by a sequence of N tasks denoted t 1 ,...,t N .

Original: The task i isdefined by a Markov Decision Process (MDP) M i = ⟨S
Revised: The task i is defined by a Markov Decision Process (MDP) M i = ⟨S

Original: i , A i , T i ,r i ,γ ⟩ with a set of states S i , a set ofactions
Revised: i , A i , T i ,r i ,γ ⟩ with a set of states S i , a set of actions

Original: A i , a transition function T i ∶ S
Revised: A i , a transition function T i ∶ S

Original: i ×
Revised: i ×

Original: A i → P(S i ) , and a reward function r i ∶ S
Revised: A i → P(S i ) , and a reward function r i ∶ S

Original: i ×
Revised: i ×

Original: A i → R .
Revised: A i → R .

Original: For58 the sake of simplicity, we consider here that all the tasks have the same state space denoted S andaction space A .
Revised: For the sake of simplicity, we consider here that all the tasks have the same state space denoted S and action space A .

Original: We also define a global policy Π ∶ [ 1 ..N ] × Z → (S → P(A)) which takes as input60 a task id i and a sequence of tasks Z , and outputs the policy which should be used when interactingwith task i .
Revised: We also define a global policy Π ∶ [ 1 ..N ] × Z → (S → P(A)) which takes as input a task id i and a sequence of tasks Z , and outputs the policy which should be used when interacting with task i .

Original: P(A) is a probability distribution over the action space.
Revised: P(A) is a probability distribution over the action space.

Original: Budgeted Constraints and Training.
Revised: Budgeted Constraints and Training.

Original: We consider that each task i is associated with a trainingbudget denoted b i .
Revised: We consider that each task i is associated with a training budget denoted b i .

Original: It means that, when facing task i , the learning algorithm can only execute b iactions.
Revised: It means that, when facing task i , the learning algorithm can only execute b i actions.

Original: Then the system will switch to task t i + 1 and will no longer have access to transitions fromthe MDP M i .
Revised: Then the system will switch to task t i + 1 and will no longer have access to transitions from the MDP M i .

Original: We do not assume access to a replay buffer of trajectories from past MDPs since thiswould lead to a significant increase in memory cost.
Revised: We do not assume access to a replay buffer of trajectories from past MDPs since this would lead to a significant increase in memory cost.

Original:  Continual Subspace of Policies (CSP)3.1 Subspace of Policies
Revised:  Continual Subspace of Policies (CSP) 3.1 Subspace of Policies

Original: Our work builds on top of [14] by leveraging a subspace of policies.
Revised: Our work builds on top of [14] by leveraging a subspace of policies.

Original: However, instead of using thesubspace to train on a single environment and adapt to new ones at test time, we use it to efficientlylearn tasks sequentially in the continual RL setting.
Revised: However, instead of using the subspace to train on a single environment and adapt to new ones at test time, we use it to efficiently learn tasks sequentially in the continual RL setting.

Original: This requires designing a new approach forlearning the subspace, as detailed below.
Revised: This requires designing a new approach for learning the subspace, as detailed below.

Original: Let θ ∈ R d represent the parameters of a policy, which we denote by π ( a ∣ s,θ ) .
Revised: Let θ ∈ R d represent the parameters of a policy, which we denote by π ( a ∣ s,θ ) .

Original: A subspace of policiesis a simplex in R d defined by a set of anchors Θ = { θ 1 ,...,θ k } , where any policy in this simplex canbe represented as a convex combination of the anchor parameters.
Revised: A subspace of policies is a simplex in R d defined by a set of anchors Θ = { θ 1 ,...,θ k } , where any policy in this simplex can be represented as a convex combination of the anchor parameters.

Original: Let the weight vector α ∈ R k + with∥ α ∥ 1 = 1 denote the coefficients of the convex combination.
Revised: Let the weight vector α ∈ R k + with ∥ α ∥ 1 = 1 denote the coefficients of the convex combination.

Original: Therefore, each value of α uniquelydefines a policy with π ( a ∣ s, [ α, Θ ])
Revised: Therefore, each value of α uniquely defines a policy with π ( a ∣ s, [ α, Θ ])

Original: =
Revised: =

Original: π ( a ∣ s, ∑ α i θ i ) .
Revised: π ( a ∣ s, ∑ α i θ i ) .

Original: Throughout the paper, we sometimes refer tothe weight vector α and its corresponding policy π ( a ∣ s, [ α, Θ ]) interchangeably (since a one-to-onemapping is defined at each point in training).
Revised: Throughout the paper, we sometimes refer to the weight vector α and its corresponding policy π ( a ∣ s, [ α, Θ ]) interchangeably (since a one-to-one mapping is defined at each point in training).

Original: 3.2 Learning Algorithm
Revised: 3.2 Learning Algorithm

Original: We propose to adaptively construct a subspace of policies for the continual RL setting in which anagent learns tasks sequentially.
Revised: We propose to adaptively construct a subspace of policies for the continual RL setting in which an agent learns tasks sequentially.

Original: We call our method Continual Subspace of Policies or CSP forshort.
Revised: We call our method Continual Subspace of Policies or CSP for short.

Original: Our model builds a sequence Θ 1 ,..., Θ N of subspaces, one new subspace after each training task,with each subspace extending the previous one.
Revised: Our model builds a sequence Θ 1 ,..., Θ N of subspaces, one new subspace after each training task, with each subspace extending the previous one.

Original: Note that ∣ Θ j ∣ ≤ j, ∀ j ∈ 1 ... N since the subspacegrows sublinearly with the number of tasks (as explained below).
Revised: Note that ∣ Θ j ∣ ≤ j, ∀ j ∈ 1 ...N since the subspace grows sublinearly with the number of tasks (as explained below).

Original: Hence, the number of anchors m ofa subspace Θ j is not the same as the number of tasks j used to create Θ j .
Revised: Hence, the number of anchors m of a subspace Θ j is not the same as the number of tasks j used to create Θ j .

Original: The learned policies arerepresented as single points in these subspaces.
Revised: The learned policies are represented as single points in these subspaces.

Original: At each stage, CSP maintains both a set of anchorsdefining the current subspace, as well as the weights α corresponding to the best policies foundfor all prior tasks.
Revised: At each stage, CSP maintains both a set of anchors defining the current subspace, as well as the weights α corresponding to the best policies found for all prior tasks.

Original: The (best found) policy for task i after training on the first j tasks is denoted asπ ji ( a ∣ s ) and can be represented as a point in the subspace Θ j with a weight vector denoted α ji suchthat π ji ( a ∣ s )
Revised: The (best found) policy for task i after training on the first j tasks is denoted as π ji ( a ∣ s ) and can be represented as a point in the subspace Θ j with a weight vector denoted α ji such that π j i ( a ∣ s )

Original: =
Revised: =

Original: π ( a ∣ s, [ α ji , Θ j ]) .
Revised: π ( a ∣ s, [ α j i , Θ j ]) .

Original: Given a set of anchors Θ j and a set of previously learned policies { α j 1 ,... ,α jj } , updating our modelon the new task t j + 1 produces a new subspace Θ j + 1 and a new set of weights { α j + 1 1 ,... ,α j + 1 j + 1 } .
Revised: Given a set of anchors Θ j and a set of previously learned policies { α j 1 ,... ,α jj } , updating our model on the new task t j + 1 produces a new subspace Θ j + 1 and a new set of weights { α j + 1 1 ,... ,α j + 1 j + 1 } .

Original: Thereare two possible cases.
Revised: There are two possible cases.

Original: One possibility is that the current subspace already contains a good policy forthe new task, so we just need to find the weight vector corresponding to a policy which performs wellon the new task.
Revised: One possibility is that the current subspace already contains a good policy for the new task, so we just need to find the weight vector corresponding to a policy which performs well on the new task.

Original: The other possibility is that the current subspace does not contain a good policyfor the new task.
Revised: The other possibility is that the current subspace does not contain a good policy for the new task.

Original: In this case, the algorithm produces a new subspace by adding one anchor to theprevious one (see next section), and converts the previous policies to be compatible with this newsubspace.
Revised: In this case, the algorithm produces a new subspace by adding one anchor to the previous one (see next section), and converts the previous policies to be compatible with this new subspace.

Original: To achieve this, CSP operates in two phases:
Revised: To achieve this, CSP operates in two phases:

Original: Grow the current subspace by adding a new anchor and learning the best possible policy for task j + 1 in this subspace.
Revised: Grow the current subspace by adding a new anchor and learning the best possible policy for task j + 1 in this subspace.

Original: Compare the quality of this policy with the best possible policy expressed in the previous subspace.
Revised: Compare the quality of this policy with the best possible policy expressed in the previous subspace.

Original: Based on this comparison, decide whether to extend the subspace to match the new one or prune it back to the previous one.
Revised: Based on this comparison, decide whether to extend the subspace to match the new one or prune it back to the previous one.

Original: See Algorithm 17 for the pseudo-code of our method.
Revised: See Algorithm 17 for the pseudo-code of our method.

Original: We now describe in detail the two phases ofthe learning process, namely how we grow the subspace and how we decide whether to extend orprune the subspace 3.3 Grow the Subspace
Revised: We now describe in detail the two phases of the learning process, namely how we grow the subspace and how we decide whether to extend or prune the subspace.

Original: Given the current subspace Θ j composed of m ≤ j anchors (with j being the number of tasks seen sofar), a new subspace ˜Θ j + 1 is built as follows.

Original: [-First a new anchor denoted θ j + 1 is added to the set ofanchors such that ˜Θ j + 
-]
Revised: [-First a new anchor denoted θ j + 1 is added to the set ofanchors such that ˜Θ j + 
-]

Original: =

Original: Θ j ⋃{ θ j + 1 } .

Original: With all previous anchors frozen, we train the new anchor by

Original: Input: θ 1 ,..., θ j (previous anchors), B (budget), ϵ (threshold)
Revised: Input: θ 1 ,..., θ j (previous anchors), B (budget), ϵ (threshold)

Original: Initialize: W ϕ (subspace critic), B uf (replay buffer)
Revised: Initialize: W ϕ (subspace critic), B uf (replay buffer)

Original: Define θ j + 1 ← 1 j ∑ ji = 1 θ i (new anchor) ; // Grow the Subspace 2 for i = 1 ,...,
Revised: Define θ j + 1 ← 1 j ∑ ji = 1 θ i (new anchor) ; // Grow the Subspace 2 for i = 1 ,...,

Original: B do
Revised: B do

Original: Sample α ∼ Dir ( j + 1 )
Revised: Sample α ∼ Dir ( j + 1 )

Original: Set policy parameters θ α ← ∑ j + 1 i = 

Revised: Set policy parameters θ α ← ∑ j + 1 i =

Original: α i θ i for l = 1 ,...,K do
Revised: α i θ i for l = 1 ,...,K do

Original: Collect and store ( s,a,r,s ′ ,α ) in B uf by sampling a ∼ π θ α (⋅∣ s ) end if time to update then
Revised: Collect and store ( s,a,r,s ′ ,α ) in B uf by sampling a ∼ π θ α (⋅∣ s ) end if time to update then

Original: Update π θ j + 1 and W ϕ using the SAC algorithm and the replay buffer B uf end 11 end 12 Use B uf and W ϕ to estimate: ; // Extend or Prune the Subspaceα old ← arg max ( α, 0 ) with α ∈ R m + , ∥ α ∥ 1 = 1 W ϕ ( α ) α new ← arg max α ∈ R m + 1 + , ∥ α ∥ 1 = 1 W ϕ ( α ) if W ϕ (⋅ ,α new ) > ( 1 + ϵ ) ⋅ W ϕ (⋅ ,α old ) then
Revised: Update π θ j + 1 and W ϕ using the SAC algorithm and the replay buffer B uf end 11 end 12 Use B uf and W ϕ to estimate: ; // Extend or Prune the Subspaceα old ← arg max ( α, 0 ) with α ∈ R m + , ∥ α ∥ 1 = 1 W ϕ ( α ) α new ← arg max α ∈ R m + 1 + , ∥ α ∥ 1 = 1 W ϕ ( α ) if W ϕ (⋅ ,α new ) > ( 1 + ϵ ) ⋅ W ϕ (⋅ ,α old ) then

Original: Return: θ 1 ,...,θ j ,θ j + 1 ,α new ; // Extend 15 else
Revised: Return: θ 1 ,...,θ j ,θ j + 1 ,α new ; // Extend 15 else

Original: Return: θ 1 ,...,θ j ,α old ; //
Revised: Return: θ 1 ,...,θ j ,α old ; //

Original: Prune 17 end sampling α values from a Dirichlet distribution and derive updates from a similar objective as  the oneproposed in [14]:{+set of anchors such that ˜Θ j + 
+}
Revised: Prune 17 end 3.3 Grow the Subspace Given the current subspace Θ j composed of m ≤ j anchors (with j being the number of tasks seen so far), a new subspace ˜Θ j + 1 is built as follows. First a new anchor denoted θ j + 1 is added to the {+set of anchors such that ˜Θ j + 
+}

Original: [Equation]
Revised: = Θ j ⋃{ θ j + 1 } .

Revised: With all previous anchors frozen, we train the new anchor by sampling α values from a Dirichlet distribution and derive updates from a similar objective as the one proposed in [14]:

Original: [Equation]
Revised: [Equation]

Original: ) where R j + 1 ( τ ) is the return obtained on task j + 1 throughout trajectory τ which was generated usingpolicy π ( a ∣ s, [ α, ˜Θ j + 1 )] .
Revised: where R j + 1 ( τ ) is the return obtained on task j + 1 throughout trajectory τ which was generated using policy π ( a ∣ s, [ α, ˜Θ j + 1 )] .

Original: Note that the anchor is trained such that not only one but all possible valuesof α tend to produce a good policy.
Revised: Note that the anchor is trained such that not only one but all possible values of α tend to produce a good policy.

Original: The resulting subspace thus aims at containing as many goodpolicies as possible for task j + 1 .
Revised: The resulting subspace thus aims at containing as many good policies as possible for task j + 1 .

Original: 3.4 Extend or Prune the Subspace
Revised: 3.4 Extend or Prune the Subspace

Original: To decide if the new anchor is kept, we propose to simply compare the best possible policy (for taskj + 1 ) in the new subspace with the best possible policy for the same task in the previous subspace( i.e. without using the new anchor).
Revised: To decide if the new anchor is kept, we propose to simply compare the best possible policy (for task j + 1 ) in the new subspace with the best possible policy for the same task in the previous subspace ( i.e. without using the new anchor).

Original: Each policy could be evaluated via Monte-Carlo (MC) estimatesby doing additional rollouts in the environment and recording the average performance.
Revised: Each policy could be evaluated via Monte-Carlo (MC) estimates by doing additional rollouts in the environment and recording the average performance.

Original: However,this typically requires a large number ( e.g. millions) of interactions which may be impossible with alimited budget.
Revised: However, this typically requires a large number ( e.g. millions) of interactions which may be impossible with a limited budget.

Original: Thus, we propose an alternative procedure to make this evaluation sample efficient.
Revised: Thus, we propose an alternative procedure to make this evaluation sample efficient.

Original: For each task j and corresponding subspace Θ j , our algorithm also learns a Q -function Q ( s,a,α )which is trained to predict the expected return on task j for all possible states, actions, and all possibleα ’s in the corresponding subspace.
Revised: For each task j and corresponding subspace Θ j , our algorithm also learns a Q -function Q ( s,a,α ) which is trained to predict the expected return on task j for all possible states, actions, and all possible α ’s in the corresponding subspace.

Original: Our algorithm is based on SAC [15], so for each task, we collecta replay buffer of interactions B uf which contains all states and actions seen by the agent duringtraining.
Revised: Our algorithm is based on SAC [15], so for each task, we collect a replay buffer of interactions B uf which contains all states and actions seen by the agent during training.

Original: Thus, the Q -function Q ( s,a,α ) can help us directly estimate the quality W ( α ) of thepolicy represented by the weights α in the new subspace which can be computed as the average overall states and actions in the replay buffer:
Revised: Thus, the Q -function Q ( s,a,α ) can help us directly estimate the quality W ( α ) of the policy represented by the weights α in the new subspace which can be computed as the average over all states and actions in the replay buffer:

Original: [Equation]
Revised: [Equation]

Original: It is thus possible to compute the value of α corresponding to the best policy in the extended subspace(denoted α new ∈ R m + 1 + , ∣ α new ∥ 1 = 1 ):
Revised: It is thus possible to compute the value of α corresponding to the best policy in the extended subspace (denoted α new [Equation] R m + 1 + , [Equation]

Original: [Equation]
Revised: [Equation]

Original: as well as the value of α corresponding to the best policy in the previous subspace (denoted α old ∈137 R m + , ∥ α old ∥ 1 = 1 ):
Revised: as well as the value of α corresponding to the best policy in the previous subspace (denoted α old ∈ R m + , ∥ α old ∥ [Equation]

Original: [Equation]
Revised: [Equation]

Original: In practice, α new and α old are estimated by uniformly sampling a number of α ’s in the correspondingsubspace as well as a number of states and actions from the buffer.
Revised: In practice, α new and α old are estimated by uniformly sampling a number of α ’s in the corresponding subspace as well as a number of states and actions from the buffer.

Original: The quality of the new subspace and the previous one can thus be evaluated by comparing W ( α new )and W ( α old ) .
Revised: The quality of the new subspace and the previous one can thus be evaluated by comparing W ( α new ) and W ( α old ) .

Original: If W ( α new ) > ( 1 + ϵ ) ⋅ W ( α old ) , the subspace is extended to the new subspace ( i.e.the one after the grow phase): Θ j + 1 = ˜Θ j + 1 .
Revised: If W ( α new ) > ( 1 + ϵ ) ⋅ W ( α old ) , the subspace is extended to the new subspace ( i.e. the one after the grow phase): Θ j + 1 = ˜Θ j + 1 .

Original: Otherwise, the subspace is pruned back to the oldsubspace ( i.e. the one before the grow phase): Θ j + 1 = Θ j .
Revised: Otherwise, the subspace is pruned back to the old subspace ( i.e. the one before the grow phase): Θ j + 1 = Θ j .

Original: Note that, if the subspace is extended,the previously learned policies have to be mapped in the new subspace such that α j + 1 i ∈ R
Revised: Note that, if the subspace is extended, the previously learned policies have to be mapped in the new subspace such that α j + 1 i ∈ R

Original: j + 1{+
+} + i.e.∀ i ≤ j, α j + 1 i ∶= ( α ji , 0 ) and α j + 1 j + 1 ∶= α new .
Revised: j + {+
+} + i.e. ∀ i ≤ j, α j + 1 i ∶= ( α ji , 0 ) and α j + 1 j + 1 ∶= α new .

Original: If the subspace is not extended, then old values can bekept i.e. ∀ i ≤ j, α j + 1 i ∶= α ji and α j + 1 j + 1 ∶= α old .
Revised: If the subspace is not extended, then old values can be kept i.e. ∀ i ≤ j, α j + 1 i ∶= α ji and α j + 1 j + 1 ∶= α old .

Original: The policy π θ j + 1 and value function W ϕ are updated using SAC [15].
Revised: The policy π θ j + 1 and value function W ϕ are updated using SAC [15].

Original: See Appendix A for moredetails about the algorithm and our implementation of CSP.
Revised: See Appendix A for more details about the algorithm and our implementation of CSP.

Original: 3.5 Scalability of CSP
Revised: 3.5 Scalability of CSP

Original: By having access to an infinite number of policies, the subspace is highly expressive so it can capturea wide range of diverse behaviors.
Revised: By having access to an infinite number of policies, the subspace is highly expressive so it can capture a wide range of diverse behaviors.

Original: This enables positive transfer to many new tasks without the needfor additional parameters.
Revised: This enables positive transfer to many new tasks without the need for additional parameters.

Original: As a consequence, the number of parameters scales sublinearly with thenumber of tasks.
Revised: As a consequence, the number of parameters scales sublinearly with the number of tasks.

Original: The final size of the subspace depends on the sequence of tasks, with longer andmore diverse sequences requiring more anchors.
Revised: The final size of the subspace depends on the sequence of tasks, with longer and more diverse sequences requiring more anchors.

Original: The speed of growth is controlled by the threshold ϵwhich trades-off performance gains for memory efficiency ( i.e. the higher the ϵ the more performancelosses are tolerated to reduce memory costs).
Revised: The speed of growth is controlled by the threshold ϵ which trades-off performance gains for memory efficiency ( i.e. the higher the ϵ the more performance losses are tolerated to reduce memory costs).

Original: See Appendix A for more intuition about CSP.
Revised: See Appendix A for more intuition about CSP.

Original:  Experiments 
Revised:  Experiments 4.1 Environments

Original: 4.1 Environments

Original: We evaluate CSP on a number of CRL scenarios focused on both locomotion using Brax [12] as wellas robotic manipulation using Continual World [49].
Revised: We evaluate CSP on a number of CRL scenarios focused on both locomotion using Brax [12] as well as robotic manipulation using Continual World [49].

Original: For Continual World, we ran experiments on all8 triplet sequences proposed in the paper.
Revised: For Continual World, we ran experiments on all 8 triplet sequences proposed in the paper.

Original: These tasks were specifically designed to challenge bothtransfer and forgetting (see Appendix C for more details).
Revised: These tasks were specifically designed to challenge both transfer and forgetting (see Appendix C for more details).

Original: For Brax, we perform an in-depth study by designing a number of continuous control scenarios toseparately evaluate capabilities specific to CRL agents such as forgetting, transfer, or scalability.
Revised: For Brax, we perform an in-depth study by designing a number of continuous control scenarios to separately evaluate capabilities specific to CRL agents such as forgetting, transfer, or scalability.

Original: Foreach capability, we create two scenarios: one based on HalfCheetah and one based on Ant.
Revised: For each capability, we create two scenarios: one based on HalfCheetah and one based on Ant.

Original: First, wegenerate a large number of environments by changing the dynamics of the standard environment.
Revised: First, we generate a large number of environments by changing the dynamics of the standard environment.

Original: Weaimed to ground our environment variations in realistic situations such as increased or decreasedgravity, friction, or limb lengths.
Revised: We aimed to ground our environment variations in realistic situations such as increased or decreased gravity, friction, or limb lengths.

Original: Then, we design task sequences based on the transfer matrices forthese environments, following [49].
Revised: Then, we design task sequences based on the transfer matrices for these environments, following [49].

Original: Unless mentioned otherwise, each CRL scenario has 8 tasks inthe sequence.
Revised: Unless mentioned otherwise, each CRL scenario has 8 tasks in the sequence.

Original: See Appendix C for more details about our designed scenarios.
Revised: See Appendix C for more details about our designed scenarios.

Original: Forgetting and Transfer are evaluated by selecting sequences of tasks with significant negative andpositive backward transfer according to the matrix.
Revised: Forgetting and Transfer are evaluated by selecting sequences of tasks with significant negative and positive backward transfer according to the matrix.

Original: Robustness probes whether the method candeal with adversarial perturbations in the environment such as the actions being inverted for eachtask, which poses a significant challenge for both forgetting and transfer.
Revised: Robustness probes whether the method can deal with adversarial perturbations in the environment such as the actions being inverted for each task, which poses a significant challenge for both forgetting and transfer.

Original: Compositionality tests theagent’s ability of combining two previously seen skills to solve a new task.
Revised: Compositionality tests the agent’s ability of combining two previously seen skills to solve a new task.

Original: Scalability is tested bylooking at how performance and memory change with the number of tasks.
Revised: Scalability is tested by looking at how performance and memory change with the number of tasks.

Original: Learning Efficiency istested by using smaller budgets (see Section 2).
Revised: Learning Efficiency is tested by using smaller budgets (see Section 2).

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We compare CSP with a number of state-of-the-art CRL methods such as PNN [38], EWC [22],as well as with FT-1 which finetunes a single model on the entire sequence of tasks, FT-L2 whichis like FT-1 with an additional L 2 regularization applied during finetuning.
Revised: We compare CSP with a number of state-of-the-art CRL methods such as PNN [38], EWC [22], as well as with FT-1 which finetunes a single model on the entire sequence of tasks, FT-L2 which is like FT-1 with an additional L 2 regularization applied during finetuning.

Original: We also compare CSP with SAC-N which trains one model for each task from scratch.
Revised: We also compare CSP with SAC-N which trains one model for each task from scratch.

Original: While SAC-N avoids forgetting, itcannot transfer knowledge across tasks.
Revised: While SAC-N avoids forgetting, it cannot transfer knowledge across tasks.

Original: Finally, we compare CSP with a method called FT-N whichcombines the best of both SAC-N and FT-1.
Revised: Finally, we compare CSP with a method called FT-N which combines the best of both SAC-N and FT-1.

Original: Like SAC-N, it stores one model per task after trainingon it and like FT-1, it finetunes the previous model to promote transfer.
Revised: Like SAC-N, it stores one model per task after training on it and like FT-1, it finetunes the previous model to promote transfer.

Original: However, FT-N and SAC-Nscale poorly ( i.e. linearly with the number of tasks) with both memory and compute, which makesthem unfeasible for real-world applications.
Revised: However, FT-N and SAC-N scale poorly ( i.e. linearly with the number of tasks) with both memory and compute, which makes them unfeasible for real-world applications.

Original: Note that our method is not directly comparable with
Revised: Note that our method is not directly comparable with

Original: CLEAR [36] since we assume no access to data from prior tasks.
Revised: CLEAR [36] since we assume no access to data from prior tasks.

Original: Storing data from all prior tasks as
Revised: Storing data from all prior tasks as

Original: CLEAR does, is impractical for long task sequences due to prohibitive memory costs.
Revised: CLEAR does, is impractical for long task sequences due to prohibitive memory costs.

Original: All methodsuse SAC [15] as a base algorithm and have been equally well-tuned on our tasks.
Revised: All methods use SAC [15] as a base algorithm and have been equally well-tuned on our tasks.

Original: Full experimentaldetails can be found in Appendix B.
Revised: Full experimental details can be found in Appendix B.

Original: 4.3 Ablations
Revised: 4.3 Ablations

Original: We also perform a number of ablations to understand how the different components of CSP influenceperformance.
Revised: We also perform a number of ablations to understand how the different components of CSP influence performance.

Original: First, we want to understand how good the critic is at finding the best policy in thesubspace.
Revised: First, we want to understand how good the critic is at finding the best policy in the subspace.

Original: To do this, we implement CSP-MID which uses the subspace’s midpoint a proxy for thebest policy in the subspace for task j ∀ j ∈ 1 ...N rather than using the critic to select the best policy.
Revised: To do this, we implement CSP-MID which uses the subspace’s midpoint a proxy for the best policy in the subspace for task j ∀ j ∈ 1 ...N rather than using the critic to select the best policy.

Original: The midpoint is defined as θ mid ∶= 1 m ∑ mi = 1 θ i where m = ∣ Θ j ∣ .
Revised: The midpoint is defined as θ mid ∶= 1 m ∑ mi = 1 θ i where m = ∣ Θ j ∣ .

Original: The midpoint is a natural choice as ituniformly combines all prior anchors so it contains knowledge of all prior tasks.
Revised: The midpoint is a natural choice as it uniformly combines all prior anchors so it contains knowledge of all prior tasks.

Original: At the other end of the spectrum, we have CSP-ORACLE which selects the best policy by samplinga large number ( i.e. thousands) of policies in the subspace and computing Monte-Carlo estimates oftheir returns.
Revised: At the other end of the spectrum, we have CSP-ORACLE which selects the best policy by sampling a large number ( i.e. thousands) of policies in the subspace and computing Monte-Carlo estimates of their returns.

Original: These estimates are expected to be more accurate than the critic’s, so CSP-ORACLEcan be considered an upper bound to CSP.
Revised: These estimates are expected to be more accurate than the critic’s, so CSP-ORACLE can be considered an upper bound to CSP.

Original: However, CSP-ORACLE is less efficient than CSP sinceit requires singificantly more interactions with the environment ( i.e. millions).
Revised: However, CSP-ORACLE is less efficient than CSP since it requires singificantly more interactions with the environment ( i.e. millions).

Original: We also want to understand how much performance we lose, if any, by not adding one anchor foreach task.
Revised: We also want to understand how much performance we lose, if any, by not adding one anchor for each task.

Original: To do this, we run CSP-LINEAR which always extends the subspace.
Revised: To do this, we run CSP-LINEAR which always extends the subspace.

Original: In addition, wevary the threshold ϵ used to decide whether to extend the subspace based on how much performanceis gained by doing so.
Revised: In addition, we vary the threshold ϵ used to decide whether to extend the subspace based on how much performance is gained by doing so.

Original: This analysis can shed more light on the trade-off between performance gainand memory cost as the threshold varies.
Revised: This analysis can shed more light on the trade-off between performance gain and memory cost as the threshold varies.

Original: The paper includes a total of 4116 experiments, over 25 different CRL scenarios with 36 different RLenvironments.
Revised: The paper includes a total of 4116 experiments, over 25 different CRL scenarios with 36 different RL environments.

Original: Each experiment takes on average 15 hours on 1 GPU (NVIDIA V100).
Revised: Each experiment takes on average 15 hours on 1 GPU (NVIDIA V100).

Original: The meansand standard deviations are computed over 3 runs.
Revised: The means and standard deviations are computed over 3 runs.

Original: 4.4 Metrics
Revised: 4.4 Metrics

Original: Agents are evaluated across a range of commonly used CRL metrics [49, 32].
Revised: Agents are evaluated across a range of commonly used CRL metrics [49, 32].

Original: For brevity, followingthe notation in Section 2, we will use π ji ∶= Π ( i, [ t 1 ,...,t j ]) to denote the policy selected by Π fortask i obtained after training on the sequence of tasks t 1 ,...,t j (in this order with j included).
Revised: For brevity, following the notation in Section 2, we will use π ji ∶= Π ( i, [ t 1 ,...,t j ]) to denote the policy selected by Π for task i obtained after training on the sequence of tasks t 1 ,...,t j (in this order with j included).

Original: Notethat Π can also be defined for a single task i.e. π t i i ∶= Π
Revised: Note that Π can also be defined for a single task i.e. π t i i ∶= Π

Original: ( i, [ t i ]) is the policy for task i after trainingthe system only on task t
Revised: ( i, [ t i ]) is the policy for task i after training the system only on task t

Original: i (for the corresponding budget b i ).
Revised: i (for the corresponding budget b i ).

Original: Average Performance is the average performance of the model across all tasks, after training on theentire sequence of tasks.
Revised: Average Performance is the average performance of the model across all tasks, after training on the entire sequence of tasks.

Original: Let P
Revised: Let P

Original: i
Revised: i

Original: ( π ) be the performance of the system on task i using policy π definedas P i ( π ) ∶=
Revised: ( π ) be the performance of the system on task i using policy π defined as P i ( π ) ∶=

Original: E π, T i [∑ r i ( s,a )] where r i ( s,a ) is the reward received on task i when taking action a instate s .
Revised: E π, T i [∑ r i ( s,a )] where r i ( s,a ) is the reward received on task i when taking action a in state s .

Original: Then, the final average performance across all tasks can be computed as P ∶= 1 N ∑ Ni = 1 P i
Revised: Then, the final average performance across all tasks can be computed as P ∶= 1 N ∑ Ni = 1 P i

Original: ( π Ni ) .
Revised: ( π Ni ) .

Revised: [Table]

Original: Forward Transfer measures how much a CRL system is able to transfer knowledge from task totask.
Revised: Forward Transfer measures how much a CRL system is able to transfer knowledge from task to task.

Original: At task i , it compares the performance of the of the system trained on all previous tasks t 1 ,...,t ito the same model trained solely on task t
Revised: At task i , it compares the performance of the of the system trained on all previous tasks t 1 ,...,t i to the same model trained solely on task t

Original: i .
Revised: i .

Original: This measure is defined as FT i ∶= P
Revised: This measure is defined as FT i ∶= P

Original: i ( π ii )− P
Revised: i ( π ii )− P

Original: i
Revised: i

Original: π tii )
Revised: π tii )

Original: 1 − P
Revised: 1 − P

Original: i ( π tii ) , andthus the forward transfer for all tasks is FT ∶= 1 N ∑ Ni = 1 FT i .
Revised: i ( π tii ) , and thus the forward transfer for all tasks is FT ∶= 1 N ∑ Ni = 1 FT i .

Original: Forgetting evaluates how much a system has forgotten about task i after training on the full sequenceof tasks.
Revised: Forgetting evaluates how much a system has forgotten about task i after training on the full sequence of tasks.

Original: It thus compares the performance of policy π ii with the performance of policy π Ni and isdefined as F i ∶= P
Revised: It thus compares the performance of policy π ii with the performance of policy π Ni and is defined as F i ∶= P

Original: i ( π Ni ) − P
Revised: i ( π Ni ) − P

Original: i ( π ii ) .
Revised: i ( π ii ) .

Original: Similarly to the average transfer, we report the average forgettingacross all tasks F ∶= 1 N ∑ Ni = 1 F i .[Table]
Revised: Similarly to the average transfer, we report the average forgetting across all tasks F ∶= 1 N ∑ Ni = 1 F i .

Original: Growing Factor measures how much the size of the model grows during training.
Revised: Growing Factor measures how much the size of the model grows during training.

Original: Formally, it isdefined as the ratio between the number of parameters of the model at task t N and its size at task t 1 .
Revised: Formally, it is defined as the ratio between the number of parameters of the model at task t N and its size at task t 1 .

Original: It is defined as GF ∶= ∣ ψ N ∣ ∣ ψ 1 ∣ where ψ i is the model’s set of parameters after training on task i .
Revised: It is defined as GF ∶= ∣ ψ N ∣ ∣ ψ 1 ∣ where ψ i is the model’s set of parameters after training on task i .

Original:  Results 
Revised:  Results 5.1 Performance on Brax

Original: 5.1 Performance on Brax

Original: Table 1 shows the aggregated results across 4 of our CRL scenarios based on HalfCheetah, namelythe ones that probe forgetting, transfer, robustness, and compositionality.
Revised: Table 1 shows the aggregated results across 4 of our CRL scenarios based on HalfCheetah, namely the ones that probe forgetting, transfer, robustness, and compositionality.

Original: The other scenarios arestudied separately to avoid skewing the results, given that they use different regimes ( e.g. budget,number of tasks).
Revised: The other scenarios are studied separately to avoid skewing the results, given that they use different regimes ( e.g. budget, number of tasks).

Original: The results are normalized using the performance of the SAC-N agents.
Revised: The results are normalized using the performance of the SAC-N agents.

Original: See
Revised: See

Original: Appendix D for results on each scenario, including those based on Ant.
Revised: Appendix D for results on each scenario, including those based on Ant.

Original: CSP outperforms all the baselines along final average performance, forward transfer, and forgetting,while the memory costs are kept relatively low.
Revised: CSP outperforms all the baselines along final average performance, forward transfer, and forgetting, while the memory costs are kept relatively low.

Original: Naive methods like FT-1 have low memory costsand good transfer, but suffer from catastrophic forgetting.
Revised: Naive methods like FT-1 have low memory costs and good transfer, but suffer from catastrophic forgetting.

Original: In contrast, methods that aim to reduceforgetting such as FT-L2 or EWC do so at the expense of transfer.
Revised: In contrast, methods that aim to reduce forgetting such as FT-L2 or EWC do so at the expense of transfer.

Original: The only competitive methods interms of performance are the ones where the number of parameters increases at least linearly withthe number of tasks, such as PNN, SAC-N, and
Revised: The only competitive methods in terms of performance are the ones where the number of parameters increases at least linearly with the number of tasks, such as PNN, SAC-N, and

Original: FT-N.
Revised: FT-N.

Original: These methods have no forgetting becausethey store the models trained on each task.
Revised: These methods have no forgetting because they store the models trained on each task.

Original: SAC-N has no transfer since it trains each model fromscratch, while FT-N promotes transfer as it finetunes the previous model.
Revised: SAC-N has no transfer since it trains each model from scratch, while FT-N promotes transfer as it finetunes the previous model.

Original: However, these methodsare unfeasible for real-world scenarios with long task sequences due to their poor scalability.
Revised: However, these methods are unfeasible for real-world scenarios with long task sequences due to their poor scalability.

Original: Tosummarize, CSP outperforms by a wide margin methods with good scalability but poor stabilityor plasticity .
Revised: To summarize, CSP outperforms by a wide margin methods with good scalability but poor stability or plasticity .

Original: At the same time, CSP is significantly more memory efficient than methods withgood stability or plasticity but poor scalability .
Revised: At the same time, CSP is significantly more memory efficient than methods with good stability or plasticity but poor scalability .

Original: In conclusion, CSP strikes a better balance betweenstability, plasticity, and scalability than prior CRL approaches.
Revised: In conclusion, CSP strikes a better balance between stability, plasticity, and scalability than prior CRL approaches.

Original: Varying the Threshold.
Revised: Varying the Threshold.

Original: Figure 2a shows how performance and memory vary with the threshold ϵused to decide whether to extend the subspace.
Revised: Figure 2a shows how performance and memory vary with the threshold ϵ used to decide whether to extend the subspace.

Original: As expected, as ϵ increases, performance decreases,but so do memory costs.
Revised: As expected, as ϵ increases, performance decreases, but so do memory costs.

Original: Note that performance is still above 80% even as the memory costs are cutby more than 60% , relative to CSP-LINEAR which trains an anchor ( i.e. new set of parameters) foreach task. Thus, CSP can drastically reduce memory costs without significantly hurting performance. Practitioners can set the threshold to trade-off performance gains for memory efficiency ( i.e. thehigher the ϵ the more performance losses are tolerated in order to reduce memory costs). Scalability.
Revised: Note that performance is still above 80% even as the memory costs are cut

Original: Figure 2b shows how performance and memory scale with the number of tasks in thesequence (on the compositionality scenario), for both CSP and FT-N which is our strongest baseline.

Original: CSP maintains both strong performance and low memory cost even as the number of tasks increases.

Original: In contrast, FT-N’s growing factor scales linearly, which makes it impractical for long task sequences.

Original: Learning Efficiency.

Original: Figure 2c shows the average performance for three different budgets ( i.e.number of interactions allowed for each task) on the robustness scenario, comparing CSP with the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A v e r age P
Revised: A v e r age P

Original: e
Revised: e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m an c
Revised: o r m an c

Original: e 0.
Revised: e 0.

Original: 1.2 G r o w i ng
Revised: 1.2 G r o w i ng

Original: F a c t o r
Revised: F a c t o r

Original: [Figure]
Revised: [Figure]

Original: Growing Factor (
Revised: Growing Factor (

Original: [Figure]

Original: Varying

Original: [Figure]
Revised: [Figure]

Original: 12 Number of Tasks 0.
Revised: 12 Number of Tasks 0.

Original: A v e r age P
Revised: A v e r age P

Original: e
Revised: e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m an c e
Revised: o r m an c e

Original: CSPFT-N
Revised: CSPFT-N

Original: G r o w i
Revised: G r o w i

Original: ng F a c t
Revised: ng F a c t

Original: o r Average Performance Growing Factor (
Revised: o r Average Performance Growing Factor (

Original: [Figure]
Revised: [Figure]

Original: ) Scalability
Revised: ) Scalability

Original: FTFT-L
Revised: FTFT-L

Original: EWC
Revised: EWC

Original: CSP 0.
Revised: CSP 0.

Original: A v e r age
Revised: A v e r age

Original: P e
Revised: P e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m an c e b
Revised: o r m an c e b

Original: = 250k b = 500k b = 1M (c) Learning Efficiency
Revised: = 250k b = 500k b = 1M (c) Learning Efficiency

Original: [Figure]
Revised: [Figure]

Revised: by more than 60% , relative to CSP-LINEAR which trains an anchor ( i.e. new set of parameters) for each task.

Revised: Thus, CSP can drastically reduce memory costs without significantly hurting performance.

Revised: Practitioners can set the threshold to trade-off performance gains for memory efficiency ( i.e. the higher the ϵ the more performance losses are tolerated in order to reduce memory costs).

Revised: Scalability.

Revised: Figure 2b shows how performance and memory scale with the number of tasks in the sequence (on the compositionality scenario), for both CSP and FT-N which is our strongest baseline.

Original:  more scalable baselines FT-1, FT-L2, and EWC.
Revised: CSP maintains both strong performance and low memory cost even as the number of tasks increases. In contrast, FT-N’s growing factor scales linearly, which makes it impractical for long task sequences. Learning Efficiency. Figure 2c shows the average performance for three different budgets ( i.e. number of interactions allowed for each task) on the robustness scenario, comparing CSP with the more scalable baselines FT-1, FT-L2, and EWC.

Original: These results demonstrate that CSP can learnefficiently even with a reduced budget, outperforming the baselines.
Revised: These results demonstrate that CSP can learn efficiently even with a reduced budget, outperforming the baselines.

Original: By keeping track of an infinitenumber of policies, CSP enables positive transfer to a wide range of tasks without requiring manyinteractions to learn new skills.
Revised: By keeping track of an infinite number of policies, CSP enables positive transfer to a wide range of tasks without requiring many interactions to learn new skills.

Original: 5.2 Properties of the Subspace
Revised: 5.2 Properties of the Subspace

Original: Evaluating the Subspace.
Revised: Evaluating the Subspace.

Original: Table 1 shows that as expected, using the middle of the subspace as aproxy for the best policy is suboptimal ( i.e. CSP-MID is worse than CSP).
Revised: Table 1 shows that as expected, using the middle of the subspace as a proxy for the best policy is suboptimal ( i.e. CSP-MID is worse than CSP).

Original: Importantly, CSP’sperformance is not too far from that of CSP-ORACLE, indicating that the learned critic can beeffectively used to find the best policy in the subspace for a given task , despite the fact that it uses noadditional interactions with the environment to do so.
Revised: Importantly, CSP’s performance is not too far from that of CSP-ORACLE, indicating that the learned critic can be effectively used to find the best policy in the subspace for a given task , despite the fact that it uses no additional interactions with the environment to do so.

Original: In contrast, CSP-ORACLE requires millionsof samples to select the best policy in the subspace using Monte-Carlo rollouts.
Revised: In contrast, CSP-ORACLE requires millions of samples to select the best policy in the subspace using Monte-Carlo rollouts.

Original: Smoothness and Critic Accuracy.
Revised: Smoothness and Critic Accuracy.

Original: Figure 3a shows a snapshot of a trained subspace, along with theexpected reward of all policies in the subspace for a given task.
Revised: Figure 3a shows a snapshot of a trained subspace, along with the expected reward of all policies in the subspace for a given task.

Original: The expected reward is computedusing both Monte-Carlo rollouts, as well as our critic’s predictions using Equation 2.
Revised: The expected reward is computed [Table] using both Monte-Carlo rollouts, as well as our critic’s predictions using Equation 2.

Original: As illustrated,the subspace is smooth, and the critic’s estimates are highly accurate.
Revised: As illustrated, the subspace is smooth, and the critic’s estimates are highly accurate.

Original: Diversity and Compositionality.
Revised: Diversity and Compositionality.

Original: Figure 3b illustrates that the subspace contains behaviors composedof previously learned skills.
Revised: Figure 3b illustrates that the subspace contains behaviors composed of previously learned skills.

Original: This allows CSP to efficiently find good policies for many differentcombinations of prior tasks without the need of training additional parameters.
Revised: This allows CSP to efficiently find good policies for many different combinations of prior tasks without the need of training additional parameters.

Original: The figure alsoshows that for a given task, the policies in the subspace cover the whole spectrum of rewards,thus emphasizing the diversity of behaviors expressed by the subspace.
Revised: The figure also shows that for a given task, the policies in the subspace cover the whole spectrum of rewards, thus emphasizing the diversity of behaviors expressed by the subspace.

Original: See Appendix E for morequantitative and qualitative analyses of the subspace.[Table]
Revised: See Appendix E for more quantitative and qualitative analyses of the subspace.

Original: 5.3 Performance on Continual World
Revised: 5.3 Performance on Continual World

Original: [Table] Table 2 shows results for 4 challenging robotic manipulation scenarios from Continual World, eachof them containing 3 tasks.
Revised: Table 2 shows results for 4 challenging robotic manipulation scenarios from Continual World, each of them containing 3 tasks.

Original: These scenarios have been designed such that the second task reducestransfer from the first to the third task.
Revised: These scenarios have been designed such that the second task reduces transfer from the first to the third task.

Original: CSP achieves comparable performance with FT-N, whichis state-of-the-art on these scenarios [49].
Revised: CSP achieves comparable performance with FT-N, which is state-of-the-art on these scenarios [49].

Original: However, in contrast with CSP, FT-N cannot be usedfor long task sequences due to prohibitive memory and computational costs.
Revised: However, in contrast with CSP, FT-N cannot be used for long task sequences due to prohibitive memory and computational costs.

Original: See Appendix D foradditional results on Continual World.
Revised: See Appendix D for additional results on Continual World.

Original:  Related Work
Revised:  Related Work

Original: Continual Reinforcement Learning.
Revised: Continual Reinforcement Learning.

Original: CRL methods aim to avoid catastrophic forgetting, as well asenable transfer to new tasks, while remaining scalable to a large number of tasks.
Revised: CRL methods aim to avoid catastrophic forgetting, as well as enable transfer to new tasks, while remaining scalable to a large number of tasks.

Original: In the past few years,multiple approaches have been proposed to address one or more of these challenges [29, 21, 32, 49].
Revised: In the past few years, multiple approaches have been proposed to address one or more of these challenges [29, 21, 32, 49].

Original: One class of methods focuses on preventing catastrophic forgetting .
Revised: One class of methods focuses on preventing catastrophic forgetting .

Original: Some algorithms achieve this bystoring the parameters of models trained on prior tasks [38, 9, 48].
Revised: Some algorithms achieve this by storing the parameters of models trained on prior tasks [38, 9, 48].

Original: However, these methods scalepoorly with the number of tasks ( i.e. at least linearly) in both compute and memory, which makesthem unfeasible for more realistic scenarios.
Revised: However, these methods scale poorly with the number of tasks ( i.e. at least linearly) in both compute and memory, which makes them unfeasible for more realistic scenarios.

Original: Other methods maintain a buffer of experience fromprior tasks to alleviate forgetting [24, 18, 34, 36, 7].
Revised: Other methods maintain a buffer of experience from prior tasks to alleviate forgetting [24, 18, 34, 36, 7].

Original: However, this is also not scalable as the memorycost increases significantly with the number and complexity of the tasks.
Revised: However, this is also not scalable as the memory cost increases significantly with the number and complexity of the tasks.

Original: In addition, many real-worldapplications in domains like healthcare or insurance prevent data storage due to privacy or ethicalconcerns.
Revised: In addition, many real-world applications in domains like healthcare or insurance prevent data storage due to privacy or ethical concerns.

Original: Another class of methods focuses on improving transfer to new tasks.
Revised: Another class of methods focuses on improving transfer to new tasks.

Original: Naive approacheslike finetuning that train a single model on each new task provide good scalability and plasticity, butsuffer from catastrophic forgetting of previously learned tasks.
Revised: Naive approaches like finetuning that train a single model on each new task provide good scalability and plasticity, but suffer from catastrophic forgetting of previously learned tasks.

Original: To overcome this effect, methodslike elastic weight consolidation (EWC) [22] alleviate catastrophic forgetting by constraining howmuch the network’s weights change, but this can in turn reduce plasticity.
Revised: To overcome this effect, methods like elastic weight consolidation (EWC) [22] alleviate catastrophic forgetting by constraining how much the network’s weights change, but this can in turn reduce plasticity.

Original: Another class of methodsemploys knowledge distillation to improve transfer in CRL [17, 37, 23, 40, 5, 20, 46].
Revised: Another class of methods employs knowledge distillation to improve transfer in CRL [17, 37, 23, 40, 5, 20, 46].

Original: However, sincethese methods train a single network, they struggle to capture a large number of diverse behaviors.
Revised: However, since these methods train a single network, they struggle to capture a large number of diverse behaviors.

Original: There is also a large body of work which leverages the shared structure of the tasks [50, 30, 25, 27,1, 43], meta-learning [19, 44, 4, 8, 33, 52, 39], or generative models [35, 42, 41, 2] to improve CRLagents, but these are less related to CSP and not very competitive.
Revised: There is also a large body of work which leverages the shared structure of the tasks [50, 30, 25, 27, 1, 43], meta-learning [19, 44, 4, 8, 33, 52, 39], or generative models [35, 42, 41, 2] to improve CRL agents, but these are less related to CSP and not very competitive.

Original: Mode Connectivity and Neural Network Subspaces.
Revised: Mode Connectivity and Neural Network Subspaces.

Original: Based on prior studies of mode connectivity [13, 11, 28], [47, 6] proposed the neural network subspaces to connect the solutions of differentmodels.
Revised: Based on prior studies of mode connectivity [13, 11, 28], [47, 6] proposed the neural network subspaces to connect the solutions of different models.

Original: More similar to our work, [10] leverages subspace properties to mitigate forgetting on asequence of supervised learning tasks, but doesn’t focus on other aspects of the continual learningproblem.
Revised: More similar to our work, [10] leverages subspace properties to mitigate forgetting on a sequence of supervised learning tasks, but doesn’t focus on other aspects of the continual learning problem.

Original: Our work was also inspired by [14] which learns a subspace of policies for a single task forfast adaptation to new environment.
Revised: Our work was also inspired by [14] which learns a subspace of policies for a single task for fast adaptation to new environment.

Original: In contrast to [14], we consider the continual RL setting andadaptively grow the subspace as the agent encounters new tasks.
Revised: In contrast to [14], we consider the continual RL setting and adaptively grow the subspace as the agent encounters new tasks.

Original: Our work is first to demonstrate theeffectiveness of mode connectivity for the continual RL setting.
Revised: Our work is first to demonstrate the effectiveness of mode connectivity for the continual RL setting.

Original:  Discussion
Revised:  Discussion

Original: In this paper, we propose CSP, a new CRL method which adaptively builds a subspace of policies.
Revised: In this paper we propose CSP, a new CRL method which adaptively builds a subspace of policies.

Original: Learning about a large number of diverse behaviors allows CSP to strike a good balance betweenstability, plasticity, and scalability.
Revised: Learning about a large number of diverse behaviors allows CSP to strike a good balance between stability, plasticity, and scalability.

Original: We demonstrate our method’s superior capabilities relative tostate-of-the-art CRL approaches on a wide range of continuous control scenarios.
Revised: We demonstrate our method’s superior capabilities relative to state-of-the-art CRL approaches on a wide range of continuous control scenarios.

Original: Our paper is first touse a subspace of models for CRL, and thus opens up many interesting directions for future work.
Revised: Our paper is first to use a subspace of models for CRL, and thus opens up many interesting directions for future work.

Original: For example, one can assume a fixed size for the subspace and update the anchors whenever the agentencounters a new task.
Revised: For example, one can assume a fixed size for the subspace and update the anchors whenever the agent encounters a new task.

Original: Another promising direction is to leverage the structure of the subspace tometa-learn or search for good policies on a given task.
Revised: Another promising direction is to leverage the structure of the subspace to meta-learn or search for good policies on a given task.

Original: Finally, active learning techniques could beemployed to more efficiently evaluate the subspace.
Revised: Finally, active learning techniques could be employed to more efficiently evaluate the subspace.

Original: A Algorithm DetailsA.1 Scalability of CSP
Revised: A Algorithm Details A.1 Scalability of CSP

Original: By having access to an infinite number of policies, the subspace is highly expressive so it can capturea wide range of diverse behaviors.
Revised: By having access to an infinite number of policies, the subspace is highly expressive so it can capture a wide range of diverse behaviors.

Original: This enables positive transfer to many new tasks without the needfor training additional parameters.
Revised: This enables positive transfer to many new tasks without the need for training additional parameters.

Original: As a consequence, the number of parameters scales sublinearlywith the number of tasks.
Revised: As a consequence, the number of parameters scales sublinearly with the number of tasks.

Original: The speed of growth is controlled by the threshold ϵ which defines howmuch performance we are willing to give up for decreasing the number of parameters (by the size ofone policy network).
Revised: The speed of growth is controlled by the threshold ϵ which defines how much performance we are willing to give up for decreasing the number of parameters (by the size of one policy network).

Original: Practitioners can set the threshold to trade-off performance gains for memoryefficiency ( i.e. the higher the ϵ the more performance losses are tolerated to reduce memory costs).
Revised: Practitioners can set the threshold to trade-off performance gains for memory efficiency ( i.e. the higher the ϵ the more performance losses are tolerated to reduce memory costs).

Original: Inpractice, we noticed that setting ϵ = 0 .
Revised: In practice, we noticed that setting ϵ = 0 .

Original: 1 allows good performance and a limited growth of parameters.
Revised: 1 allows good performance and a limited growth of parameters.

Original: As the agent learns to solve more and more tasks, we expect the subspace to grow more slowly (orstop growing entirely) since it already contains many useful behaviors which transfer to new tasks.
Revised: As the agent learns to solve more and more tasks, we expect the subspace to grow more slowly (or stop growing entirely) since it already contains many useful behaviors which transfer to new tasks.

Original: On the other hand, if the agent encounters a task that is significantly different than previous onesand all other behaviors in the subspace, the subspace still has the flexibility to grow and incorporateentirely new skills.
Revised: On the other hand, if the agent encounters a task that is significantly different than previous ones and all other behaviors in the subspace, the subspace still has the flexibility to grow and incorporate entirely new skills.

Original: The number of anchors in the final subspace is adaptive and depends on thesequence of tasks.
Revised: The number of anchors in the final subspace is adaptive and depends on the sequence of tasks.

Original: The longer and more diverse the task sequence, the more anchors are needed.
Revised: The longer and more diverse the task sequence, the more anchors are needed.

Original: Thisproperty is important for real-world applications with open-ended interactions where it’s unlikely toknow a priori how much capacity is required to express all useful skills.
Revised: This property is important for real-world applications with open-ended interactions where it’s unlikely to know a priori how much capacity is required to express all useful skills.

Original: A.2 Using the critic as a discriminator
Revised: A.2 Using the critic as a discriminator

Original: The subspace critic W ϕ plays a central role in our method.
Revised: The subspace critic W ϕ plays a central role in our method.

Original: Compared to the vanilla SAC critic, weonly add the convex combination α as an input (concatenated with the states and actions).
Revised: Compared to the vanilla SAC critic, we only add the convex combination α as an input (concatenated with the states and actions).

Original: In thisway, it is optimized not only to evaluate the future averaged return on ( s,a ) pairs of a single policy,but an infinity of policies, characterized by the convex combination α .
Revised: In this way, it is optimized not only to evaluate the future averaged return on ( s,a ) pairs of a single policy, but an infinity of policies, characterized by the convex combination α .

Original: At the end of each task, the subspace critic has the difficult task to estimate by how much the newanchor policy θ j + 1 improves the performance of the current subspace Θ j .
Revised: At the end of each task, the subspace critic has the difficult task to estimate by how much the new anchor policy θ j + 1 improves the performance of the current subspace Θ j .

Original: To do so, one has to findthe best combination of policies in the last subspace Θ j , calling it α old and the one in the currentsubspace Θ j + 1 , calling it α new .
Revised: To do so, one has to find the best combination of policies in the last subspace Θ j , calling it α old and the one in the current subspace Θ j + 1 , calling it α new .

Original: In practice, we found that sampling 1024 random ( s,a ) pairs fromthe replay buffer at the end of the task allows to have an accurate estimation of the best policies.
Revised: In practice, we found that sampling 1024 random ( s,a ) pairs from the replay buffer at the end of the task allows to have an accurate estimation of the best policies.

Original: A.3 Sampling combinations in the subspace
Revised: A.3 Sampling combinations in the subspace

Original: Yet, it is important to allow the critic to estimate α old .
Revised: Yet, it is important to allow the critic to estimate α old .

Original: During training, we noticed that sampling witha simple flat Dirichlet distribution (i.e. a uniform distribution over the simplex induced by the currentsubspace) is not enough to make the critic able to accurately estimate the performance of the lastsubspace (indeed, the chances of sampling a policy in the last subspace are almost surely 0.).
Revised: During training, we noticed that sampling with a simple flat Dirichlet distribution (i.e. a uniform distribution over the simplex induced by the current subspace) is not enough to make the critic able to accurately estimate the performance of the last subspace (indeed, the chances of sampling a policy in the last subspace are almost surely 0.).

Original: This iswhy we decided to sample in both the current and the last subspace.
Revised: This is why we decided to sample in both the current and the last subspace.

Original: The distribution we use is then amixture of two Dirichlet (equal chances of sampling in the last subspace and in the current subspace).
Revised: The distribution we use is then a mixture of two Dirichlet (equal chances of sampling in the last subspace and in the current subspace).

Original: We did not perform an ablation to see if balancing the mixture would increase performances.
Revised: We did not perform an ablation to see if balancing the mixture would increase performances.

Original: We also tried to sample with a peaked distribution (concentration of the Dirichlet equal to the inverseof the number of the anchors) to see if it increased performances.
Revised: We also tried to sample with a peaked distribution (concentration of the Dirichlet equal to the inverse of the number of the anchors) to see if it increased performances.

Original: In some cases the new subspaceis able to find good policies faster with this distribution.
Revised: In some cases the new subspace is able to find good policies faster with this distribution.

Original: It can be a good trade off between alwayschoosing the last anchor and sampling uniformly.
Revised: It can be a good trade off between always choosing the last anchor and sampling uniformly.

Original: B Experimental Details
Revised: B Experimental Details

Original: All the experiments were implemented with SaLinA [26], a flexible and simple python library forlearning sequential agents.
Revised: All the experiments were implemented with SaLinA [26], a flexible and simple python library for learning sequential agents.

Original: We used Soft Actor Critic [15] as the routine algorithm for each method.
Revised: We used Soft Actor Critic [15] as the routine algorithm for each method.

Original: It is a popular and efficient off-policy algorithm for continuous domains.
Revised: It is a popular and efficient off-policy algorithm for continuous domains.

Original: We use a version wherethe entropy parameter is learned, as proposed in [16].
Revised: We use a version where the entropy parameter is learned, as proposed in [16].

Original: For each task and each methods, the twincritic networks are reset as well as the replay buffer (which has a maximum size of 1M interactions,corresponding to the maximum budget of each task).
Revised: For each task and each methods, the twin critic networks are reset as well as the replay buffer (which has a maximum size of 1M interactions, corresponding to the maximum budget of each task).

Original: B.1 Architecture DetailsBrax scenarios:
Revised: B.1 Architecture Details

Revised: Brax scenarios:

Original: For the twin critics and policy, We use a network with 4 linear layers, each consisting of 256 neurons.
Revised: For the twin critics and policy, We use a network with 4 linear layers, each consisting of 256 neurons.

Original: We use leaky ReLU activations (with α = 0 . 2 ) after every layer.
Revised: We use leaky ReLU activations (with α = 0 . 2 ) after every layer.

Original: Continual World triplets:
Revised: Continual World triplets:

Original: We use the same architectures as for Brax scenarios, but following [49], we add Layer Normalization[3] after the first layer, followed by a tanh activation.
Revised: We use the same architectures as for Brax scenarios, but following [49], we add Layer Normalization [3] after the first layer, followed by a tanh activation.

Original: For the subspace method, we did not add thislayer.
Revised: For the subspace method, we did not add this layer.

Original: Specific architecture of CSP:
Revised: Specific architecture of CSP

Original: Our Pytorch implementation of CSP uses the nn.
Revised: Our Pytorch implementation of CSP uses the nn.

Original: ModuleList object to store anchor networks.
Revised: ModuleList object to store anchor networks.

Original: Theadditional computational cost compared to a single network is negligible during both training andinference as it is mentioned in [47].
Revised: The additional computational cost compared to a single network is negligible during both training and inference as it is mentioned in [47].

Original: B.2 Experimental ProtocolBrax scenarios:
Revised: B.2 Experimental Protocol Brax scenarios:

Original: Considering FT-N as the upper bound with which we want to compare, we decided to apply thefollowing protocol for each short scenario.
Revised: Considering FT-N as the upper bound with which we want to compare, we decided to apply the following protocol for each short scenario.

Original: First, we run a gridsearch on SAC hyper-parameters(see Table 3) on FT-N and select the best set in terms of final average performance (see Section 4for the details about this metric).
Revised: First, we run a gridsearch on SAC hyper-parameters (see Table 3) on FT-N and select the best set in terms of final average performance (see Sectionfor the details about this metric).

Original: Then, we freeze these hyper-parameters and performed a specificgridsearch for CSP and each baseline (see Table 4).
Revised: Then, we freeze these hyper-parameters and performed a specific gridsearch for CSP and each baseline (see Table 4).

Original: Each hyper-parameter set is evaluated over 3seeds.
Revised: Each hyper-parameter set is evaluated overseeds.

Original: We believe this ensures fair comparisons, and even gives a slight advantage to FT-N comparedto our method.
Revised: We believe this ensures fair comparisons, and even gives a slight advantage to FT-N compared to our method.

Original: Note that we set the number of parallel environments to 128 and the number of updateper step to 0 .
Revised: Note that we set the number of parallel environments to 128 and the number of update per step to 0 .

Original: They can be seen as tasks constraints.
Revised: They can be seen as tasks constraints.

Original: These values are reasonable, and FineTuningthem would have increased the number of experiments by a lot.
Revised: These values are reasonable, and FineTuning them would have increased the number of experiments by a lot.

Original: Continual World triplets:
Revised: Continual World triplets:

Original: We bypassed the preliminary SAC gridsearch and use the set of hyper-parameters proposed in [49].
Revised: We bypassed the preliminary SAC gridsearch and use the set of hyper-parameters proposed in [49].

Original: We performed a specific gridsearch for CSP and each baseline (see Table 4).
Revised: We performed a specific gridsearch for CSP and each baseline (see Table 4).

Original: B.3 BaselinesEWC:
Revised: B.3 Baselines EWC:

Original: This regularization based method aims to protect parameters that are important for the previous tasks.
Revised: This regularization based method aims to protect parameters that are important for the previous tasks.

Original: After each task, it uses the Fisher information matrix to approximate the importance of each weight.
Revised: After each task, it uses the Fisher information matrix to approximate the importance of each weight.

Original: We followed [49] to compute the Fisher matrix and use an analytical derivation of it.
Revised: We followed [49] to compute the Fisher matrix and use an analytical derivation of it.

Original: FT-L2:
Revised: FT-L2:

Original: This baseline is proposed by [49].
Revised: This baseline is proposed by [49].

Original: It can be seen as a simplified version of EWC where the regularization coefficients for each parameters are equal to 1.
Revised: It can be seen as a simplified version of EWC where the regularization coefficients for each parameters are equal to 1.

Original: PNN:
Revised: PNN:

Original: This method proposed by [38] creates a new network at the beginning of each task, as well as lateralnetworks that will take as inputs - for each hidden layer - the output of the networks trained onformer tasks.
Revised: This method proposed by [38] creates a new network at the beginning of each task, as well as lateral networks that will take as inputs - for each hidden layer - the output of the networks trained on former tasks.

Original: In this method, the number of parameters, training and inference times are growingexponentially with respect to the number of tasks.
Revised: In this method, the number of parameters, training and inference times are growing exponentially with respect to the number of tasks.

Original: We used a simple linear layer for each lateral connection.
Revised: We used a simple linear layer for each lateral connection.

Original: B.4 Ablations
Revised: B.4 Ablations

Original: Ablations on the threshold parameter (see Figure 2) of CSP have been performed on the longversion of the Distraction scenario of HalfCheetah.
Revised: Ablations on the threshold parameter (see Figure 2) of CSP have been performed on the long version of the Distraction scenario of HalfCheetah.

Original: Concerning the scalability ablation, we used the
Revised: Concerning the scalability ablation, we used the

Original: Compositional scenario of HalfCheetah
Revised: Compositional scenario of HalfCheetah

Original: and duplicated it in 3 versions (4 tasks, 8 tasks, 12 tasks).
Revised: and duplicated it in 3 versions (4 tasks, 8 tasks, 12 tasks).

Original: Concerning the learning efficiency, we used the Distraction Scenario (short version) and ran it with abudget of 250k,500k and 1M interactions.
Revised: Concerning the learning efficiency, we used the Distraction Scenario (short version) and ran it with a budget of 250k,500k and 1M interactions.

Original: Results were averaged over 3 seeds.
Revised: Results were averaged over 3 seeds.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C Environment Details 
Revised: C Environment Details C.1 Designing the Tasks

Original: C.1 Designing the Tasks

Original: We used the flexibility of Brax physics engine [12] [Table] and two of its continuous control environments
Revised: We used the flexibility of Brax physics engine [12], and two of its continuous control environments

Original: Halfcheetah (obs dim: 23, action dim: 6) and Ant (obs dim: 27, action dim: 8) to derive multipletasks from them.
Revised: Halfcheetah (obs dim: 23, action dim: 6) and Ant (obs dim: 27, action dim: 8) to derive multiple tasks from them.

Original: To do so, we tweaked multiple environment parameters (Table 5)) and tried tolearn a policy on these new tasks.
Revised: To do so, we tweaked multiple environment parameters (Table 5)) and tried to learn a policy on these new tasks.

Original: From that pool of trials, we kept tasks (see Table 6) that were bothchallenging and diversified. 
Revised: From that pool of trials, we kept tasks (see Table 6) that were both challenging and diversified. [Table]

Original: [Table]
Revised: [Table]

Revised: h ee

Original: [Table]
Revised: [Table]

Revised: a h

Original: [Table]
Revised: [Table]

Original: C.2 Designing the Scenarios
Revised: C.2 Designing the Scenarios

Original: Inspired by [49], we studied the relationship between these changes with a simple protocol: we learna new task with a policy that has been pre-trained on a former task.
Revised: Inspired by [49], we studied the relationship between these changes with a simple protocol: we learn a new task with a policy that has been pre-trained on a former task.

Original: We drew forgetting and transfertables for each pair of tasks (see Figure 4).
Revised: We drew forgetting and transfer tables for each pair of tasks (see Figure 4).

Original: With this information, we designed 2 types of scenariosrepresenting a particular challenge in continual learning: the Forgetting Scenarios are designedsuch that a single policy tends to forget the former task when learning a new one.
Revised: With this information, we designed 2 types of scenarios representing a particular challenge in continual learning: the Forgetting Scenarios are designed such that a single policy tends to forget the former task when learning a new one.

Original: The Transfer
Revised: The Transfer

Original: Scenarios are designed such that a single policy has more difficulties to learn a new task after havinglearned the former one, rather than learning it from scratch.
Revised: Scenarios are designed such that a single policy has more difficulties to learn a new task after having learned the former one, rather than learning it from scratch.

Original: In addition we designed Distraction
Revised: In addition we designed Distraction

Original: Scenarios , that alternate between a normal task and a very different distraction task that disturbsthe whole learning process of a single policy (we simply inverted the actions).
Revised: Scenarios , that alternate between a normal task and a very different distraction task that disturbs the whole learning process of a single policy (we simply inverted the actions).

Original: While this challengelooks particularly simple from a human perspective (a simple -1 vector applied on the output is fineto find an optimal policy in a continual setting), we figured out that the Fine-tuning policies struggleto recover good performances (the final average reward actually decreases).
Revised: While this challenge looks particularly simple from a human perspective (a simple -1 vector applied on the output is fine to find an optimal policy in a continual setting), we figured out that the Fine-tuning policies struggle to recover good performances (the final average reward actually decreases).

Original: Finally, we created
Revised: Finally, we created

Original: Compositional Scenarios , that present two first tasks that will be useful to learn the last one, but avery different distraction task is put at the third place to disturb this forward transfer.
Revised: Compositional Scenarios , that present two first tasks that will be useful to learn the last one, but a very different distraction task is put at the third place to disturb this forward transfer.

Original: In order to givea complete overview of the performances of our model and baselines, we designed a short version (4tasks) and a long version (8 tasks, i.e. the short version repeated twice) of all the scenarios.
Revised: In order to give a complete overview of the performances of our model and baselines, we designed a short version ( tasks) and a long version (8 tasks, i.e. the short version repeated twice) of all the scenarios.

Original: Here isthe detailed sequence for each scenario (short version):
Revised: Here is the detailed sequence for each scenario (short version):

Original: Forgetting Scenarios:∎ HalfCheetah:hugefeet → moon → carrystuff → rainfall ∎ Ant:normal → hugefeet → rainfall → moon 2.
Revised: Forgetting Scenarios: ∎ HalfCheetah: hugefeet → moon → carrystuff → rainfall ∎ Ant: normal → hugefeet → rainfall → moon 2.

Original: Transfer Scenarios:∎ HalfCheetah:
Revised: Transfer Scenarios: ∎ HalfCheetah:

Original: carrystuff_hugegravity → moon → defectivesensor → hugefeet_rainfall ∎ Ant:disableddiagonalfeet_1 → disableddiagonalfeet_2 → disabledforefeet →disabledhindfeet
Revised: carrystuff_hugegravity → moon → defectivesensor → hugefeet_rainfall ∎ Ant: disableddiagonalfeet_1 → disableddiagonalfeet_2 → disabledforefeet → disabledhindfeet 3.

Original: Distraction Scenarios: ∎ HalfCheetah:normal → inverted_actions → normal → inverted_actions ∎Ant:
Revised: Distraction Scenarios: ∎ HalfCheetah: normal → inverted_actions → normal → inverted_actions ∎

Original:  normal → inverted_actions → normal → inverted_actions 
Revised: Ant: normal → inverted_actions → normal → inverted_actions 4.

Original: Compositional Scenarios:∎ HalfCheetah:tinyfeet → moon → carrystuff_hugegravity → tinyfeet_moon
Revised: Compositional Scenarios: ∎ HalfCheetah: tinyfeet → moon → carrystuff

Original: ∎ Ant:disabled3feet _1 → disabled3feet_2 → disabledforefeet → disabledhindfeet
Revised: [Table]

Original: C.3 Continual World Tasks
Revised: C.3 Continual World Tasks

Original: We also tested our model and our baselines on the 8 triplets (3-tasks scenarios) proposed by [49].
Revised: We also tested our model and our baselines on the 8 triplets (3-tasks scenarios) proposed by [49].

Original: The tasks originally come from Metaworld [51], and have a budget of 1M interactions each.
Revised: The tasks originally come from Metaworld [51], and have a budget of 1M interactions each.

Original: Thescenarios were specially designed such that there is a positive forward transfer from task 1 to task3, but task 2 is used as a distraction that interferes with these learning dynamics.
Revised: The scenarios were specially designed such that there is a positive forward transfer from task 1 to task 3, but task 2 is used as a distraction that interferes with these learning dynamics.

Original: These tasks aredesigned with the 2.0 version of the open-source MuJoCo physics engine [45].
Revised: These tasks are designed with the 2.0 version of the open-source MuJoCo physics engine [45].

Original: Here is the detailedsequence of the triplets:
Revised: Here is the detailed sequence of the triplets:

Original: push-v1 → window-close-v1 → hammer-v1 2.
Revised: push-v1 → window-close-v1 → hammer-v 2.

Original: hammer-v1 → window-close-v1 → faucet-close-v 
Revised: hammer-v1 → window-close-v1 → faucet-close-v 3.

Original: [Table]

Original: window-close-v1 → handle-press-side-v1 → peg-unplug-side-v 
Revised: window-close-v1 → handle-press-side-v1 → peg-unplug-side-v 4.

Original: [Table]

Original: faucet-close-v1 → shelf-place-v1 → peg-unplug-side-v 
Revised: faucet-close-v1 → shelf-place-v1 → peg-unplug-side-v 5.

Original: [Table]

Original: faucet-close-v1 → shelf-place-v1 → push-back-v 
Revised: faucet-close-v1 → shelf-place-v1 → push-back-v 6.

Original: [Table]

Original: stick-pull-v1 → peg-unplug-side-v1 → stick-pull-v 
Revised: stick-pull-v1 → peg-unplug-side-v1 → stick-pull-v 7.

Original: [Table]

Original: stick-pull-v1 → push-back-v1 → push-wall-v 
Revised: stick-pull-v1 → push-back-v1 → push-wall-v 8.

Original: [Table]

Original: push-wall-v1 → shelf-place-v1 → push-back-v
Revised: push-wall-v1 → shelf-place-v1 → push-back-v

Original: [Table]

Original: c a rr y s t u ff c a rr y s t u ff _ hu g e g r a v
Revised: c a rr y s t u ff c a rr y s t u ff _ hu g e g r a v

Original: i t y d e f e c t
Revised: i t y d e f e c t

Original: i v e s e n s o r s hu g e f ee t hu
Revised: i v e s e n s o r s hu g e f ee t hu

Original: g e f ee t _ r a i n f a
Revised: g e f ee t _ r a i n f a

Original: ll m oo n n o r m a l r a i n f a ll t i n y f ee t t i n y f ee t _ m oo n carrystuff carrystuff
Revised: ll m oo n n o r m a l r a i n f a ll t i n y f ee t t i n y f ee t _ m oo n carrystuff carrystuff

Original: [Table]
Revised: [Table]

Original: defectivesensors hugefeet
Revised: defectivesensors hugefeet

Original: [Figure]
Revised: [Figure]

Original: moon normal rainfall tinyfeet
Revised: moon normal rainfall tinyfeet

Original: [Figure]
Revised: [Figure]

Original: moon
Revised: moon

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: defectivesensors hugefeet hugefeet_rainfall moon normal rainfall tinyfeet tinyfeet_moon
Revised: defectivesensors hugefeet hugefeet_rainfall moon normal rainfall tinyfeet tinyfeet_moon

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +0. 
Revised: +0. +0. +0. +1.

Original: +0. 
Revised: +0. +0. +0. +0.

Revised: +0.

Original: +0. 
Revised: +0. +0. +0. +0.

Original: +1. 
Revised: +1. +0. +0. +0.

Original: +0. +0. 
Revised: +0. +0. +0.

Original: +0.
Revised: +0.

Original: +0.+0. +0. +0.
Revised: +0.

Original: +0. +0.
Revised: +0. +0.

Original: +1.
Revised: +1.

Original: +0. +0.
Revised: +0. +0.

Original: +0.
Revised: +0.

Original: +0. +0. +0.
Revised: +1.

Original: +0. +0.
Revised: +0. +0.

Original: +0. +0.
Revised: +0. +0.

Original: +1.
Revised: +1.

Original: +0.
Revised: +0.

Original: +0. +0.
Revised: +0. +0.

Original: +1. +0.
Revised: +1. +0.

Original: +0.+0. +0. +1.
Revised: +0.

Original: +0. +0.+0.
Revised: +0. +0.

Original: +1.
Revised: +1.

Original: +0. +0.
Revised: +0. +0.

Original: +0. +0.
Revised: +0. +0.

Original: +1.

Original: +0. +0.+0.
Revised: +0. +0.

Original: +0.+0. +0. +0.
Revised: +0.

Original: +0. +0.
Revised: +0. +0.

Original: [Figure]
Revised: [Figure]

Original: D Detailed ResultsD.1 HalfCheetah
Revised: D Detailed Results D.1 HalfCheetah

Original: For each of the 4 scenarios, we ran two versions: A short one (4 tasks) and a long one (8 tasks, theshort one repeated twice).
Revised: For each of the 4 scenarios, we ran two versions: A short one (4 tasks) and a long one (8 tasks, the short one repeated twice).

Original: The normalized results are respectively presented in Table 9 and Table 10.
Revised: The normalized results are respectively presented in Table 9 and Table 10.

Original: For a complete transparency, we also provide in Table 7 the raw rewards that were used to normalizethe results.
Revised: For a complete transparency, we also provide in Table 7 the raw rewards that were used to normalize the results.

Original: See Section B for more details about the computation of the normalized results.
Revised: See Section B for more details about the computation of the normalized results.

Original: The besthyper-parameter sets for each scenario are provided in Table[Table] [Table] [Table]
Revised: The best hyper-parameter sets for each scenario are provided in Table

Original: L

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D.2 Ant
Revised: D.2 Ant

Original: We only ran the short version of the 4 scenarios.
Revised: We only ran the short version of the 4 scenarios.

Original: The long version is currently running and willbe displayed in a future version of this paper.
Revised: The long version is currently running and will be displayed in a future version of this paper.

Original: The normalized results are respectively presented in
Revised: The normalized results are respectively presented in

Original: Table 13.
Revised: Table 13.

Original: For a complete transparency, we also provide in Table 11 the raw rewards that were used tonormalize the results.
Revised: For a complete transparency, we also provide in Table 11 the raw rewards that were used to normalize the results.

Original: See Section B for more details about the computation of the normalized results.
Revised: See Section B for more details about the computation of the normalized results.

Original: The best hyper-parameter sets for each scenario are provided in Table[-
 [Table] [Table] [Table]-]
Revised: The best hyper-parameter sets for each scenario are provided in Table[-
 [Table] [Table] [Table]-]

Original: [Table]
Revised: [Table]

Original: D.3 Continual World
Revised: D.3 Continual World

Original: We provide the final average performance of the methods over the 8 triplets proposed in [49].
Revised: We provide the final average performance of the methods over the 8 triplets proposed in [49].

Original: See
Revised: See

Original: Table 
Revised: Table 14.

Original: [Table]
Revised: [Table]

Original: E Analysis of the Subspace
Revised: E Analysis of the Subspace

Original: The he best way to visualize the reward and critic value landscapes of the subspaces is when there are3 anchors (see Figure 3).
Revised: The he best way to visualize the reward and critic value landscapes of the subspaces is when there are 3 anchors (see Figure 3).

Original: To do saw, we draw 8192 evenly spaced points in the 3-dimensional simplexof R 3 , and average the return over 10 rollouts for the reward landscape, and 1024 pairs of ( s,a ) forthe critic landscape.
Revised: To do saw, we draw 8192 evenly spaced points in the 3-dimensional simplex of R 3 , and average the return over 10 rollouts for the reward landscape, and 1024 pairs of ( s,a ) for the critic landscape.

Original: We used the short version of the Compositional Scenario of HalfCheetah todisplay the results. 
Revised: We used the short version of the Compositional Scenario of HalfCheetah to display the results. [Table] [Table]

Original: F Limitations of CSP
Revised: F Limitations of CSP

Original: CSP prevents forgetting of prior tasks, promotes transfer to new tasks, and scales sublinearly with thenumber of tasks.
Revised: CSP prevents forgetting of prior tasks, promotes transfer to new tasks [Table] and scales sublinearly with the number of tasks.

Original: Despite all these advantages, our method still has a number of limitations.
Revised: Despite all these advantages, our method still has a number of limitations.

Original: Whilethe subspace grows only sublinearly with the number of tasks, this number is highly dependent onthe task sequence.
Revised: While the subspace grows only sublinearly with the number of tasks, this number is highly dependent on the task sequence.

Original: In the worst case scenario, it increases linearly with the number of tasks.
Revised: In the worst case scenario, it increases linearly with the number of tasks.

Original: On theother hand, the more similar the tasks are, the lower the size of the subspace needed to learn goodpolicies for all tasks.
Revised: On the other hand, the more similar the tasks are, the lower the size of the subspace needed to learn good policies for all tasks.

Original: Thus, one important direction for future work is to learn a subspace of policieswith a fixed number of anchors.
Revised: Thus, one important direction for future work is to learn a subspace of policies with a fixed number of anchors.

Original: Instead of training an additional anchor for each new task, one couldoptimize a policy in the current subspace on the new task while ensuring that the best policies forprior tasks don’t change too much.
Revised: Instead of training an additional anchor for each new task, one could optimize a policy in the current subspace on the new task while ensuring that the best policies for prior tasks don’t change too much.

Original: This could be formulated as a constrained optimization problemwhere all the anchors defining the subspace are updated for each new task, but some regions of thesubspace are regularized to not change very much.
Revised: This could be formulated as a constrained optimization problem where all the anchors defining the subspace are updated for each new task, but some regions of the [Table] subspace are regularized to not change very much.

Original: This would result in the subspace having differentregions which are good for different tasks.
Revised: This would result in the subspace having different regions which are good for different tasks.

Original: While the memory costs increase sublinearly with the number of tasks, the computational costsincrease linearly with the number of tasks, in the current implementation of CSP.
Revised: While the memory costs increase sublinearly with the number of tasks [Table] the computational costs increase linearly with the number of tasks, in the current implementation of CSP.

Original: This is because wetrain an additional anchor for each new task, which can be removed if it doesn’t significantly improveperformance.
Revised: This is because we train an additional anchor for each new task, which can be removed if it doesn’t significantly improve [Table] performance.

Original: However, the computational costs can also be reduced if you have access to maximumreward on a task.
Revised: However, the computational costs can also be reduced if you have access to maximum reward on a task.

Original: This is typically the case for sparse reward tasks where if the agent succeeds, itreceives a reward of 1 and 0 otherwise.
Revised: This is typically the case for sparse reward tasks where if the agent succeeds, it receives a reward of 1 and 0 otherwise.

Original: In this case, there is no need to train one anchor per task.
Revised: In this case, there is no need to train one anchor per task.

Original: Instead, one can simply find the best policy in the current subspace and compare its performance withthe maximum reward to decide whether to train an additional anchor or not.
Revised: Instead, one can simply find the best policy in the current subspace and compare its performance with the maximum reward to decide whether to train an additional anchor or not.

Original: Hence, in this versionof CSP (which is a minor modification of the current implementation) both memory and computescale sublinearly with the number of tasks.
Revised: Hence, in this version of CSP (which is a minor modification of the current implementation) both memory and compute scale sublinearly with the number of tasks.

Original: However, this assumption doesn’t always hold, so herewe decided to implement the more general version of CSP.
Revised: However, this assumption doesn’t always hold, so here we decided to implement the more general version of CSP.

Original: In this work, we don’t specifically leverage the structure of the subspace in order to find good policies.
Revised: In this work, we don’t specifically leverage the structure of the subspace in order to find good policies.

Original: Hence, one promising research direction is to further improve transfer efficiency by leveraging thestructure of the subspace to find good policies for new tasks.
Revised: Hence, one promising research direction is to further improve transfer efficiency by leveraging the structure of the subspace to find good policies for new tasks.

Original: For example, this could be done byfinding the convex combination of the anchors which maximizes return on a given task.
Revised: For example, this could be done by finding the convex combination of the anchors which maximizes return on a given task.

Original: Regularizingthe geometry of the subspace to impose certain inductive biases could also be a fruitful direction forfuture work.
Revised: Regularizing the geometry of the subspace to impose certain inductive biases could also be a fruitful direction for future work.

Original: G Broader Impact Statement
Revised: G Broader Impact Statement

Original: This work proposes a new method for the continual reinforcement learning setting, which is avery broad framework.
Revised: This work proposes a new method for the continual reinforcement learning setting, which is a very broad framework.

Original: Many real-world applications can be framed as continual RL problems.
Revised: Many real-world applications can be framed as continual RL problems.

Original: For example, household robots need to constantly learn new skills and adapt to their changingenvironments.
Revised: For example, household robots need to constantly learn new skills and adapt to their changing environments.

Original: Similarly, online recommendation systems need to adapt to each user and theirchanging preferences.
Revised: Similarly, online recommendation systems need to adapt to each user and their changing preferences.

Original: Like other continual RL algorithms, our method aims to learn a policy whichmaximizes reward on a sequence of tasks (each of them with a corresponding reward function).
Revised: Like other continual RL algorithms, our method aims to learn a policy which maximizes reward on a sequence of tasks (each of them with a corresponding reward function).

Original: Thetasks or reward functions are typically specified by the designer.
Revised: The tasks or reward functions are typically specified by the designer.

Original: Depending on the goals of thedesigner, executing the resulting policy could result in positive or negative consequences.
Revised: Depending on the goals of the designer, executing the resulting policy could result in positive or negative consequences.

Original: H Compute Resources
Revised: H Compute Resources

Original: Each algorithm was trained using two Intel(R)
Revised: Each algorithm was trained using two Intel(R)

Original: Xeon(R) CPU cores (E5-2698 v4 @ 2.20GHz)and one NVIDIA GP100 GPU.
Revised: Xeon(R) CPU cores (E5-2698 v4 @ 2.20GHz) and one NVIDIA GP100 GPU.

Original: We used PyTorch [31] for all our experiments.
Revised: We used PyTorch [31] for all our experiments.

Original: Each run tookbetween approximately 8 and 30 hours to complete.
Revised: Each run took between approximately 8 and 30 hours to complete.

Original: The total runtime depended on three factors: thecomputation time of the algorithm, the length of the scenario, and the behavior of the policy.
Revised: The total runtime depended on three factors: the computation time of the algorithm, the length of the scenario, and the behavior of the policy.