Original: Disentangling Transfer in Continual Reinforcement Learning
Revised: Disentangling Transfer in Continual Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The ability of continual learning systems to transfer knowledge from previouslyseen tasks in order to maximize forward transfer is a signiﬁcant challenge for theﬁeld, limiting the applicability of continual learning solutions to realistic scenarios.
Revised: The ability of continual learning systems to transfer knowledge from previouslyseen tasks in order to maximize forward transfer is a signiﬁcant challenge for theﬁeld, limiting the applicability of continual learning solutions to realistic scenarios.

Original: Consequently, this study aims to broaden our understanding of transfer and itsdriving forces in the speciﬁc case of continual reinforcement learning.
Revised: Consequently, this study aims to broaden our understanding of transfer and itsdriving forces in the speciﬁc case of continual reinforcement learning.

Original: We adopt
Revised: We adopt

Original: SAC as the underlying RL algorithm and Continual World as a suite of continuouscontrol tasks.
Revised: SAC as the underlying RL algorithm and Continual World as a suite of continuouscontrol tasks.

Original: We systematically study how different components of SAC (theactor and the critic, exploration, and data) affect transfer efﬁcacy, and we providerecommendations regarding various modeling options.
Revised: We systematically study how different components of SAC (theactor and the critic, exploration, and data) affect transfer efﬁcacy, and we providerecommendations regarding various modeling options.

Original: The best set of choices,dubbed ClonEx-SAC , is evaluated on the recent Continual World benchmark.
Revised: The best set of choices,dubbed ClonEx-SAC , is evaluated on the recent Continual World benchmark.

Original: ClonEx-SAC achieves 87% ﬁnal success rate compared to 80% of PackNet, thebest method in the benchmark.
Revised: ClonEx-SAC achieves 87% ﬁnal success rate compared to 80% of PackNet, thebest method in the benchmark.

Original: Moreover, the transfer grows from 0 .
Revised: Moreover, the transfer grows from 0 .

Original: 18 to 0 .
Revised: 18 to 0 .

Original: inthe metric provided by Continual World.
Revised: inthe metric provided by Continual World.

Original:  Introduction
Revised:  Introduction

Original: The ability of continual learning (CL) systems ([16, 21]) to utilize knowledge from previouslyseen tasks in order to maximize transfer on the current task is a signiﬁcant challenge for the ﬁeld.
Revised: The ability of continual learning (CL) systems ([17, 22]) to utilize knowledge from previouslyseen tasks in order to maximize transfer on the current task is a signiﬁcant challenge for the ﬁeld.

Original: Achieving progress in this area would bring beneﬁts both for real-life applications and multiplemachine learning domains [23, 17, 44, 9] including reinforcement learning (RL), as advocated in[45].
Revised: Achieving progress in this area would bring beneﬁts both for real-life applications and multiplemachine learning domains [24, 18, 46, 10] including reinforcement learning (RL), as advocated in[47].

Original: In particular, it would constitute a critical step towards making efﬁcient lifelong learning agentsa reality.
Revised: In particular, it would constitute a critical step towards making efﬁcient lifelong learning agentsa reality.

Original: The goal of this paper is to expand our understanding of transfer and its driving factors in continualreinforcement learning (CRL).
Revised: The goal of this paper is to expand our understanding of transfer and its driving factors in continualreinforcement learning (CRL).

Original: As the underlying RL algorithm we assume soft actor-critic (SAC), see[15] and use Continual World [45] as the suite of continuous control environments.
Revised: As the underlying RL algorithm we assume soft actor-critic (SAC), see[16] and use Continual World [47] as the suite of continuous control environments.

Original: We systematicallystudy the critic and actor networks, the key components of SAC with regard to their inﬂuence ontransfer.
Revised: We systematicallystudy the critic and actor networks, the key components of SAC with regard to their inﬂuence ontransfer.

Original: Similarly, we measure the impact of various choices for exploration and using buffer data.
Revised: Similarly, we measure the impact of various choices for exploration and using buffer data.

Original: The low-level mechanisms of transfer are not yet fully understood even in the supervised learningcase [27].
Revised: The low-level mechanisms of transfer are not yet fully understood even in the supervised learningcase [28].

Original: To the best of our knowledge, our work is the ﬁrst one that undertakes a comprehensivestudy of this important topic in RL.
Revised: To the best of our knowledge, our work is the ﬁrst one that undertakes a comprehensivestudy of this important topic in RL.

Original: To this end, we proceed in two stages: exploring a two-tasksetting and a full continual learning scenario.
Revised: To this end, we proceed in two stages: exploring a two-tasksetting and a full continual learning scenario.

Original: We start by exploring a simpliﬁed two-tasks setting in Section 4.
Revised: We start by exploring a simpliﬁed two-tasks setting in Section 4.

Original: This allows us to leave out theimpact of forgetting, as well as limit the choices regarding exploration and data handling.
Revised: This allows us to leave out theimpact of forgetting, as well as limit the choices regarding exploration and data handling.

Original: We use 100pairs of robotic tasks from the Continual World benchmark.
Revised: We use 100pairs of robotic tasks from the Continual World benchmark.

Original: Here, we render two key observations:
Revised: Here, we render two key observations:

Original: 1) the role of the critic is the most important for transfer, while exploration and actor play smaller,
Revised: 1) the role of the critic is the most important for transfer, while exploration and actor play smaller,

Original: but non-negligible, parts; 2) contributions of the individual components are mostly independent.
Revised: but non-negligible, parts; 2) contributions of the individual components are mostly independent.

Original: Additionally, we show that the concept of feature reuse which is often used to explain supervisedtransfer learning [27, 32] might not be directly applicable in RL.
Revised: Additionally, we show that the concept of feature reuse which is often used to explain supervisedtransfer learning [28, 33] might not be directly applicable in RL.

Original: In Section 5, we aim to understand new effectswhich emerge for the full continual learning scenario, typically in longer sequences.
Revised: In Section 5, we aim to understand new effectswhich emerge for the full continual learning scenario, typically in longer sequences.

Original: In CL context, we need to take into account forgetting, being mindful of the fact that existing CL methodsoften favor mitigating forgetting at the expenseof transfer, see [45].
Revised: In CL context, we need to take into account forgetting, being mindful of the fact that existing CL methodsoften favor mitigating forgetting at the expenseof transfer, see [47].

Original: Main results include: 1)reusing policies from previous tasks for exploration considerably improves performance; 2)behavioral cloning to rehearse past tasks is beneﬁcial for both average performance and forwardtransfer, outperforming other considered methods; 3) regularizing the critic typically does nothelp for the performance of CL methods.
Revised: Main results include: 1)reusing policies from previous tasks for exploration considerably improves performance; 2)behavioral cloning to rehearse past tasks is beneﬁcial for both average performance and forwardtransfer, outperforming other considered methods; 3) regularizing the critic typically does nothelp for the performance of CL methods.

Original: The result of our comprehensive analysis is a setof general recommendations.
Revised: The result of our comprehensive analysis is a setof general recommendations.

Original: We also determinethe combination which outperforms all otheroptions, dubbed ClonEx-SAC .
Revised: We also determinethe combination which outperforms all otheroptions, dubbed ClonEx-SAC .

Original: This method utilizes behavioral cloning to mitigate catastrophicforgetting for the actor.
Revised: This method utilizes behavioral cloning to mitigate catastrophicforgetting for the actor.

Original: Moreover, at the beginning of each task, ClonEx-SAC queries all previouspolicies, the best of which generates initial exploration data.
Revised: Moreover, at the beginning of each task, ClonEx-SAC queries all previouspolicies, the best of which generates initial exploration data.

Original: ClonEx-SAC achieves 87% ﬁnal successrate compared to 80% of PackNet, the best method in the Continual World benchmark, see Figure 1.
Revised: ClonEx-SAC achieves 87% ﬁnal successrate compared to 80% of PackNet, the best method in the Continual World benchmark, see Figure 1.

Original: Importantly, we observe a sharp transfer increase from 0 .
Revised: Importantly, we observe a sharp transfer increase from 0 .

Original: 18 to 0 .
Revised: 18 to 0 .

Original: 54 in the metric provided in thebenchmark.
Revised: 54 in the metric provided in thebenchmark.

Original: Notably, the value of forward transfer closely matches the reference forward transferadjusted for exploration, which is a soft upper bound for transfer, as introduced in [45].
Revised: Notably, the value of forward transfer closely matches the reference forward transferadjusted for exploration, which is a soft upper bound for transfer, as introduced in [47].

Original:  Related work
Revised:  Related work

Original: Continual learning algorithms are often categorized into three classes: regularization-based e.g.[1, 22, 30], parameter isolation e.g. [25] and rehearsal methods e.g. [5, 6]; see also CL survey papers[7, 16, 31].
Revised: Continual learning algorithms are often categorized into three classes: regularization-based e.g.[2, 23, 31], parameter isolation e.g. [26] and rehearsal methods e.g. [6, 7]; see also CL survey papers[8, 17, 32].

Original: [42, 3] advocate the need to develop CL methods suitable for reinforcement learningtraining as a necessary step towards learning artiﬁcial intelligent agents to operate in open-endedand changing environments.
Revised: [44, 4] advocate the need to develop CL methods suitable for reinforcement learningtraining as a necessary step towards learning artiﬁcial intelligent agents to operate in open-endedand changing environments.

Original: [21] provides a detailed review of this combination and a taxonomy ofpossible setups.
Revised: [22] provides a detailed review of this combination and a taxonomy ofpossible setups.

Original: [45] proposes a sequence of robotic tasks as a benchmark, comparing popular CLmethods adapted to RL and advocating for putting more emphasis on transfer.
Revised: [47] proposes a sequence of robotic tasks as a benchmark, comparing popular CLmethods adapted to RL and advocating for putting more emphasis on transfer.

Original: The authors of [19]show how the synaptic Benna-Fusi model can be added on top of value-based RL methods to mitigateforgetting at both intra- and inter-task scales.
Revised: The authors of [20]show how the synaptic Benna-Fusi model can be added on top of value-based RL methods to mitigateforgetting at both intra- and inter-task scales.

Original: A simple approach to cloning policies from previoustasks is employed in [43], and a similar replay strategy has been used in [34].
Revised: A simple approach to cloning policies from previoustasks is employed in [45], and a similar replay strategy has been used in [35].

Original: [20] tackles the casewhen task boundaries are not provided.
Revised: [21] tackles the casewhen task boundaries are not provided.

Original: Although most of the research is concerned with model-freecontinual reinforcement learning, an approach to model-based continual RL was presented in [18].
Revised: Although most of the research is concerned with model-freecontinual reinforcement learning, an approach to model-based continual RL was presented in [19].

Original: Transfer learning, which focuses on the reuse of machine learning models, has been extremely successful recently.
Revised: Transfer learning, which focuses on the reuse of machine learning models, has been extremely successful recently.

Original: In computer vision, convolutional neural networks [23, 17] and vision transformers[10] pre-trained on large datasets can be repurposed and ﬁne-tuned on the target task.
Revised: In computer vision, convolutional neural networks [24, 18] and vision transformers[11] pre-trained on large datasets can be repurposed and ﬁne-tuned on the target task.

Original: Moderntransformer-based models [44, 9] train on large natural language corpora turned out to be very ﬂexibleand can be adapted to diverse downstream tasks, sometimes quite surprisingly [33, 24].
Revised: Moderntransformer-based models [46, 10] train on large natural language corpora turned out to be veryﬂexible and can be adapted to diverse downstream tasks, sometimes quite surprisingly [34, 25].

Original: Generalsurveys of transfer learning techniques are provided in [50, 39].
Revised: General surveys of transfer learning techniques are provided in [52, 41].

Original: Interestingly, recent research [28, 48]suggests that there are still some gaps in our understanding of transfer learning.
Revised: Interestingly, recent research[29, 50] suggests that there are still some gaps in our understanding of transfer learning.

Original: [28] analyses thelow-level reasons for transfer, exhibiting surprising phenomena such as transfer between datasets with permuted images.
Revised: [29] analyzesthe low-level reasons for transfer, exhibiting surprising phenomena such as transfer between datasets with permuted images.

Original: [48] performs large-scale experiments investigating representation transfer ina wide variety of visual tasks.
Revised: [50] performs large-scale experiments investigating representation transfer ina wide variety of visual tasks.

Original: In reinforcement learning scenarios, the structure of the underlying MDP can be exploited to facilitatethe transfer.
Revised: In reinforcement learning scenarios, the structure of the underlying MDP can be exploited to facilitatethe transfer.

Original: [40, 49, 41, 38] present methods on how to ﬁnd and use mappings between differentdomains.
Revised: [42, 51, 43, 40] present methods on how to ﬁnd and use mappings between differentdomains.

Original: [29, 4] apply reward function reshaping.
Revised: [30, 5] apply reward function reshaping.

Original: [26, 37] achieve transfer by means of high-levelskills and hierarchical RL.
Revised: [27, 39] achieve transfer by means of high-levelskills and hierarchical RL.

Original: Other lines of work exploits the model structure [35, 12] or enforcemodularity [2, 8].
Revised: Other lines of work exploits the model structure [36, 13] or enforcemodularity [3, 9].

Original: In this work, we aim to complement these studies, by focusing on the beneﬁts ofreusing neural network parameters, and other choices that exploit the RL structure, like explorationand data reuse.
Revised: In this work, we aim to complement these studies, by focusing on the beneﬁts ofreusing neural network parameters, and other choices that exploit the RL structure, like explorationand data reuse.

Original:  Background3.1 Continual learning and reinforcement learning
Revised:  Background3.1 Continual learning and reinforcement learning

Original: Continual learning tackles the problem of learning in non-stationary settings [7].
Revised: Continual learning tackles the problem of learning in non-stationary settings [8].

Original: Typically, thesolution is expected to perform well on all encountered tasks, although various metrics expressingdifferent requirements are formulated.
Revised: Typically, thesolution is expected to perform well on all encountered tasks, although various metrics expressingdifferent requirements are formulated.

Original: The popular CL desiderata include reducing the forgetting onprevious tasks and increasing the forward transfer , i.e. speeding up the learning by reusing knowledgefrom previous tasks [11, 45].
Revised: The popular CL desiderata include reducing the forgetting onprevious tasks and increasing the forward transfer , i.e. speeding up the learning by reusing knowledgefrom previous tasks [12, 47].

Original: Other desiderata focus on limiting resources, such as the number ofsamples, computation time, model size, or additional memory size.
Revised: Other desiderata focus on limiting resources, such as the number ofsamples, computation time, model size, or additional memory size.

Original: These requirements are oftenconﬂicting, so usually some trade-offs have to be made [16, 45, 31].
Revised: These requirements are oftenconﬂicting, so usually some trade-offs have to be made [17, 47, 32].

Original: Combining CL with RL adds another layer of complexity.
Revised: Combining CL with RL adds another layer of complexity.

Original: In this work, we focus on the SACalgorithm [15], which is often considered to be the method of choice for continuous control RL[47, 46, 36].
Revised: In this work, we focus on the SACalgorithm [16], which is often considered to be the method of choice for continuous control RL[49, 48, 38].

Original: As an actor-critic algorithm, it is based on the interplay between its two parts, see
Revised: As an actor-critic algorithm, it is based on the interplay between its two parts, see

Original: Section 3.2.
Revised: Section 3.2.

Original: This is a fairly complicated algorithmic setup, which presents a number of challengeswhen using jointly with CL.
Revised: This is a fairly complicated algorithmic setup, which presents a number of challengeswhen using jointly with CL.

Original: In particular, since optimization of the critic and actor networks isintertwined, it is hard to understand and decouple the impact of individual components.
Revised: In particular, since optimization of the critic and actor networks isintertwined, it is hard to understand and decouple the impact of individual components.

Original: Additionally,because of this interplay, training biases get easily exacerbated, often leading to inferior performanceor even a collapse.
Revised: Additionally,because of this interplay, training biases get easily exacerbated, often leading to inferior performanceor even a collapse.

Original: Another complication is that the training objectives for the actor and the criticare different.
Revised: Another complication is that the training objectives for the actor and the criticare different.

Original: The critic minimizes the Bellman error which is known to be a fragile objective [13]susceptible to training biases, and might correlate poorly with the value error (which we would liketo minimize).
Revised: The critic minimizes the Bellman error which is known to be a fragile objective [14]susceptible to training biases, and might correlate poorly with the value error (which we would liketo minimize).

Original: As the actor optimizes over predictions of the critic, it might also suffer from theseproblems, even if less directly.
Revised: As the actor optimizes over predictions of the critic, it might also suffer from theseproblems, even if less directly.

Original: Finally, since the policy and the data we see change during the training,there is an inherent distribution shift present, even within a single task.
Revised: Finally, since the policy and the data we see change during the training,there is an inherent distribution shift present, even within a single task.

Original: 3.2 SAC
Revised: 3.2 SAC

Original: In our study, we focus on soft actor-critic (SAC) [14], an off-policy actor-critic RL algorithm, based onthe maximum entropy principle.
Revised: In our study, we focus on soft actor-critic (SAC) [15], an off-policy actor-critic RL algorithm, based onthe maximum entropy principle.

Original: The critic strives to approximate the entropy-corrected Q -functionunder the current policy, optimizing the Bellman error.
Revised: The critic strives to approximate the entropy-corrected Q -functionunder the current policy, optimizing the Bellman error.

Original: The actor tries to ﬁnd actions that maximizethe Q -function.
Revised: The actor tries to ﬁnd actions that maximizethe Q -function.

Original: The replay buffer holds the seen experience and provides data for the actor andcritic updates at each learning step.
Revised: The replay buffer holds the seen experience and provides data for the actor andcritic updates at each learning step.

Original: The exploration policy is used to gather data at the beginningof each task for a set number of K steps.
Revised: The exploration policy is used to gather data at the beginningof each task for a set number of K steps.

Original: By default, in most SAC implementations, this meanssampling actions uniformly from the action space.
Revised: By default, in most SAC implementations, this meanssampling actions uniformly from the action space.

Original: 3.3 Continual World
Revised: 3.3 Continual World

Original: We perform our experiments on the Continual World [45] benchmark.
Revised: We perform our experiments on the Continual World [47] benchmark.

Original: It contains a set of realisticrobotic tasks, where a simulated Sawyer robot manipulates everyday objects.
Revised: It contains a set of realisticrobotic tasks, where a simulated Sawyer robot manipulates everyday objects.

Original: The structure of theobservation and action spaces remains the same between the tasks; an observation is a 12 -dimensionalvector describing the coordinates of the robot’s gripper and relevant objects.
Revised: The structure of theobservation and action spaces remains the same between the tasks; an observation is a 12 -dimensionalvector describing the coordinates of the robot’s gripper and relevant objects.

Original: The 4 -dimensionalaction space describes the gripper movement.
Revised: The 4 -dimensionalaction space describes the gripper movement.

Original: In training, a dense reward function is used to make thetasks solvable; in evaluation, the binary success metric is used to indicate whether the desired goalhas been reached.
Revised: In training, a dense reward function is used to make thetasks solvable; in evaluation, the binary success metric is used to indicate whether the desired goalhas been reached.

Original: The tasks are arranged in sequences, training in each task lasts for 1M steps.
Revised: The tasks are arranged in sequences, training in each task lasts for 1M steps.

Original: CW10 sequence contains 10 different tasks arranged in a ﬁxed order.
Revised: CW10 sequence contains 10 different tasks arranged in a ﬁxed order.

Original: CW20 consists of CW10 repeatedtwice, allowing to measure how much knowledge can be transferred in case of task repetitions.
Revised: CW20 consists of CW10 repeatedtwice, allowing to measure how much knowledge can be transferred in case of task repetitions.

Original: Weuse both CW10 and CW20 in our evaluations, as well as shorter sequences containing pairs of tasksfrom CW10.
Revised: Weuse both CW10 and CW20 in our evaluations, as well as shorter sequences containing pairs of tasksfrom CW10.

Original: 3.4 Metrics
Revised: 3.4 Metrics

Original: Following standard practice in continual learning literature, we report average performance andforgetting metrics.
Revised: Following standard practice in continual learning literature, we report average performance andforgetting metrics.

Original: We also measure transfer as deﬁned in [45].
Revised: We also measure transfer as deﬁned in [47].

Original: Below we brieﬂy recall these threemetrics.
Revised: Below we brieﬂy recall these threemetrics.

Original: Assume p
Revised: Assume p

Original: i ( t ) 2 [0 , 1] to be the performance (success rate) of task i at time t , and that eachof the N tasks is trained for  steps, so the total number of steps is T = N ·  .
Revised: i ( t ) 2 [0 , 1] to be the performance (success rate) of task i at time t , and that eachof the N tasks is trained for  steps, so the total number of steps is T = N ·  .

Original: Average performance The average performance at time t is deﬁned as P ( t ) :
Revised: Average performance The average performance at time t is deﬁned as P ( t ) :

Original: = 1 N
Revised: = 1 N

Original: P
Revised: P

Original: Ni =1 p
Revised: Ni =1 p

Original: i ( t ) .
Revised: i ( t ) .

Original: Itsﬁnal value, P ( T ) , is a scalar summary of the performance and is presented in the result tables.
Revised: Itsﬁnal value, P ( T ) , is a scalar summary of the performance and is presented in the result tables.

Original: Forward transfer The forward transfer is computed as a normalized area between the trainingcurve of the measured run and the training curve of a reference curve from training from scratch.
Revised: Forward transfer The forward transfer is computed as a normalized area between the trainingcurve of the measured run and the training curve of a reference curve from training from scratch.

Original: Let us denote by p b i 2 [0 , 1] the reference performance.
Revised: Let us denote by p b i 2 [0 , 1] the reference performance.

Original: Then the forward transfer on task i , FT i , isdeﬁned as
Revised: Then the forward transfer on task i , FT i , isdeﬁned as

Original: [Equation]
Revised: [Equation]

Original: The average forward transfer for all tasks, FT, is deﬁned as FT = 1 N
Revised: The average forward transfer for all tasks, FT, is deﬁned as FT = 1 N

Original: P
Revised: P

Original: Ni =1 FT i .
Revised: Ni =1 FT i .

Original: Forgetting For the task i , one can measure drop in performance after the end of learning on thistask as F i = p
Revised: Forgetting For the task i , one can measure drop in performance after the end of learning on thistask as F i = p

Original: i ( i ·  )  p
Revised: i ( i ·  )  p

Original: i ( T ) .
Revised: i ( T ) .

Original: Forgetting metric is deﬁned as F = 1 N
Revised: Forgetting metric is deﬁned as F = 1 N

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: 1 F i .
Revised: 1 F i .

Original: 3.5 Experimental setup
Revised: 3.5 Experimental setup

Original: We follow the experimental setup from [45].
Revised: We follow the experimental setup from [47].

Original: The actor and the critic are implemented as two separate
Revised: The actor and the critic are implemented as two separate

Original: MLP networks, each with 4 hidden layers of 256 neurons.
Revised: MLP networks, each with 4 hidden layers of 256 neurons.

Original: We refer to the 4 hidden layers as thebackbone and the last output layer as the head .
Revised: We refer to the 4 hidden layers as thebackbone and the last output layer as the head .

Original: By default, we assume the multi-head (MH) setting,where each task has its separate output head, but we also consider the single-head (SH) setting,where only a single head is used for all tasks.
Revised: By default, we assume the multi-head (MH) setting,where each task has its separate output head, but we also consider the single-head (SH) setting,where only a single head is used for all tasks.

Original: The SAC exploration phase takes K = 10 k steps.
Revised: The SAC exploration phase takes K = 10 k steps.

Original: All experiments in this paper were performed with 10 different seeds unless noted otherwise.
Revised: All experiments in this paper were performed with 10 different seeds unless noted otherwise.

Original: Wecompute 90% conﬁdence intervals through bootstrapping.
Revised: Wecompute 90% conﬁdence intervals through bootstrapping.

Original: More details on the experimental setupcan be found in Appendix A.
Revised: More details on the experimental setupcan be found in Appendix A.

Original:  Transfer in isolation
Revised:  Transfer in isolation

Original: In this section, we study what enables transfer between RL tasks .
Revised: In this section, we study what enables transfer between RL tasks .

Original: We assume a two-task setting,where we measure the forward transfer from the ﬁrst to the second task, disregarding continuallearning speciﬁc issues (e.g. forgetting), which we defer to the next section.
Revised: We assume a two-task setting,where we measure the forward transfer from the ﬁrst to the second task, disregarding continuallearning speciﬁc issues (e.g. forgetting), which we defer to the next section.

Original: We utilize all 100 pairsof CW10 tasks, see Section 3.3, to evaluate the impact of critic, actor, and exploration given by SAC.
Revised: We utilize all 100 pairsof CW10 tasks, see Section 3.3, to evaluate the impact of critic, actor, and exploration given by SAC.

Original: We will say that the actor or the critic are carried over (from the previous tasks) if their parametersare reused as the initialization in the next task; otherwise, the parameters are re-initialized.
Revised: We will say that the actor or the critic are carried over (from the previous tasks) if their parametersare reused as the initialization in the next task; otherwise, the parameters are re-initialized.

Original: We alsorefer to the exploration policy as being carried over, if we use the policy from the previous task (ortasks) to gather the data during the ﬁrst 10 k steps of the SAC exploration phase (see Section 3.2);otherwise, a uniform random exploration policy is being used.
Revised: We alsorefer to the exploration policy as being carried over, if we use the policy from the previous task (ortasks) to gather the data during the ﬁrst 10 k steps of the SAC exploration phase (see Section 3.2);otherwise, a uniform random exploration policy is being used.

Original: We use both multi-head (MH) andsingle-head (SH) settings, with the former being default.
Revised: We use both multi-head (MH) andsingle-head (SH) settings, with the former being default.

Original: Figure 2 illustrates the impact of the individual components on transfer for each pair.
Revised: Figure 2 illustrates the impact of the individual components on transfer for each pair.

Original: The ( i, j ) -thentry in the matrix contains the forward transfer value when carrying over components from task i totask j .
Revised: The ( i, j ) -thentry in the matrix contains the forward transfer value when carrying over components from task i totask j .

Original: Table 1 presents the aggregated statistics from the matrices given in Figure 2: the average FT
Revised: Table 1 presents the aggregated statistics from the matrices given in Figure 2: the average FT

Original: [Figure]
Revised: [Figure]

Original: (including and excluding the diagonal), and the number of pairs with positive, negative and neutral
Revised: (including and excluding the diagonal), and the number of pairs with positive, negative and neutral

Original: FT 1 .
Revised: FT 1 .

Original: Table 2 reports the transfer properties for all possible combinations of components present in
Revised: Table 2 reports the transfer properties for all possible combinations of components present in

Original: Table 1, omitting single-head critic (since it performs worse in Table 1).
Revised: Table 1, omitting single-head critic (since it performs worse in Table 1).

Original: 4.1 Carrying over SAC components
Revised: 4.1 Carrying over SAC components

Original: This section has two key observations.
Revised: This section has two key observations.

Original: First, the role of the critic is the most important for FT, whileexploration and actor play smaller, but non-negligible, parts.
Revised: First, the role of the critic is the most important for FT, whileexploration and actor play smaller, but non-negligible, parts.

Original: Second, the components are "transferindependent", in the sense that the transfer of the combination of the components is close to the sumof transfers yielded by each component alone.
Revised: Second, the components are "transferindependent", in the sense that the transfer of the combination of the components is close to the sumof transfers yielded by each component alone.

Original: The evidence for the ﬁrst ﬁnding is presented in detail for each pair in Figure 2 and summarized in
Revised: The evidence for the ﬁrst ﬁnding is presented in detail for each pair in Figure 2 and summarized in

Original: Table 1.
Revised: Table 1.

Original: More precisely, the average forward transfer across all pairs attributed to carrying over of thecritic equals 0 .
Revised: More precisely, the average forward transfer across all pairs attributed to carrying over of thecritic equals 0 .

Original: 2 (resp. 0 . 15 ) for MH (resp.
Revised: 2 (resp. 0 . 15 ) for MH (resp.

Original: SH) setup.
Revised: SH) setup.

Original: This separates the critic from the actor andexploration, which yield (for the default MH setup) 0 .
Revised: This separates the critic from the actor andexploration, which yield (for the default MH setup) 0 .

Original: 06 and 0 . 09 , respectively.
Revised: 06 and 0 . 09 , respectively.

Original: The importance of the critic is further emphasized by showing that restraining its learning capabilities,even when the weights are initialized to the parameters learned in the previous task, negativelyimpacts FT.
Revised: The importance of the critic is further emphasized by showing that restraining its learning capabilities,even when the weights are initialized to the parameters learned in the previous task, negativelyimpacts FT.

Original: This is shown in the last row of Table 1, where only the critic’s head is allowed to train,while the body of the network is kept frozen and carried over from the previous task.
Revised: This is shown in the last row of Table 1, where only the critic’s head is allowed to train,while the body of the network is kept frozen and carried over from the previous task.

Original: On the one hand,this result could be surprising, since feature reuse is a common technique in supervised learning (e.g.in vision [27, 32]).
Revised: On the one hand,this result could be surprising, since feature reuse is a common technique in supervised learning (e.g.in vision [28, 33]).

Original: On the other hand, the deterioration in FT can be explained by RL-speciﬁc factors.
Revised: On the other hand, the deterioration in FT can be explained by RL-speciﬁc factors.

Original: Namely, this procedure can hinder both the policy training (since the mechanics of SAC intertwinesactor and critic) and the critic training (due to inﬂated Bellman errors).
Revised: Namely, this procedure can hinder both the policy training (since the mechanics of SAC intertwinesactor and critic) and the critic training (due to inﬂated Bellman errors).

Original: As to the second ﬁnding, i.e. the "transfer independence" of the components, the results of theunderlying analysis can be found in Table 2.
Revised: As to the second ﬁnding, i.e. the "transfer independence" of the components, the results of theunderlying analysis can be found in Table 2.

Original: We observe that the reported FT for the combinationof components follows closely the sum of FTs for individual components (reported in Table 1).
Revised: We observe that the reported FT for the combinationof components follows closely the sum of FTs for individual components (reported in Table 1).

Original: Furthermore, we observe that including all the components results in the highest transfer of 0 . 35 .
Revised: Furthermore, we observe that including all the components results in the highest transfer of 0 . 35 .

Original: There is a couple of remaining interesting observations.
Revised: There is a couple of remaining interesting observations.

Original: First, Figure 2 exhibits several verticalpatterns, meaning that transfer depends more on the second task.
Revised: First, Figure 2 exhibits several verticalpatterns, meaning that transfer depends more on the second task.

Original: Second, the effect on transferincreases on the diagonal, when the exploration is carried over.
Revised: Second, the effect on transferincreases on the diagonal, when the exploration is carried over.

Original: This seems reasonable since thepolicy in the new task is initialized to the already learned policy on the same task.
Revised: This seems reasonable since thepolicy in the new task is initialized to the already learned policy on the same task.

Original: Finally, resettingthe head (MH setup) is beneﬁcial in the case of the critic, while it hurts the actor.
Revised: Finally, resettingthe head (MH setup) is beneﬁcial in the case of the critic, while it hurts the actor.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Transfer in continual learning
Revised:  Transfer in continual learning

Original: In Section 4.1, we focused on direct transfer in the two-task setting.
Revised: In Section 4.1, we focused on direct transfer in the two-task setting.

Original: Now, we move to the fullcontinual learning scenario, which brings two main differences: 1) we measure the performance ofall tasks in the sequence, so forgetting now plays a signiﬁcant role; 2) typically, we consider longersequences of tasks of length 10 and 20 (CW10 and CW20, respectively).
Revised: Now, we move to the fullcontinual learning scenario, which brings two main differences: 1) we measure the performance ofall tasks in the sequence, so forgetting now plays a signiﬁcant role; 2) typically, we consider longersequences of tasks of length 10 and 20 (CW10 and CW20, respectively).

Original: For longer sequences,forgetting and transfer may have complex mutual interactions.
Revised: For longer sequences,forgetting and transfer may have complex mutual interactions.

Original: To reduce forgetting, CL methodsusually regularize neural network weights, which in turn may be harmful to transfer.
Revised: To reduce forgetting, CL methodsusually regularize neural network weights, which in turn may be harmful to transfer.

Original: On the otherhand, transfer beneﬁts from accumulated knowledge – if forgetting is not mitigated, there might benothing to transfer from.
Revised: On the otherhand, transfer beneﬁts from accumulated knowledge – if forgetting is not mitigated, there might benothing to transfer from.

Original: We will investigate three main themes.
Revised: We will investigate three main themes.

Original: The ﬁrst one is reusing previous policies for exploration.
Revised: The ﬁrst one is reusing previous policies for exploration.

Original: For long sequences, there are multiple design choices available compared to the two-task scenarios.
Revised: For long sequences, there are multiple design choices available compared to the two-task scenarios.

Original: Secondly, we investigate CL with data reuse, an approach successful in supervised learning.
Revised: Secondly, we investigate CL with data reuse, an approach successful in supervised learning.

Original: We showthat the CRL setup is more complex and requires careful investigation.
Revised: We showthat the CRL setup is more complex and requires careful investigation.

Original: Finally, given the importanceof the critic for transfer (see Section 4), we study whether the critic should be regularized or not, andconclude that typically, the answer is negative.
Revised: Finally, given the importanceof the critic for transfer (see Section 4), we study whether the critic should be regularized or not, andconclude that typically, the answer is negative.

Original: We study these issues in conjunction with various CL methods: Fine-tuning, Perfect memory, EWC,
Revised: We study these issues in conjunction with various CL methods: Fine-tuning, Perfect memory, EWC,

Original: PackNet, L2, A-GEM, MAS, and VCL.
Revised: PackNet, L2, A-GEM, MAS, and VCL.

Original: These are standard CL approaches adopted and tested in the
Revised: These are standard CL approaches adopted and tested in the

Original: RL setting [45], see details in Appendix B.
Revised: RL setting [47], see details in Appendix B.

Original: We note that CL methods used here are mostly successfulin mitigating forgetting; in this section, we report average performance and forward transfer, deferringforgetting to the Appendices C and E.
Revised: We note that CL methods used here are mostly successfulin mitigating forgetting; in this section, we report average performance and forward transfer, deferringforgetting to the Appendices C and E.

Original: 5.1 Exploration
Revised: 5.1 Exploration

Original: When using the SAC algorithm, at the beginning of each task, there is a short period of explorationwith a random policy, see Section 3.2.
Revised: When using the SAC algorithm, at the beginning of each task, there is a short period of explorationwith a random policy, see Section 3.2.

Original: The experiments in Section 4.1 showed that the transferincreases if the policy from the previous task is used instead.
Revised: The experiments in Section 4.1 showed that the transferincreases if the policy from the previous task is used instead.

Original: Now we pass from two-task scenarios to longer ones, and analyze the following options for choosingexploration policy, which we call: random, preceding, uniform-previous , and best-return , and deﬁnethem as follows.
Revised: Now we pass from two-task scenarios to longer ones, and analyze the following options for choosingexploration policy, which we call: random, preceding, uniform-previous , and best-return , and deﬁnethem as follows.

Original: In the ﬁrst task, we always use a random policy, and assume that the tasks arenumbered from 1 to N .
Revised: In the ﬁrst task, we always use a random policy, and assume that the tasks arenumbered from 1 to N .

Original: Consider now i 2 { 2 , . .
Revised: Consider now i 2 { 2 , . .

Original: . , N } .
Revised: . , N } .

Original: For the random strategy, we randomly samplefrom the action space, which is a default choice for SAC.
Revised: For the random strategy, we randomly samplefrom the action space, which is a default choice for SAC.

Original: For other strategies, at the beginning ofeach exploration episode, we choose a previous actor head to generate data instead of the randompolicy.
Revised: For other strategies, at the beginning ofeach exploration episode, we choose a previous actor head to generate data instead of the randompolicy.

Original: In the case of preceding strategy, we use the ( i  1) -th actor’s head.
Revised: In the case of preceding strategy, we use the ( i  1) -th actor’s head.

Original: For uniform-previouspolicy, we use the j -th actor’s head, where j := RANDOM _ UNIFORM ( { 1 , . .
Revised: For uniform-previouspolicy, we use the j -th actor’s head, where j := RANDOM _ UNIFORM ( { 1 , . .

Original: , i  1 } ) .
Revised: , i  1 } ) .

Original: Finally, inbest-return strategy, we ﬁrst try every possible head, and then act using the j max -th actor’s head,where j max := argmax j 2 { 1 ,...,i  1 } R ij ; R ij is the return of the j -th head policy on the i -th task.
Revised: Finally, inbest-return strategy, we ﬁrst try every possible head, and then act using the j max -th actor’s head,where j max := argmax j 2 { 1 ,...,i  1 } R ij ; R ij is the return of the j -th head policy on the i -th task.

Original: We evaluate how these strategies interact with various CL methods.
Revised: We evaluate how these strategies interact with various CL methods.

Original: We pick Fine-tuning, Behavioralcloning, L2, EWC, and PackNet.
Revised: We pick Fine-tuning, Behavioralcloning, L2, EWC, and PackNet.

Original: The results for two well-performing methods, EWC and PackNet,are presented in Table 3, with the rest being deferred to Appendix E. For EWC, choosing anynon-random policy signiﬁcantly improves upon the baseline random strategy.
Revised: The results for two well-performing methods, EWC and PackNet,are presented in Table 3, with the rest being deferred to Appendix E. For EWC, choosing anynon-random policy signiﬁcantly improves upon the baseline random strategy.

Original: This is particularlyvisible in the CW20 sequence, which contains repeated tasks, and arguably can beneﬁt more from aninformed strategy like best-return.
Revised: This is particularlyvisible in the CW20 sequence, which contains repeated tasks, and arguably can beneﬁt more from aninformed strategy like best-return.

Original: Interestingly, the results for the rather simple uniform-previousapproach are quite competitive.
Revised: Interestingly, the results for the rather simple uniform-previousapproach are quite competitive.

Original: We observe increased performance also for other methods except for
Revised: We observe increased performance also for other methods except for

Original: PackNet, for which effects are negligible.
Revised: PackNet, for which effects are negligible.

Original: [Table]
Revised: [Table]

Original: 5.2 Data rehearsal
Revised: 5.2 Data rehearsal

Original: Rehearsal techniques work very well in supervised continual learning [6].
Revised: Rehearsal techniques work very well in supervised continual learning [7].

Original: In RL, two major approaches to utilizing previous data have been studied: applying them as ofﬂine data using SAC loss,and behavioral cloning of the previous policies.
Revised: In RL, two major approaches to utilizing previous data have been studied: applying them as ofﬂine data using SAC loss,and behavioral cloning of the previous policies.

Original: The former, dubbed Perfect memory, was reported toperform poorly [45].
Revised: The former, dubbed Perfect memory, was reported toperform poorly [47].

Original: Behavioral cloning achieves more promising results [43, 34].
Revised: Behavioral cloning achieves more promising results [45, 35].

Original: We study thesetwo approaches with an emphasis on the effects on transfer.
Revised: We study thesetwo approaches with an emphasis on the effects on transfer.

Original: In Perfect memory, all the experiences are kept in the buffer.
Revised: In Perfect memory, all the experiences are kept in the buffer.

Original: SAC training is applied to data fromthe current task and ofﬂine data from the previous ones.
Revised: SAC training is applied to data fromthe current task and ofﬂine data from the previous ones.

Original: For Behavioral cloning, an additional smallbuffer is ﬁlled at the end of training on each task.
Revised: For Behavioral cloning, an additional smallbuffer is ﬁlled at the end of training on each task.

Original: We annotate a subset of samples from the main
Revised: We annotate a subset of samples from the main

Original: SAC buffer using the trained actor and critic networks.
Revised: SAC buffer using the trained actor and critic networks.

Original: When training the new task, we sampledata from expert buffers and apply auxiliary losses (with different weights), minimizing the KLdivergence between current and saved outputs for the actor and L2 distance for the critic; see detailsin Appendix B.
Revised: When training the new task, we sampledata from expert buffers and apply auxiliary losses (with different weights), minimizing the KLdivergence between current and saved outputs for the actor and L2 distance for the critic; see detailsin Appendix B.

Original: Firstly, we study the effect of rehearsal on transfer in the two-task scenario, using 100 task pairs from
Revised: Firstly, we study the effect of rehearsal on transfer in the two-task scenario, using 100 task pairs from

Original: CW10, as in Section 4.1.
Revised: CW10, as in Section 4.1.

Original: We observe that using Perfect memory has a detrimental effect on transfer,and cloning the actor only has a rather neutral effect; we report results for these and more setups in
Revised: We observe that using Perfect memory has a detrimental effect on transfer,and cloning the actor only has a rather neutral effect; we report results for these and more setups in

Original: Appendix E.
Revised: Appendix E.

Original: Secondly, we perform experiments on longer sequences, CW10 and CW20; see Table 4.
Revised: Secondly, we perform experiments on longer sequences, CW10 and CW20; see Table 4.

Original: For reference,we include two methods tested in [45], Fine-tuning and PackNet.
Revised: For reference,we include two methods tested in [47], Fine-tuning and PackNet.

Original: Fine-tuning achieves the highest
Revised: Fine-tuning achieves the highest

Original: [Table]
Revised: [Table]

Original: transfer and PackNet the highest overall performance out of the methods tested in [45].
Revised: transfer and PackNet the highest overall performance out of the methods tested in [47].

Original: Behavioralcloning performs very well.
Revised: Behavioralcloning performs very well.

Original: In terms of the average performance, it is on par with PackNet on CW10and better on CW20.
Revised: In terms of the average performance, it is on par with PackNet on CW10and better on CW20.

Original: Importantly, it signiﬁcantly outperforms the baselines in terms of transfer.
Revised: Importantly, it signiﬁcantly outperforms the baselines in terms of transfer.

Original: Wecan see that Perfect memory works poorly, in line with existing literature.
Revised: Wecan see that Perfect memory works poorly, in line with existing literature.

Original: In Appendix C, we presentthe results for ﬁve other CL methods benchmarked in [45].
Revised: In Appendix C, we presentthe results for ﬁve other CL methods benchmarked in [47].

Original: We observe an interesting phenomenon.
Revised: We observe an interesting phenomenon.

Original: While transfer was not improved in two-task scenario, it hasincreased in the case of longer sequences.
Revised: While transfer was not improved in two-task scenario, it hasincreased in the case of longer sequences.

Original: This hints that perhaps the learner accumulates knowledgeof the previous tasks and, thus, can reuse the most relevant one to improve the training of the currenttask.
Revised: This hints that perhaps the learner accumulates knowledgeof the previous tasks and, thus, can reuse the most relevant one to improve the training of the currenttask.

Original: Additionally, perhaps behavioral cloning loss acts as a regularizer and helps shaping moregeneral features, thus further improving transfer.
Revised: Additionally, perhaps behavioral cloning loss acts as a regularizer and helps shaping moregeneral features, thus further improving transfer.

Original: 5.3 Regularizing the critic
Revised: 5.3 Regularizing the critic

Original: This section is devoted to the study of critic regularization in CRL methods.
Revised: This section is devoted to the study of critic regularization in CRL methods.

Original: Since the primaryobjective of CRL is the ﬁnal performance of the actor, we have some ﬂexibility in how we treat thecritic.
Revised: Since the primaryobjective of CRL is the ﬁnal performance of the actor, we have some ﬂexibility in how we treat thecritic.

Original: We can even completely ignore forgetting in the critic, as recommended in [45].
Revised: We can even completely ignore forgetting in the critic, as recommended in [47].

Original: Other workssuggest that regularization might be beneﬁcial [20].
Revised: Other workssuggest that regularization might be beneﬁcial [21].

Original: To understand this better, we carefully measure the performance for various critic regularizationweights for EWC, L2, and Behavioral cloning.
Revised: To understand this better, we carefully measure the performance for various critic regularizationweights for EWC, L2, and Behavioral cloning.

Original: We ﬁrst ﬁnd a good value for the actor weight, withthe one corresponding to the critic being set to 0 .
Revised: We ﬁrst ﬁnd a good value for the actor weight, withthe one corresponding to the critic being set to 0 .

Original: Then, with this value, we perform the sweep overthe critic weight.
Revised: Then, with this value, we perform the sweep overthe critic weight.

Original: For Behavioral cloning, we consider a range of values from 0 .
Revised: For Behavioral cloning, we consider a range of values from 0 .

Original: 0001 to 100 (the latterbeing the coefﬁcient for the actor).
Revised: 0001 to 100 (the latterbeing the coefﬁcient for the actor).

Original: First, we perform experiments for the pairs of tasks from CW10.
Revised: First, we perform experiments for the pairs of tasks from CW10.

Original: Small weight values exhibit similar performance to non-regularized critics i.e. with the weight 0 .
Revised: Small weight values exhibit similar performance to non-regularized critics i.e. with the weight 0 .

Original: Forlarger, we see a performance drop.
Revised: Forlarger, we see a performance drop.

Original: Then, we study a longer CW10 sequence, for which there is asharp drop, even for the smallest weights: 0.82 (critic coef.
Revised: Then, we study a longer CW10 sequence, for which there is asharp drop, even for the smallest weights: 0.82 (critic coef.

Original: = 0.0001), 0.77 (critic coef.
Revised: = 0.0001), 0.77 (critic coef.

Original: = 0.001);compared to 0.84 for the non-regularized critic.
Revised: = 0.001);compared to 0.84 for the non-regularized critic.

Original: The complete results are presented in Appendix E.
Revised: The complete results are presented in Appendix E.

Original: This conﬁrms the practical recommendation from [45] to regularize only the actor.
Revised: This conﬁrms the practical recommendation from [47] to regularize only the actor.

Original: One possible explanation is that TD -learning used for the critic is highly sensitive to biases introduced byregularization.
Revised: One possible explanation is that TD -learning used for the critic is highly sensitive to biases introduced byregularization.

Original:  Combining the improvements: ClonEx-SAC
Revised:  Combining the improvements: ClonEx-SAC

Original: Based on the experimental ﬁndings presented so far, we propose to combine the discovered enhancements in a simple method for continual reinforcement learning.
Revised: Based on the experimental ﬁndings presented so far, we propose to combine the discovered enhancements in a simple method for continual reinforcement learning.

Original: This method signiﬁcantly improvesthe performance in the Continual World benchmark [45].
Revised: This method signiﬁcantly improvesthe performance in the Continual World benchmark [47].

Original: In particular, we observe a sharp transferincrease to a value that matches a soft upper bound for transfer introduced in [45].
Revised: In particular, we observe a sharp transferincrease to a value that matches a soft upper bound for transfer introduced in [47].

Original: We incorporate the following choices in the method:
Revised: We incorporate the following choices in the method:

Original: • We use behavioral cloning for the actor, which, as we showed in Section 5.2, effectivelymitigates forgetting and increases transfer.
Revised: • We use behavioral cloning for the actor, which, as we showed in Section 5.2, effectivelymitigates forgetting and increases transfer.

Original: • We use best-return exploration, as described in Section 5.1, which efﬁciently reuses old
Revised: • We use best-return exploration, as described in Section 5.1, which efﬁciently reuses old

Original: policy heads for faster exploration.
Revised: policy heads for faster exploration.

Original: • As indicated in Section 5.3, we do not use any CL regularization for the critic.
Revised: • As indicated in Section 5.3, we do not use any CL regularization for the critic.

Original: • We use multiple output heads for both actor and critic to proﬁt from transferred repre-
Revised: • We use multiple output heads for both actor and critic to proﬁt from transferred repre-

Original: sentations without introducing too much bias in the new tasks, as discussed in Section4.1.We dub the method ClonEx-SAC to reﬂect the usage of the behavioral clon ing, improved ex ploration,
Revised: sentations without introducing too much bias in the new tasks, as discussed in Section4.1.We dub the method ClonEx-SAC to reﬂect the usage of the behavioral clon ing, improved ex ploration,

Original: and SAC algorithm
Revised: and SAC algorithm

Original: We compare ClonEx-SAC with the behavioral cloning and 7 methods considered in [45], on the
Revised: We compare ClonEx-SAC with the behavioral cloning and 7 methods considered in [47], on the

Original: CW10 and CW20 sequences.
Revised: CW10 and CW20 sequences.

Original: We present results in Figure 1 (see Introduction) and
Revised: We present results in Figure 1 (see Introduction) and

Original: Appendix C.
Revised: Appendix C.

Original: ClonEx-SAC achieves 87% ﬁnal performance compared to 80% of PackNet, the best method in[45].
Revised: ClonEx-SAC achieves 87% ﬁnal performance compared to 80% of PackNet, the best method in[47].

Original: The forward transfer of ClonEx-SAC , improves sharply from 0 .
Revised: The forward transfer of ClonEx-SAC , improves sharply from 0 .

Original: 19 , the best previous result,to 0 .
Revised: 19 , the best previous result,to 0 .

Original: Notably, ClonEx-SAC ’s result closely matches the reference forward transfer, see below.
Revised: Notably, ClonEx-SAC ’s result closely matches the reference forward transfer, see below.

Original: We conjecture that this excellent transfer is an important factor in the ﬁnal performance.
Revised: We conjecture that this excellent transfer is an important factor in the ﬁnal performance.

Original: We alsonotice that improvements brought separately by the behavioral cloning and the best-return explorationstrategy work well together.
Revised: We alsonotice that improvements brought separately by the behavioral cloning and the best-return explorationstrategy work well together.

Original: Reference forward transfer (RT) was introduced in [45] as a soft upper bound for transfer.
Revised: Reference forward transfer (RT) was introduced in [47] as a soft upper bound for transfer.

Original: Fora sequence of tasks t 1 , . .
Revised: Fora sequence of tasks t 1 , . .

Original: , t N , it is deﬁned deﬁned as RT :
Revised: , t N , it is deﬁned deﬁned as RT :

Original: = 1 N
Revised: = 1 N

Original: P
Revised: P

Original: N i =2 max j<i FT ( t j , t i ) , where
Revised: N i =2 max j<i FT ( t j , t i ) , where

Original: F ( t j , t i ) denotes the forward transfer for the pair of tasks t j , t i .
Revised: F ( t j , t i ) denotes the forward transfer for the pair of tasks t j , t i .

Original: Intuitively, RT estimates the levelof forward transfer, which could be achieved when a method is able to remember and transfer allmeaningful aspects of previously seen tasks.
Revised: Intuitively, RT estimates the levelof forward transfer, which could be achieved when a method is able to remember and transfer allmeaningful aspects of previously seen tasks.

Original: Note that in principle, higher values of RT could still beachievable if the knowledge from the previous tasks is composed.
Revised: Note that in principle, higher values of RT could still beachievable if the knowledge from the previous tasks is composed.

Original: In our setup, the values of RT are0 .
Revised: In our setup, the values of RT are0 .

Original: 44 for CW10 and 0 .
Revised: 44 for CW10 and 0 .

Original: 55 for CW20.
Revised: 55 for CW20.

Original: In both cases, they are closely matched by the forward transferof ClonEx-SAC .
Revised: In both cases, they are closely matched by the forward transferof ClonEx-SAC .

Original: We note that our RT values are higher than the one reported in [45], since theirwork does not take into account the effects of improved exploration.
Revised: We note that our RT values are higher than the one reported in [47], since theirwork does not take into account the effects of improved exploration.

Original:  Limitations
Revised:  Limitations

Original: We are fully aware that our analyses do not cover the entire spectrum of problems that one might beinterested in when studying transfer in CRL.
Revised: We are fully aware that our analyses do not cover the entire spectrum of problems that one might beinterested in when studying transfer in CRL.

Original: Here we summarize a few limitations of our work:
Revised: Here we summarize a few limitations of our work:

Original: • We build on top of the SAC algorithm.
Revised: • We build on top of the SAC algorithm.

Original: There is a risk that some of the conclusions fromthis paper would differ for another choice of the underlying RL method.
Revised: There is a risk that some of the conclusions fromthis paper would differ for another choice of the underlying RL method.

Original: • We focus on the Continual World suite.
Revised: • We focus on the Continual World suite.

Original: There is a possibility that some of the results from
Revised: There is a possibility that some of the results from

Original: this paper would differ in environments from other domains or with different, potentiallystructured state spaces.•
Revised: this paper would differ in environments from other domains or with different, potentiallystructured state spaces.•

Original: ClonEx-SAC requires retaining data from previous tasks, which may not always be feasible(e
Revised: ClonEx-SAC requires retaining data from previous tasks, which may not always be feasible(e

Original: g., due to privacy concerns).
Revised: g., due to privacy concerns).

Original:  Conclusions
Revised:  Conclusions

Original: In this work, we identify and study some of the key factors contributing to transfer in continualreinforcement learning.
Revised: In this work, we identify and study some of the key factors contributing to transfer in continualreinforcement learning.

Original: In the ﬁrst part of the study, we focus on the transfer alone, disregardingother CL desiderata, and analyze how much different components of the SAC algorithm (actor, critic,exploration) contribute to it.
Revised: In the ﬁrst part of the study, we focus on the transfer alone, disregardingother CL desiderata, and analyze how much different components of the SAC algorithm (actor, critic,exploration) contribute to it.

Original: We identify the critic as the leading component.
Revised: We identify the critic as the leading component.

Original: In the second part, westudy further effects that are relevant to the full continual learning setup with long task sequences.
Revised: In the second part, westudy further effects that are relevant to the full continual learning setup with long task sequences.

Original: In particular, we show that behavioral cloning and reusing previous policies for exploration signiﬁcantly improve both transfer and the ﬁnal performance.
Revised: In particular, we show that behavioral cloning and reusing previous policies for exploration signiﬁcantly improve both transfer and the ﬁnal performance.

Original: This leads to a new method, ClonEx-SAC ,which outperforms considered baselines.
Revised: This leads to a new method, ClonEx-SAC ,which outperforms considered baselines.

Original: We believe that this work constitutes the ﬁrst step towardunderstanding the mechanisms behind transfer in continual reinforcement learning.
Revised: We believe that this work constitutes the ﬁrst step towardunderstanding the mechanisms behind transfer in continual reinforcement learning.

Original: There are stillimportant issues to be resolved, e.g., pinpointing the exact role of feature reuse or the interplaybetween transfer and forgetting.
Revised: There are stillimportant issues to be resolved, e.g., pinpointing the exact role of feature reuse or the interplaybetween transfer and forgetting.

Original: We hope that these will be addressed by the community in the future.
Revised: We hope that these will be addressed by the community in the future.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•
Revised: [Yes] See Section ?? .

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] In Section (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] In Section 7(c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do notsee this work as having a signiﬁcant societal impact.
Revised: [No] We do not see this work as having a signiﬁcant societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] The code, including the scripts used to run the experiments from the paper, are in the supplemen-
Revised: [Yes] The code,including the scripts used to run the experiments from the paper, are in the supplemen-

Original: tary materials.
Revised: tary materials.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] We conduct each experiments with multiple seeds (at
Revised: [Yes] We conduct each experiments with multiple seeds (at

Original: least 10)
Revised: least 10)

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We describe these details in
Revised: [Yes] We describe these details in

Original: Appendix F.
Revised: Appendix F.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research withhuman subjects...
Revised: If you used crowdsourcing or conducted research with

Revised: human subjects

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c [Table]
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]