Original: Evaluations of the Gap between Supervised and Reinforcement Lifelong Learning on Robotic Manipulation Tasks
Revised: Evaluations of the Gap between Supervised and Reinforcement Lifelong Learning on Robotic Manipulation Tasks

Original: Abstract: Overcoming catastrophic forgetting is of great importance for deep learning and robotics.
Revised: Abstract: Overcoming catastrophic forgetting is of great importance for deep learning and robotics.

Original: Recent lifelong learning research has great advances in supervised learning.
Revised: Recent lifelong learning research has great advances in supervised learning.

Original: However, little work focuses on reinforcement learning(RL).
Revised: However, little work focuses on reinforcement learning(RL).

Original: We focus on evaluating the performances of state-of-the-art lifelong learning algorithms on robotic reinforcement learning tasks.
Revised: We focus on evaluating the performances of state-of-the-art lifelong learning algorithms on robotic reinforcement learning tasks.

Original: We mainly focus on the properties of overcoming catastrophic forgetting for these algorithms.
Revised: We mainly focus on the properties of overcoming catastrophic forgetting for these algorithms.

Original: We summarize the pros and cons for each category of lifelong learning algorithms when applied in
Revised: We summarize the pros and cons for each category of lifelong learning algorithms when applied in

Original: RL scenarios.
Revised: RL scenarios.

Original: We propose a framework to modify supervised lifelong learning algorithms to be compatible with RL.
Revised: We propose a framework to modify supervised lifelong learning algorithms to be compatible with RL.

Original: We also develop a manipulation benchmark task set for our evaluations.
Revised: We also develop a manipulation benchmark task set for our evaluations.

Original:  Introduction
Revised:  Introduction

Original: Humans and animals continuously learn different skills and knowledge in their lifetime.
Revised: Humans and animals continuously learn different skills and knowledge in their lifetime.

Original: It is natural for them to master a new skill without forgetting those learned before in most cases.
Revised: It is natural for them to master a new skill without forgetting those learned before in most cases.

Original: However, neural networks suffer a lot from memorizing previously learned tasks.
Revised: However, neural networks suffer a lot from memorizing previously learned tasks.

Original: This phenomenon is called catastrophic forgetting [1, 2, 3].
Revised: This phenomenon is called catastrophic forgetting [1, 2, 3].

Original: Recent research has focused on this problem and proposed various solutions called lifelong learning or continual learning.
Revised: Recent research has focused on this problem and proposed various solutions called lifelong learning or continual learning.

Original: On the other hand, manipulation has been a challenging task in the robotics community.

Original: Speciﬁcally, lifelong learning is of great importance for robotics.
Revised: On the other hand, lifelong learning is of great importance for robotics.

Original: It provides possible solutions for developmental robotics [4].
Revised: It provides possible solutions for developmental robotics [4].

Original: Humans learn different skills as they grow up rather than completely rely on innate skills.
Revised: Humans learn different skills as they grow up rather than completely rely on innate skills.

Original: It would also be difﬁcult to pre-program all the required skills for advanced robots when setting up in the factory.
Revised: It would also be difﬁcult to pre-program all the required skills for advanced robots when setting up in the factory.

Original: Future robots will have to continuously master different tasks as humans do in their lifetime.
Revised: Future robots will have to continuously master different tasks as humans do in their lifetime.

Original: The ability of lifelong learning will make robots ﬁt better in various applications.
Revised: The ability of lifelong learning will make robots ﬁt better in various applications.

Original: In this work, we focus on the lifelong learning problem for robotic manipulation tasks in the reinforcement learning(RL) domain, especially forgetting properties.
Revised: In this work, we focus on the lifelong learning problem for robotic manipulation tasks in the reinforcement learning(RL) domain, especially forgetting properties.

Original: Most of the recent lifelong learning research focuses on supervised learning.
Revised: Most of the recent lifelong learning research focuses on supervised learning.

Original: However, lifelong learning for RL is much more challenging than supervised learning.
Revised: However, lifelong learning for RL is much more challenging than supervised learning.

Original: Not only because the training of RL itself is challenging and unstable[5], but in most RL tasks, policies are sensitive to forgetting.
Revised: Not only because the training of RL itself is challenging and unstable[5], but in most RL tasks, policies are sensitive to forgetting.

Original: Forgetting could be considered as an attack in the action space of the agent.
Revised: Forgetting could be considered as an attack in the action space of the agent.

Original: Recent work has demonstrated the vulnerability of
Revised: Recent work has demonstrated the vulnerability of

Original: RL agents to attack on action space[6].
Revised: RL agents to attack on action space[6].

Original: Moreover, in RL tasks, an action at one step will inﬂuence states and actions followed up, which could aggravate the inﬂuence of forgetting.
Revised: Moreover, in RL tasks, an action at one step will inﬂuence states and actions followed up, which could aggravate the inﬂuence of forgetting.

Original: For example, in grasping tasks, slight misplacement of a ﬁngertip is likely to break the balance of a grasp and lead to its failure.
Revised: For example, in grasping tasks, slight misplacement of a ﬁngertip is likely to break the balance of a grasp and lead to its failure.

Original: Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks.
Revised: Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks.

Original: Research on how the state-of-the-art lifelong learning algorithms perform on robotic reinforcement learning tasks is urgently needed in the research community, which is the major motivation of our work.
Revised: Research on how the state-of-the-art lifelong learning algorithms perform on robotic reinforcement learning tasks is urgently needed in the research community, which is the major motivation of our work.

Original: In this paper, we establish a framework to study the possibilities to extend the existing supervised lifelong learning method to the reinforcement scenarios, demonstrating their effectiveness on a5th Conference on Learning (CoRL 2021), London, UK. carefully-designed multi-task manipulation scenario.
Revised: In this paper, we establish a framework to study the possibilities to extend the existing supervised lifelong learning method to the reinforcement scenarios, demonstrating their effectiveness on a carefully-designed multi-task manipulation scenario.

Original: Our contributions can be summarized into three-folds:
Revised: Our contributions can be summarized into three-folds 1 :

Original: We evaluate Soft-Actor-Critic[7] with the state-of-the-art lifelong learning algorithms from four different categories using our benchmark.
Revised: We evaluate Soft-Actor-Critic[7] with the state-of-the-art lifelong learning algorithms from four different categories using our benchmark.

Original: Forgetting performances and robustness to task orders are evaluated in experiments.
Revised: Forgetting performances and robustness to task orders are evaluated in experiments.

Original: We further analyze the possibility of applying different lifelong learning algorithms into RL based on our experiments.
Revised: We further analyze the possibility of applying different lifelong learning algorithms into RL based on our experiments.

Original: We propose a manipulation benchmark for lifelong learning with ten different manipulation tasks.
Revised: We propose a manipulation benchmark for lifelong learning with ten different manipulation tasks.

Original: Policies trained on our benchmark could quickly converge.
Revised: Policies trained on our benchmark could quickly converge.

Original: We propose a framework which could modify traditional lifelong learning algorithms from supervised learning into reinforcement learning.
Revised: We propose a framework which could modify traditional lifelong learning algorithms from supervised learning into reinforcement learning.

Original:  Related Works 2.1 Reinforcement Learning with Multiple Tasks
Revised:  Related Works 2.1 Reinforcement Learning with Multiple Tasks

Original: In addition to lifelong learning, meta-learning and multi-task learning both include multiple tasks in their problem formulations.
Revised: In addition to lifelong learning, meta-learning and multi-task learning both include multiple tasks in their problem formulations.

Original: They are similar domains but share plenty of differences.
Revised: They are similar domains but share plenty of differences.

Original: It is important to notice that in traditional lifelong learning formulations, tasks are assigned to the agent sequentially , which means that the agent will not be able to have access to the training data and environments after ﬁnishing training on this task.
Revised: It is important to notice that in traditional lifelong learning formulations, tasks are assigned to the agent sequentially , which means that the agent will not be able to have access to the training data and environments after ﬁnishing training on this task.

Original: Furthermore, recent researches in lifelong learning focus on overcoming catastrophic forgetting and memorizing all the tasks learned.
Revised: Furthermore, recent researches in lifelong learning focus on overcoming catastrophic forgetting and memorizing all the tasks learned.

Original: Though the abilities of transfer learning may also be discussed, they are not the core questions in this paper.
Revised: Though the abilities of transfer learning may also be discussed, they are not the core questions in this paper.

Original: In the following sub-sections, we will discuss meta-learning and multi-task learning.
Revised: In the following sub-sections, we will discuss meta-learning and multi-task learning.

Original: Especially on the differences and similarities between these domains and lifelong learning.
Revised: Especially on the differences and similarities between these domains and lifelong learning.

Original: 2.1.1 Meta-Learning
Revised: 2.1.1 Meta-Learning

Original: In meta-learning, research focuses on achieving fast adaptation based on agents’ previous learning procedures.
Revised: In meta-learning, research focuses on achieving fast adaptation based on agents’ previous learning procedures.

Original: Speciﬁcally, [8] introduces an additional gradient descent procedure to update the network parameters for a novel task.
Revised: Speciﬁcally, [8] introduces an additional gradient descent procedure to update the network parameters for a novel task.

Original: [9] applies meta-learning to quickly adapt to a new dynamic model in a new environment, which leads to a faster generalization in a new environment.
Revised: [9] applies meta-learning to quickly adapt to a new dynamic model in a new environment, which leads to a faster generalization in a new environment.

Original: [10, 11] introduces a latent variable to represent the dynamic environment, and it could be used to help fast adaptation of robotic tasks.
Revised: [10, 11] introduces a latent variable to represent the dynamic environment, and it could be used to help fast adaptation of robotic tasks.

Original: 2.1.2 Multi-Task Learning
Revised: 2.1.2 Multi-Task Learning

Original: Multi-task learning is deﬁned as solving multiple tasks at the same time.
Revised: Multi-task learning is deﬁned as solving multiple tasks at the same time.

Original: Multi-task learning includes lifelong learning but is not limited to it.
Revised: Multi-task learning includes lifelong learning but is not limited to it.

Original: Recent work on multi-task learning focuses on exploiting the similarities of tasks to accelerate the learning of a novel task.
Revised: Recent work on multi-task learning focuses on exploiting the similarities of tasks to accelerate the learning of a novel task.

Original: Speciﬁcally, [12, 13] focuses on how auxiliary tasks help improve the performance and learning speed of current tasks.
Revised: Speciﬁcally, [12, 13] focuses on how auxiliary tasks help improve the performance and learning speed of current tasks.

Original: Distral[14] introduces a condensed shared policy for faster learning of a new task.
Revised: Distral[14] introduces a condensed shared policy for faster learning of a new task.

Original: IMPALA[15] and its updated version PopArt-IMPALA[16] uses multiple agents to explore and train in different tasks at the same time.
Revised: IMPALA[15] and its updated version PopArt-IMPALA[16] uses multiple agents to explore and train in different tasks at the same time.

Original: Note that in multi-task learning, tasks are not required to be learned sequentially, which is different from lifelong learning.
Revised: Note that in multi-task learning, tasks are not required to be learned sequentially, which is different from lifelong learning.

Original: In summary, lifelong learning, meta-learning, and multi-task learning are mutually similar in the
Revised: In summary, lifelong learning, meta-learning, and multi-task learning are mutually similar in the

Original: RL domain.
Revised: RL domain.

Original: However, they focus on different performances.
Revised: However, they focus on different performances.

Original: In lifelong learning, tasks are trained sequentially and the performance of memorizing policies for previous tasks is highlighted.
Revised: In lifelong learning, tasks are trained sequentially and the performance of memorizing policies for previous tasks is highlighted.

Original: 2.2 Lifelong Learning in Reinforcement Learning
Revised: 2.2 Lifelong Learning in Reinforcement Learning

Original: In terms of overcoming catastrophic forgetting, most lifelong learning algorithms only focus on solving supervised learning tasks, while work on RL has received little attention.
Revised: In terms of overcoming catastrophic forgetting, most lifelong learning algorithms only focus on solving supervised learning tasks, while work on RL has received little attention.

Original: There are some algorithms trying to solve RL tasks, but they are applied in image-based video games instead of robotic tasks, e.g., [17] can only be used in video games due to its generative image-based methods.
Revised: There are some algorithms trying to solve RL tasks, but they are applied in image-based video games instead of robotic tasks, e.g., [17] can only be used in video games due to its generative image-based methods.

Original: [1] and [18] evaluates their methods only on Atari video games instead of robotic tasks.
Revised: [1] and [18] evaluates their methods only on Atari video games instead of robotic tasks.

Original: Some work on lifelong learning is applied in the robotics domain, but they choose preliminary tasks and they only provide little insight on how the state-of-the-art lifelong learning algorithms perform in robotic tasks.
Revised: Some work on lifelong learning is applied in the robotics domain, but they choose preliminary tasks and they only provide little insight on how the state-of-the-art lifelong learning algorithms perform in robotic tasks.

Original: For example, [19] is implemented to real robots, but their method can be only used in locomotion tasks with a limited control space.
Revised: For example, [19] is implemented to real robots, but their method can be only used in locomotion tasks with a limited control space.

Original: [20]evaluated lifelong learning on reinforcement tasks for robotics.
Revised: [20]evaluated lifelong learning on reinforcement tasks for robotics.

Original: It is the most related work in reinforcement lifelong learning.
Revised: It is the most related work in reinforcement lifelong learning.

Original: They modify the method mentioned in [15] to make it compatible with lifelong settings.
Revised: They modify the method mentioned in [15] to make it compatible with lifelong settings.

Original: Though  good performance, their method is only evaluated on up to three tasks in sequence.
Revised: Though having good performances, their method is only evaluated on up to three tasks in sequence.

Original: [21] presents an algorithm which demonstrates exceeding performance in lifelong learning and multi-task learning.
Revised: [21] presents an algorithm which demonstrates exceeding performance in lifelong learning and multi-task learning.

Original: However, it focuses on how previous learning experiences would help boost the learning of a new related task, while discusses little about catastrophic forgetting, which is the major concern in lifelong learning.
Revised: However, it focuses on how previous learning experiences would help boost the learning of a new related task, while discusses little about catastrophic forgetting, which is the major concern in lifelong learning.

Original: In conclusion, the lifelong learning problem, especially catastrophic forgetting, of reinforcement learning in robotics is poorly studied.
Revised: In conclusion, the lifelong learning problem, especially catastrophic forgetting, of reinforcement learning in robotics is poorly studied.

Original: An overall investigation about how the state-of-the-art lifelong learning methods perform on robotic reinforcement learning tasks is needed, which it the motivation of our work to modify traditional lifelong learning algorithms and compare their performance in RL.
Revised: An overall investigation about how the state-of-the-art lifelong learning methods perform on robotic reinforcement learning tasks is needed, which it the motivation of our work to modify traditional lifelong learning algorithms and compare their performance in RL.

Original:  Preliminaries 3.1 Markov Decision Process
Revised:  Preliminaries 3.1 Markov Decision Process

Original: We formulate our the problem as a Markov Decision Process (MDP).
Revised: We formulate our the problem as a Markov Decision Process (MDP).

Original: We use s kt ∈ R d s to denote the state of the agent at time step t in task k .
Revised: We use s kt ∈ R d s to denote the state of the agent at time step t in task k .

Original: The action vector a kt ∈ R d a are sampled from the policy π θ ( a kt ) , where θ denotes the policy parameters.
Revised: The action vector a kt ∈ R d a are sampled from the policy π θ ( a kt ) , where θ denotes the policy parameters.

Original: p ( s kt +1 | s kt , a kt ) denotes the probability of transforming from the state s kt to s kt +1 by executing the action a kt .
Revised: p ( s kt +1 | s kt , a kt ) denotes the probability of transforming from the state s kt to s kt +1 by executing the action a kt .

Original: The agent will receives a reward r ( s kt , a kt , s kt +1 ) at each step.
Revised: The agent will receives a reward r ( s kt , a kt , s kt +1 ) at each step.

Original: Our goal is to maximizing the sum of the discounted reward for all the tasks E π
Revised: Our goal is to maximizing the sum of the discounted reward for all the tasks E π

Original: θ
Revised: θ

Original: [ (cid:80) Kk =1 (cid:80) ∞ t =

Revised: [ (cid:80) Kk =1 (cid:80) ∞ t =

Original: γ t r ( s kt , a kt , s kt +1 )]
Revised: γ t r ( s kt , a kt , s kt +1 )]

Original: 3.2 Actor-Critic Framework
Revised: 3.2 Actor-Critic Framework

Original: Actor-Critic-based algorithms compose a majority of RL algorithms.
Revised: Actor-Critic-based algorithms compose a majority of RL algorithms.

Original: [7, 22, 23] all incorporate the framework.
Revised: [7, 22, 23] all incorporate the framework.

Original: In traditional supervised learning problems, the loss for each trial can be computed by comparing the outputs with the ground-truth labels.
Revised: In traditional supervised learning problems, the loss for each trial can be computed by comparing the outputs with the ground-truth labels.

Original: However, in most reinforcement learning cases, such labels do not exist.
Revised: However, in most reinforcement learning cases, such labels do not exist.

Original: Instead, in the Actor-Critic framework, a Critic is trained simultaneously with an Actor to evaluate the action the Actor chooses.
Revised: Instead, in the Actor-Critic framework, a Critic is trained simultaneously with an Actor to evaluate the action the Actor chooses.

Original: The Critic loss is usually deﬁned by the mean square Bellman error.
Revised: The Critic loss is usually deﬁned by the mean square Bellman error.

Original: [Equation]
Revised: [Equation]

Original: where Q denotes the Critic network with parameters φ , d kt denotes whether the state s kt +1 is terminal state in k -th task.
Revised: where Q denotes the Critic network with parameters φ , d kt denotes whether the state s kt +1 is terminal state in k -th task.

Original: The Actor loss is deﬁned by: L ( θ ) =
Revised: The Actor loss is deﬁned by: L ( θ ) =

Original: − Q φ ( s k t , π θ ( s k t )) .
Revised: − Q φ ( s kt , π θ ( s kt )) .

Original:  Reinforcement Lifelong Learning Framework
Revised:  Reinforcement Lifelong Learning Framework

Original: In this section, we propose the framework to modify traditional supervised lifelong learning algorithms to reinforcement learning.
Revised: In this section, we propose the framework to modify traditional supervised lifelong learning algorithms to reinforcement learning.

Original: We purposefully design this framework to be as simple as possible.
Revised: We purposefully design this framework to be as simple as possible.

Original: Our main contribution lies in the evaluation of lifelong learning algorithms.
Revised: Our main contribution lies in the evaluation of lifelong learning algorithms.

Original: We want to minimize the inﬂuence of the additional network structure on the evaluation results.
Revised: We want to minimize the inﬂuence of the additional network structure on the evaluation results.

Original: Traditional lifelong learning algorithms cannot be directly applied to RL because of the following two reasons: (a) the loss in supervised learning is clearly deﬁned, which is the error between predictions and labels.
Revised: Traditional lifelong learning algorithms cannot be directly applied to RL because of the following two reasons: (a) the loss in supervised learning is clearly deﬁned, which is the error between predictions and labels.

Original: However, it is not straightforward in RL.
Revised: However, it is not straightforward in RL.

Original: A critic network is needed for the evaluation of the policy network.
Revised: A critic network is needed for the evaluation of the policy network.

Original: (b) It would be hard to indicate which task the robot is performing, while in supervised learning, e.g., in image classiﬁcation, the network usually does not need to know the task ID since the images themselves have sufﬁcient information to indicate which task they are within.
Revised: (b) It would be hard to indicate which task the robot is performing, while in supervised learning, e.g., in image classiﬁcation, the network usually does not need to know the task ID since the images themselves have sufﬁcient information to indicate which task they are within.

Original: In order to solve these two problems, we modify the traditional lifelong learning algorithms in the following ways.
Revised: In order to solve these two problems, we modify the traditional lifelong learning algorithms in the following ways.

Original: 4.1 Actor-Critic Framework in the Lifelong Learning Algorithms
Revised: 4.1 Actor-Critic Framework in the Lifelong Learning Algorithms

Original: We adopt the Actor-Critic algorithm into our lifelong learning framework to have a parametric evaluation of the actor policy.
Revised: We adopt the Actor-Critic algorithm into our lifelong learning framework to have a parametric evaluation of the actor policy.

Original: we replace the loss (usually MSE or cross entropy) in traditional lifelong learning algorithms with critic loss.
Revised: we replace the loss (usually MSE or cross entropy) in traditional lifelong learning algorithms with critic loss.

Original: In addition, since in real-world applications, critics are usually discarded and only the actor (policy network) is used, we only consider the continual learning properties of the actor while we use traditional deep RL algorithms to learn task-speciﬁc critics.
Revised: In addition, since in real-world applications, critics are usually discarded and only the actor (policy network) is used, we only consider the continual learning properties of the actor while we use traditional deep RL algorithms to learn task-speciﬁc critics.

Original: However, it is also straightforward to expand the lifelong learning algorithms for the critics.
Revised: However, it is also straightforward to expand the lifelong learning algorithms for the critics.

Original: 4.2 Task-Speciﬁc Sub-Network
Revised: 4.2 Task-Speciﬁc Sub-Network

Original: The observation spaces in RL tasks could be identical.
Revised: The observation spaces in RL tasks could be identical.

Original: In such cases, the policy network will have to distinguish different tasks by task IDs.
Revised: In such cases, the policy network will have to distinguish different tasks by task IDs.

Original: It would be challenging if the task ID is directly input to the network for learning.
Revised: It would be challenging if the task ID is directly input to the network for learning.

Original: Instead, we add task-speciﬁc layers into the policy network if lifelong learning algorithms themselves do not include such task-speciﬁc parts.
Revised: Instead, we add task-speciﬁc layers into the policy network if lifelong learning algorithms themselves do not include such task-speciﬁc parts.

Original: Formally, given k tasks, the parameters of the policy network π θ can be decomposed into common parts and speciﬁc parts: θ = { θ c } ∪ Ki =1 { θ is } .
Revised: Formally, given k tasks, the parameters of the policy network π θ can be decomposed into common parts and speciﬁc parts: θ = { θ c } ∪ Ki =1 { θ is } .

Original: For task i , the action a it can be given by: a it = π θ ( a it ) =
Revised: For task i , the action a it can be given by: a it = π θ ( a it ) =

Original: π θ is ( π θ c
Revised: π θ is ( π θ c

Original: ( s it )) .
Revised: ( s it )) .

Original: Different sub-networks are chosen for different tasks according to task IDs.
Revised: Different sub-networks are chosen for different tasks according to task IDs.

Original: In our experiments, the task-speciﬁc sub-networks are only a single-layer afﬁne transformation.
Revised: In our experiments, the task-speciﬁc sub-networks are only a single-layer afﬁne transformation.

Original: If sub-networks with deeper structures are implemented, the shared parts could be simply an identity transformation while the policy totally relies on the sub-network, which is basically training a policy network for each task.
Revised: If sub-networks with deeper structures are implemented, the shared parts could be simply an identity transformation while the policy totally relies on the sub-network, which is basically training a policy network for each task.

Original: Details are shown in the appendix.
Revised: Details are shown in the appendix.

Original:  Lifelong Manipulation Benchmarks
Revised:  Lifelong Manipulation Benchmarks

Original: In this section, we introduce the lifelong manipulation benchmarks.
Revised: In this section, we introduce the lifelong manipulation benchmarks.

Original: Our benchmarks adapt from
Revised: Our benchmarks adapt from

Original: ROBEL D’Claw[24], a mujoco-based robotic manipulation platform.
Revised: ROBEL D’Claw[24], a mujoco-based robotic manipulation platform.

Original: We modify the tasks and get ten different manipulation tasks.
Revised: We modify the tasks and get ten different manipulation tasks.

Original: The benchmarks consist of a robotic hand with three independent ﬁngers.
Revised: The benchmarks consist of a robotic hand with three independent ﬁngers.

Original: Each ﬁnger has three degrees of freedom.
Revised: Each ﬁnger has three degrees of freedom.

Original: In our task, the robotic hand needs to turn the valve below 180 degrees.
Revised: In our task, the robotic hand needs to turn the valve below 180 degrees.

Original: The observation space of the benchmarks consists of joint angles, joint velocities, and the valve angle.
Revised: The observation space of the benchmarks consists of joint angles, joint velocities, and the valve angle.

Original: The action space of the robot consists of robot joint angles.
Revised: The action space of the robot consists of robot joint angles.

Original: The reward function is designed based on robot conﬁgurations and the valve position to ensure a safe manipulation.
Revised: The reward function is designed based on robot conﬁgurations and the valve position to ensure a safe manipulation.

Original: Each task only differs in the valve shape.
Revised: Each task only differs in the valve shape.

Original: Though each task seems to be similar, a policy trained on a single task cannot achieve high rewards on the other tasks.
Revised: Though each task seems to be similar, a policy trained on a single task cannot achieve high rewards on the other tasks.

Original: All the tasks are designed to have dense rewards for a faster learning procedure.
Revised: All the tasks are designed to have dense rewards for a faster learning procedure.

Original: Note that the absolute reward value itself does not have much meaning.
Revised: Note that the absolute reward value itself does not have much meaning.

Original: A higher or a lower reward only indicates a better or worse performance.
Revised: A higher or a lower reward only indicates a better or worse performance.

Original: Details of the reward function can be seen in [24].
Revised: Details of the reward function can be seen in [24].

Original: Though Meta-World[25] provides a multi-task benchmark and it is likely to be applied in lifelong learning, Their tasks are far more challenging than ours.
Revised: Though Meta-World[25] provides a multi-task benchmark and it is likely to be applied in lifelong learning, Their tasks are far more challenging than ours.

Original: Tasks in Meta-World could be difﬁcult in lifelong learning settings.
Revised: Tasks in Meta-World could be difﬁcult in lifelong learning settings.

Original: Our benchmarks are used for different levels of evaluations.
Revised: Our benchmarks are used for different levels of evaluations.

Original: One of the important challenges in reinforcement lifelong learning lies in the training time.
Revised: One of the important challenges in reinforcement lifelong learning lies in the training time.

Original: RL policies are notably hard to train.
Revised: RL policies are notably hard to train.

Original: This problem will be aggravated in the lifelong learning scenarios since each task should be trained one after another by deﬁnition.
Revised: This problem will be aggravated in the lifelong learning scenarios since each task should be trained one after another by deﬁnition.

Original: In our benchmark, each task is stable and easy to train.
Revised: In our benchmark, each task is stable and easy to train.

Original: In our experiments, each policy can converge after 80,000 steps of updates using SAC.
Revised: In our experiments, each policy can converge after 80,000 steps of updates using SAC.

Original: The details of the benchmark could be seen in Fig.
Revised: The details of the benchmark could be seen in Fig.

Original:  Experiments 6.1 Evaluation Metrics
Revised:  Experiments 6.1 Evaluation Metrics

Original: We will mainly focus on the forgetting of lifelong learning.
Revised: We will mainly focus on the forgetting of lifelong learning.

Original: We evaluate each algorithm using the average performance.
Revised: We evaluate each algorithm using the average performance.

Original: It is slightly modiﬁed from the average accuracy, a standard lifelong learning evaluation metric used in [26][27] and [28].:
Revised: It is slightly modiﬁed from the average accuracy, a standard lifelong learning evaluation metric used in [26][27] and [28].:

Original: Average Performance : Let p i,j be the cumulative reward of the task j after the model ﬁnishes learning task i .
Revised: Average Performance : Let p i,j be the cumulative reward of the task j after the model ﬁnishes learning task i .

Original: Then the average performance P is deﬁned as: P = 1 K (cid:80)
Revised: Then the average performance P is deﬁned as: P = 1 K (cid:80)

Original: Kj =1 p K,j , where K denotes the total number of tasks.
Revised: Kj =1 p K,j , where K denotes the total number of tasks.

Original: When evaluating, since we use the cumulative reward to evaluate policies, the time steps for each episode are set to be constant for fairness, no matter whether the task has already converged.
Revised: When evaluating, since we use the cumulative reward to evaluate policies, the time steps for each episode are set to be constant for fairness, no matter whether the task has already converged.

Original: 6.2 Taxonomy of Lifelong Learning
Revised: 6.2 Taxonomy of Lifelong Learning

Original: Recent lifelong learning algorithms can be classiﬁed into the following four categories:
Revised: Recent lifelong learning algorithms can be classiﬁed into the following four categories:

Original: 6.2.1 Regularization-Based Methods
Revised: 6.2.1 Regularization-Based Methods

Original: Regularization-based methods add an additional regularization term in the loss function to ensure the learned network in the new task will not forget important information for the previous tasks.
Revised: Regularization-based methods add an additional regularization term in the loss function to ensure the learned network in the new task will not forget important information for the previous tasks.

Original: The most intuitive method would add an MSE loss to minimize the parameter changes between the current task and previous tasks.
Revised: The most intuitive method would add an MSE loss to minimize the parameter changes between the current task and previous tasks.

Original: Instead, [1] proposed Elastic Weight Consolidation(EWC).
Revised: Instead, [1] proposed Elastic Weight Consolidation(EWC).

Original: It incorporates the Fisher information matrix to compute the importance of network parameters and tries to keep the important ones unchanged.
Revised: It incorporates the Fisher information matrix to compute the importance of network parameters and tries to keep the important ones unchanged.

Original: The most important advantage of regularization-based methods is that the model size will not change when the number of tasks increases.
Revised: The most important advantage of regularization-based methods is that the model size will not change when the number of tasks increases.

Original: 6.2.2 Memory-Based Methods
Revised: 6.2.2 Memory-Based Methods

Original: Memory-based methods would use a tiny replay memory to store several samples from previously learned tasks and replay when learning new tasks.
Revised: Memory-based methods would use a tiny replay memory to store several samples from previously learned tasks and replay when learning new tasks.

Original: The key idea of memory-based methods is trying to transform lifelong learning problems into classic deep learning tasks.
Revised: The key idea of memory-based methods is trying to transform lifelong learning problems into classic deep learning tasks.

Original: Episodic Replay(ER)[28] combines samples from previous tasks with current tasks when computing the loss for the current task.
Revised: Episodic Replay(ER)[28] combines samples from previous tasks with current tasks when computing the loss for the current task.

Original: [29] is an updated version of [28], which adds weight to learn the importance of the current task and previous task.
Revised: [29] is an updated version of [28], which adds weight to learn the importance of the current task and previous task.

Original: 6.2.3 Gradient-Constraint Methods
Revised: 6.2.3 Gradient-Constraint Methods

Original: Gradient-constraint methods modify the gradient to ensure that the update for the current task would have minimal inﬂuence on previous tasks.
Revised: Gradient-constraint methods modify the gradient to ensure that the update for the current task would have minimal inﬂuence on previous tasks.

Original: Gradient Episodic Memory(GEM)[26] projects the gradient into spaces to ensure the update for the current task will not decrease the performance of previous tasks.
Revised: Gradient Episodic Memory(GEM)[26] projects the gradient into spaces to ensure the update for the current task will not decrease the performance of previous tasks.

Original: [27] proposed A-GEM and optimize the computation efﬁciency of GEM.
Revised: [27] proposed A-GEM and optimize the computation efﬁciency of GEM.

Original: Orthogonal Gradient Descent(OGD)[30] believes moving orthogonally to the gradients of previous tasks would have the smallest inﬂuence on them.
Revised: Orthogonal Gradient Descent(OGD)[30] believes moving orthogonally to the gradients of previous tasks would have the smallest inﬂuence on them.

Original: Therefore, it projects the gradients for the current tasks into the orthogonal space of previous tasks.
Revised: Therefore, it projects the gradients for the current tasks into the orthogonal space of previous tasks.

Original: Continual Learning in Orthogonal Subspaces(CLOS)[31] designs a framework to ensure that the gradients are orthogonal by nature.
Revised: Continual Learning in Orthogonal Subspaces(CLOS)[31] designs a framework to ensure that the gradients are orthogonal by nature.

Original: It combines gradient-constraint methods and memory-based methods for a better performance.
Revised: It combines gradient-constraint methods and memory-based methods for a better performance.

Original: 6.2.4 Expansion-Based Methods
Revised: 6.2.4 Expansion-Based Methods

Original: Expansion-based methods add a new sub-module when learning a new task.
Revised: Expansion-based methods add a new sub-module when learning a new task.

Original: Thus, the size of the model increases with the learning procedure.
Revised: Thus, the size of the model increases with the learning procedure.

Original: Additive Parameter Decomposition(APD)[32] includes a shared parameter space with task-speciﬁc sub-networks.
Revised: Additive Parameter Decomposition(APD)[32] includes a shared parameter space with task-speciﬁc sub-networks.

Original: Learning a new task would add a new sub-network and the algorithm would also constrain the changes in the shared parameters.
Revised: Learning a new task would add a new sub-network and the algorithm would also constrain the changes in the shared parameters.

Original: In order to have an overall comparison of lifelong learning algorithms on RL, we choose one or two algorithms from the categories mentioned above.
Revised: In order to have an overall comparison of lifelong learning algorithms on RL, we choose one or two algorithms from the categories mentioned above.

Original: Oracle means that each task is trained with an individual policy.
Revised: Oracle means that each task is trained with an individual policy.

Original: Oracle does not consider the lifelong learning problem.
Revised: Oracle does not consider the lifelong learning problem.

Original: It is used to indicate the best possible performance each algorithm can achieve.
Revised: It is used to indicate the best possible performance each algorithm can achieve.

Original: Random means using a random agent in the benchmark tasks.
Revised: Random means using a random agent in the benchmark tasks.

Original: It is used to indicate the worst possible performance for each algorithm.
Revised: It is used to indicate the worst possible performance for each algorithm.

Original: SAC [7] means using the original Soft-Actor-Critic algorithm trained continuously from one task to another.
Revised: SAC [7] means using the original Soft-Actor-Critic algorithm trained continuously from one task to another.

Original: However, for the fairness of comparison, we also modify the actor of SAC into shared parts and task-speciﬁc parts.
Revised: However, for the fairness of comparison, we also modify the actor of SAC into shared parts and task-speciﬁc parts.

Original: It is designed to demonstrate the performance of the state-of-the-art RL algorithms without lifelong learning.
Revised: It is designed to demonstrate the performance of the state-of-the-art RL algorithms without lifelong learning.

Original: EWC [1] and L2 are representatives of regularization-based methods.
Revised: EWC [1] and L2 are representatives of regularization-based methods.

Original: L2 only introduces an MSE loss for the network differences between the current actor parameters and previous task parameters, as is mentioned in Sec.
Revised: L2 only introduces an MSE loss for the network differences between the current actor parameters and previous task parameters, as is mentioned in Sec.

Original: L2 is the most intuitive lifelong learning algorithm.
Revised: L2 is the most intuitive lifelong learning algorithm.

Original: AGEM [27] and GEM [26] and CLOS [31] are chosen as representatives for gradient-constraint methods.
Revised: AGEM [27] and GEM [26] and CLOS [31] are chosen as representatives for gradient-constraint methods.

Original: ER [28] is chosen for memory-based methods.
Revised: ER [28] is chosen for memory-based methods.

Original: APD [32] is chosen for expansion-based methods.
Revised: APD [32] is chosen for expansion-based methods.

Original: In addition, APD and CLOS are the state-of-the-art ones to demonstrate the performance of the frontier research on lifelong learning.
Revised: In addition, APD and CLOS are the state-of-the-art ones to demonstrate the performance of the frontier research on lifelong learning.

Original: 6.3 Results
Revised: 6.3 Results

Original: We ﬁrst show the performances on each task after ﬁnishing learning the last task.
Revised: We ﬁrst show the performances on each task after ﬁnishing learning the last task.

Original: Each algorithm is trained and updated with nearly the same steps for a fair comparison.
Revised: Each algorithm is trained and updated with nearly the same steps for a fair comparison.

Original: The environments for the previous tasks are not allowed to be re-accessed after ﬁnishing the training.
Revised: The environments for the previous tasks are not allowed to be re-accessed after ﬁnishing the training.

Original: All the algorithms are trained with the same task order.
Revised: All the algorithms are trained with the same task order.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: The average performance is shown in Table 1.
Revised: The average performance is shown in Table 1.

Original: In Fig.
Revised: In Fig.

Original: 2, SAC only achieves a high reward on the last task.
Revised: 2, SAC only achieves a high reward on the last task.

Original: It indicates that traditional RL cannot solve the lifelong tasks designed in our benchmark tasks.
Revised: It indicates that traditional RL cannot solve the lifelong tasks designed in our benchmark tasks.

Original: Previous tasks still have a slightly higher performance than a random agent, which suggests each task is not totally independent from others.
Revised: Previous tasks still have a slightly higher performance than a random agent, which suggests each task is not totally independent from others.

Original: L2 achieves worse results than SAC.
Revised: L2 achieves worse results than SAC.

Original: It indicates that improper constraints of lifelong learning, e.g., regularization, are not only detrimental to memorizing previous tasks but will impede the learning of the current task.
Revised: It indicates that improper constraints of lifelong learning, e.g., regularization, are not only detrimental to memorizing previous tasks but will impede the learning of the current task.

Original: In comparison, EWC achieves better results.
Revised: In comparison, EWC achieves better results.

Original: However, it has a much worse forgetting performance compared to its applications in supervised learning.
Revised: However, it has a much worse forgetting performance compared to its applications in supervised learning.

Original: It could result from the sensitiveness of manipulation tasks mentioned in Section 1.
Revised: It could result from the sensitiveness of manipulation tasks mentioned in Section 1.

Original: GEM has better performances than
Revised: GEM has better performances than

Original: AGEM.
Revised: AGEM.

Original: Compared to AGEM, GEM samples more data from previous tasks when updating.
Revised: Compared to AGEM, GEM samples more data from previous tasks when updating.

Original: It could lead to a more conservative update, which is important in the sensitive RL environment.
Revised: It could lead to a more conservative update, which is important in the sensitive RL environment.

Original: Though simple and intuitive, ER achieves a good performance.
Revised: Though simple and intuitive, ER achieves a good performance.

Original: It is worth noting that in RL, we do not need to design additional replay memories since RL algorithms will need to use replay memories themselves.
Revised: It is worth noting that in RL, we do not need to design additional replay memories since RL algorithms will need to use replay memories themselves.

Original: CLOS includes memory-based methods into gradient-based methods by replaying previous tasks when training.
Revised: CLOS includes memory-based methods into gradient-based methods by replaying previous tasks when training.

Original: It achieves better performances than gradient-based ones, which suggests that memory-based methods could help improve the performance of other lifelong algorithms.
Revised: It achieves better performances than gradient-based ones, which suggests that memory-based methods could help improve the performance of other lifelong algorithms.

Original: Though
Revised: Though

Original: APD achieves a good result, expansion-based methods will add a sub-module when learning a new task in practice.
Revised: APD achieves a good result, expansion-based methods will add a sub-module when learning a new task in practice.

Original: The size of the added sub-module could be as large as training a new policy from scratch, which degrades the effect of lifelong learning.
Revised: The size of the added sub-module could be as large as training a new policy from scratch, which degrades the effect of lifelong learning.

Original: [Table]
Revised: [Table]

Original: To better demonstrate the forgetting details of each algorithm, we provide ﬁgures of cumulative rewards during the training procedures.
Revised: To better demonstrate the forgetting details of each algorithm, we provide ﬁgures of cumulative rewards during the training procedures.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: The x-axis denotes training steps.
Revised: The x-axis denotes training steps.

Original: The y-axis denotes cumulative rewards.
Revised: The y-axis denotes cumulative rewards.

Original: Each column shows performances of a policy for all
Revised: Each column shows performances of a policy for all

Original: 10 Task ID

Revised: 10 Task ID

Original: [Figure]
Revised: [Figure]

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomSAC (a) SAC    10 Task ID

Revised: OracleRandomSAC (a) SAC    10 Task ID

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomL (b) L     10 Task ID

Revised: OracleRandomL (b) L     10 Task ID

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomEWC (c) EWC    10 Task ID

Revised: OracleRandomEWC (c) EWC    10 Task ID

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomAGEM (d) AGEM    10 Task ID
Revised: OracleRandomAGEM (d) AGEM    10 Task ID

Original: [Figure]
Revised: [Figure]

Original: R e w a r d
Revised: R e w a r d

Original: [Figure]
Revised: [Figure]

Original: 10 Task ID
Revised: 10 Task ID

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomER (f) ER    10 Task ID

Revised: OracleRandomER (f) ER    10 Task ID

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomAPD (g) APD    10 Task ID

Revised: OracleRandomAPD (g) APD    10 Task ID

Original: R e w a r d
Revised: R e w a r d

Original: OracleRandomCLOS (h) CLOS the tasks, from before training a certain task to after training the last task.
Revised: OracleRandomCLOS (h) CLOS the tasks, from before training a certain task to after training the last task.

Original: It is used to demonstrate the inﬂuence of training other tasks on the current task.
Revised: It is used to demonstrate the inﬂuence of training other tasks on the current task.

Original: In Fig.
Revised: In Fig.

Original: the performance for each task in SAC only peaks when training it, which means it quickly forgets the policy for previous tasks.
Revised: the performance for each task in SAC only peaks when training it, which means it quickly forgets the policy for previous tasks.

Original: The rewards in ER, APD, CLOS will keep at a high level even after training a certain task.
Revised: The rewards in ER, APD, CLOS will keep at a high level even after training a certain task.

Original: CLOS combines several methods together and has the best performance among the algorithms mentioned above.
Revised: CLOS combines several methods together and has the best performance among the algorithms mentioned above.

Original: However, the training of CLOS is less stable than APD and ER.
Revised: However, the training of CLOS is less stable than APD and ER.

Original: Speciﬁcally, the performance will vary a lot during the training procedure.
Revised: Speciﬁcally, the performance will vary a lot during the training procedure.

Original: One has to save several policies and search for the best one when using CLOS.
Revised: One has to save several policies and search for the best one when using CLOS.

Original: [Figure]
Revised: [Figure]

Original: A reliable lifelong learning algorithm would not be inﬂuenced by the task orders.
Revised: A reliable lifelong learning algorithm would not be inﬂuenced by the task orders.

Original: We train each algorithm in three different task orders and evaluate the policies on all the ten tasks after ﬁnishing training the last task.
Revised: We train each algorithm in three different task orders and evaluate the policies on all the ten tasks after ﬁnishing training the last task.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: Task order experiments in Fig.
Revised: Task order experiments in Fig.

Original: 4 demonstrate that the algorithms with satisfying performance in the previous experiments are usually also robust to order changes.
Revised: 4 demonstrate that the algorithms with satisfying performance in the previous experiments are usually also robust to order changes.

Original: APD demonstrates the best robustness to task orders, since expanding the network makes it similar to training an independent policy for each task.
Revised: APD demonstrates the best robustness to task orders, since expanding the network makes it similar to training an independent policy for each task.

Original: [Figure]
Revised: [Figure]

Original:  Discussion
Revised:  Discussion

Original: In this section, we conclude several concise wake-away conclusions of our evaluations.
Revised: In this section, we conclude several concise wake-away conclusions of our evaluations.

Original: Firstly, RL tasks would intensify the weakness of lifelong learning algorithms.
Revised: Firstly, RL tasks would intensify the weakness of lifelong learning algorithms.

Original: EWC, GEM, and
Revised: EWC, GEM, and

Original: AGEM, which are notably effective in supervised lifelong learning, have a poor performance in RL.
Revised: AGEM, which are notably effective in supervised lifelong learning, have a poor performance in RL.

Original: Even some could have a good result on supervised learning, the performance could be dramatically degraded in RL.
Revised: Even some could have a good result on supervised learning, the performance could be dramatically degraded in RL.

Original: Speciﬁcally, lifelong learning in robotic manipulation tasks is far more sensitive than that in supervised learning.
Revised: Speciﬁcally, lifelong learning in robotic manipulation tasks is far more sensitive than that in supervised learning.

Original: In RL settings, the gaps between different lifelong learning methods are greater than that in supervised learning.
Revised: In RL settings, the gaps between different lifelong learning methods are greater than that in supervised learning.

Original: Secondly, expansion-based methods and memory-based methods have a good performance on lifelong RL methods.
Revised: Secondly, expansion-based methods and memory-based methods have a good performance on lifelong RL methods.

Original: Future work could consider incorporating these two methods as a part of its algorithm for a better performance.
Revised: Future work could consider incorporating these two methods as a part of its algorithm for a better performance.

Original: Regularization-based methods and gradient-based methods still suffer in RL tasks.
Revised: Regularization-based methods and gradient-based methods still suffer in RL tasks.

Original: But on the other hand, the model size is limited in these methods, which has great potential in real robot applications.
Revised: But on the other hand, the model size is limited in these methods, which has great potential in real robot applications.

Original: Thirdly, existing lifelong learning methods requires memorizing lots of data, which makes it impractical in robotic applications.
Revised: Thirdly, existing lifelong learning methods requires memorizing lots of data, which makes it impractical in robotic applications.

Original: They require memorizing samples or models from previous tasks.
Revised: They require memorizing samples or models from previous tasks.

Original: However, for applications on real robots, it will not be possible for the robot to remember too many samples or will not allow a substantial increase in the model size.
Revised: However, for applications on real robots, it will not be possible for the robot to remember too many samples or will not allow a substantial increase in the model size.

Original: Existing lifelong learning methods still do not have a perfect solution to it.
Revised: Existing lifelong learning methods still do not have a perfect solution to it.

Original: We also point out that a more comprehensive evaluation metrics in reinforcement lifelong learning is needed .
Revised: We also point out that a more comprehensive evaluation metrics in reinforcement lifelong learning is needed .

Original: Unlike supervised learning, even if the cumulative reward of a task could decrease greatly, it does not really mean the policy forgets greatly about the task.
Revised: Unlike supervised learning, even if the cumulative reward of a task could decrease greatly, it does not really mean the policy forgets greatly about the task.

Original: Since RL tasks could be highly sensitive and the policy might not be robust, a slight forgetting could lead to a complete failure of the task.
Revised: Since RL tasks could be highly sensitive and the policy might not be robust, a slight forgetting could lead to a complete failure of the task.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we focus on the problem of how the state-of-the-art lifelong learning algorithms perform on RL manipulation tasks.
Revised: In this paper, we focus on the problem of how the state-of-the-art lifelong learning algorithms perform on RL manipulation tasks.

Original: We design a framework to modify traditional lifelong learning algorithms used in supervised learning into the ones in RL.
Revised: We design a framework to modify traditional lifelong learning algorithms used in supervised learning into the ones in RL.

Original: We also develop a manipulation benchmark for evaluation.
Revised: We also develop a manipulation benchmark for evaluation.

Original: We test seven lifelong learning algorithms with three baselines on our benchmark.
Revised: We test seven lifelong learning algorithms with three baselines on our benchmark.

Original: According to our experiments, we point out the sensitiveness challenge of lifelong RL tasks compared to lifelong supervised learning tasks.
Revised: According to our experiments, we point out the sensitiveness challenge of lifelong RL tasks compared to lifelong supervised learning tasks.

Original: We also summarize the potentials of applying different kinds of lifelong learning tasks in RL.
Revised: We also summarize the potentials of applying different kinds of lifelong learning tasks in RL.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported in part by the National Natural Science Foundation of China under Grant and the Joint Fund of Science & Technology Department of Liaoning Province and
Revised: This work was supported in part by the National Natural Science Foundation of China under Grant and the Joint Fund of Science & Technology Department of Liaoning Province and

Original: State Key Laboratory of Robotics, China (2020-KF-22-06)
Revised: State Key Laboratory of Robotics, China (2020-KF-22-06)