Original: SparCL: Sparse Continual Learning on the Edge
Revised: SparCL: Sparse Continual Learning on the Edge

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a newtask.
Revised: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a newtask.

Original: However, the training efﬁciency of a CL system is under-investigated, whichlimits the real-world application of CL systems under resource-limited scenarios.
Revised: However, the training efﬁciency of a CL system is under-investigated, whichlimits the real-world application of CL systems under resource-limited scenarios.

Original: In this work, we propose a novel framework called Sparse Continual Learning(SparCL), which is the ﬁrst study that leverages sparsity to enable cost-effectivecontinual learning on edge devices.
Revised: In this work, we propose a novel framework called Sparse Continual Learning(SparCL), which is the ﬁrst study that leverages sparsity to enable cost-effectivecontinual learning on edge devices.

Original: SparCL achieves both training accelerationand accuracy preservation through the synergy of three aspects: weight sparsity ,data efﬁciency , and gradient sparsity .
Revised: SparCL achieves both training accelerationand accuracy preservation through the synergy of three aspects: weight sparsity ,data efﬁciency , and gradient sparsity .

Original: Speciﬁcally, we propose task-aware dynamicmasking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamicgradient masking (DGM) to sparsify the gradient updates.
Revised: Speciﬁcally, we propose task-aware dynamicmasking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamicgradient masking (DGM) to sparsify the gradient updates.

Original: Each of them not onlyimproves efﬁciency, but also further mitigates catastrophic forgetting.
Revised: Each of them not onlyimproves efﬁciency, but also further mitigates catastrophic forgetting.

Original: SparCLconsistently improves the training efﬁciency of existing state-of-the-art (SOTA)
Revised: SparCLconsistently improves the training efﬁciency of existing state-of-the-art (SOTA)

Original: CLmethods by at most 23 ⇥ less training FLOPs, and, surprisingly, further improvesthe SOTA accuracy by at most 1 . 7% .
Revised: CLmethods by at most 23 ⇥ less training FLOPs, and, surprisingly, further improvesthe SOTA accuracy by at most 1 . 7% .

Original: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting inboth efﬁciency and accuracy.
Revised: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting inboth efﬁciency and accuracy.

Original: We also evaluate the effectiveness of SparCL on areal mobile phone, further indicating the practical potential of our method.
Revised: We also evaluate the effectiveness of SparCL on areal mobile phone, further indicating the practical potential of our method.

Original: Sourcecode will be released.
Revised: Sourcecode will be released.

Original:  Introduction
Revised:  Introduction

Original: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledgefrom a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [28].
Revised: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledgefrom a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [23].

Original: Recent methods mostly focus on addressing the catastrophic forgetting [39] problem – learningmodel tends to suffer performance deterioration on previously seen tasks.
Revised: Recent methods mostly focus on addressing the catastrophic forgetting [32] problem – learningmodel tends to suffer performance deterioration on previously seen tasks.

Original: However, in the real world,when the CL applications are deployed in resource-limited platforms [44] such as edge devices,the learning efﬁciency, w.r.t.
Revised: However, in the real world,when the CL applications are deployed in resource-limited platforms [36] such as edge devices,the learning efﬁciency, w.r.t.

Original: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.
Revised: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.

Original: Existing CL methods can be categorized into regularization-based [2, 28, 33, 62], rehearsal-based [8,12, 46, 56], and architecture-based [27, 38, 48, 53, 54].
Revised: Existing CL methods can be categorized into regularization-based [2, 23, 27, 53], rehearsal-based [7,9, 38, 48], and architecture-based [22, 31, 40, 45, 46].

Original: Both regularization- and rehearsal-basedmethods directly train a dense model, which might even be over-parametrized for the union of alltasks [18, 35]; Though several architecture-based methods [47, 53, 58] start with a sparse sub-networkfrom the dense model, they still grow the model size progressively to learn emerging tasks.
Revised: Both regularization- and rehearsal-basedmethods directly train a dense model, which might even be over-parametrized for the union of alltasks [15, 28]; Though several architecture-based methods [39, 45, 50] start with a sparse sub-networkfrom the dense model, they still grow the model size progressively to learn emerging tasks.

Original: Theaforementioned methods, although striving for greater performance with less forgetting, still introducesigniﬁcant memory and computation overhead during the whole CL process.
Revised: Theaforementioned methods, although striving for greater performance with less forgetting, still introducesigniﬁcant memory and computation overhead during the whole CL process.

Original: Recently, another stream of work, sparse training [4, 19, 31] has emerged as a new training trend toachieve training acceleration, which embraces the promising training-on-the-edge paradigm.
Revised: Recently, another stream of work, sparse training [4, 16, 26] has emerged as a new training trend toachieve training acceleration, which embraces the promising training-on-the-edge paradigm.

Original: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity, under the traditional i.i.d.
Revised: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity, under the traditional i.i.d.

Original: learning setting.
Revised: learning setting.

Original: Inspired by these sparse training methods, we naturallythink about introducing sparse training to the ﬁeld of CL.
Revised: Inspired by these sparse training methods, we naturallythink about introducing sparse training to the ﬁeld of CL.

Original: A straightforward idea is to directly combineexisting sparse training methods, such as SNIP [31], RigL [19], with a rehearsal buffer under the
Revised: A straightforward idea is to directly combineexisting sparse training methods, such as SNIP [26], RigL [16], with a rehearsal buffer under the

Original: CL setting.
Revised: CL setting.

Original: However, these methods fail to consider key challenges in CL to mitigate catastrophicforgetting, for example, properly handling transition between tasks.
Revised: However, these methods fail to consider key challenges in CL to mitigate catastrophicforgetting, for example, properly handling transition between tasks.

Original: As a result, these sparse trainingmethods, though enhancing training efﬁciency, cause signiﬁcant accuracy drop (see Section 5.2).
Revised: As a result, these sparse trainingmethods, though enhancing training efﬁciency, cause signiﬁcant accuracy drop (see Section 5.2).

Original: Thus, we would like to explore a general strategy, which is orthogonal to existing CL methods, thatnot only leverages the idea of sparse training for efﬁciency, but also addresses key challenges in CLto preserve (or even improve) accuracy.
Revised: Thus, we would like to explore a general strategy, which is orthogonal to existing CL methods, thatnot only leverages the idea of sparse training for efﬁciency, but also addresses key challenges in CLto preserve (or even improve) accuracy.

Original: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effectivecontinual learning, aiming at enabling practical CL on edge devices.
Revised: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effectivecontinual learning, aiming at enabling practical CL on edge devices.

Original: As shown in Figure 1 (left),
Revised: As shown in Figure 1 (left),

Original: SparCL achieves both learning acceleration and accuracy preservation through the synergy of threeaspects: weight sparsity , data efﬁciency , and gradient sparsity .
Revised: SparCL achieves both learning acceleration and accuracy preservation through the synergy of threeaspects: weight sparsity , data efﬁciency , and gradient sparsity .

Original: Speciﬁcally, to maintain a smalldynamic sparse network during the whole CL process, we develop a novel task-aware dynamicmasking (TDM) strategy to keep only important weights for both the current and past tasks, withspecial consideration during task transitions.
Revised: Speciﬁcally, to maintain a smalldynamic sparse network during the whole CL process, we develop a novel task-aware dynamicmasking (TDM) strategy to keep only important weights for both the current and past tasks, withspecial consideration during task transitions.

Original: Moreover, we propose a dynamic data removal (DDR)scheme, which progressively removes “easy-to-learn” examples from training iterations, whichfurther accelerates the training process and also improves accuracy of CL by balancing current andpast data and keeping more informative samples in the buffer.
Revised: Moreover, we propose a dynamic data removal (DDR)scheme, which progressively removes “easy-to-learn” examples from training iterations, whichfurther accelerates the training process and also improves accuracy of CL by balancing current andpast data and keeping more informative samples in the buffer.

Original: Finally, we provide an additionaldynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efﬁciencyand knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.
Revised: Finally, we provide an additionaldynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efﬁciencyand knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.

Original: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and signiﬁcantlyimproves efﬁciency over DER++ [8], one of the SOTA CL methods, under different sparsity ratios.
Revised: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and signiﬁcantlyimproves efﬁciency over DER [7], one of the SOTA CL methods, under different sparsity ratios.

Original: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, andefﬁcient under practical scenarios.
Revised: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, andefﬁcient under practical scenarios.

Original: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.
Revised: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.

Original: We show that SparCL works collaboratively withexisting CL methods, greatly accelerates the learning process under different sparsity ratios, andeven sometimes improves upon the state-of-the-art accuracy.
Revised: We show that SparCL works collaboratively withexisting CL methods, greatly accelerates the learning process under different sparsity ratios, andeven sometimes improves upon the state-of-the-art accuracy.

Original: We also establish competing baselinesby combining representative sparse training methods with advanced rehearsal-based CL methods.
Revised: We also establish competing baselinesby combining representative sparse training methods with advanced rehearsal-based CL methods.

Original: SparCL again outperforms these baselines in terms of both efﬁciency and accuracy.
Revised: SparCL again outperforms these baselines in terms of both efﬁciency and accuracy.

Original: Most importantly,we evaluate our SparCL framework on real edge devices to demonstrate the practical potential ofour method.
Revised: Most importantly,we evaluate our SparCL framework on real edge devices to demonstrate the practical potential ofour method.

Original: We are not aware of any prior CL works that explored this area and considered theconstraints of limited resources during training.
Revised: We are not aware of any prior CL works that explored this area and considered theconstraints of limited resources during training.

Original: In summary, our work makes the following contributions:
Revised: In summary, our work makes the following contributions:

Original: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continuallearning, which achieves learning acceleration through the synergy of weight sparsity , data efﬁciency , and gradient sparsity .
Revised: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continuallearning, which achieves learning acceleration through the synergy of weight sparsity , data efﬁciency , and gradient sparsity .

Original: To the best of our knowledge, our work is the ﬁrst to introduce the
Revised: To the best of our knowledge, our work is the ﬁrst to introduce the

Original: idea of sparse training to enable efﬁcient CL on edge devices.
Revised: idea of sparse training to enable efﬁcient CL on edge devices.

Original: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted
Revised: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted

Original: sparse training methods on all benchmark datasets, leading to at most 23 ⇥ less training FLOPs and, surprisingly, 1 . 7% improvement over SOTA accuracy.
Revised: sparse training methods on all benchmark datasets, leading to at most 23 ⇥ less training FLOPsand, surprisingly, 1 . 7% improvement over SOTA accuracy.

Original: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of ourmethod and also encouraging future research on CL on-the-edge.
Revised: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of ourmethod and also encouraging future research on CL on-the-edge.

Original: The results indicate that ourframework can achieve at most 3 . 1 ⇥ training acceleration.
Revised: The results indicate that ourframework can achieve at most 3 . 1 ⇥ training acceleration.

Original:  Related work2.1 Continual Learning
Revised:  Related work2.1 Continual Learning

Original: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.
Revised: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.

Original: Existingmethods can be classiﬁed into three major categories.
Revised: Existingmethods can be classiﬁed into three major categories.

Original: Regularization-based methods [2, 28, 33, 62]limit updates of important parameters for the prior tasks by adding corresponding regularization terms.
Revised: Regularization-based methods [2, 23, 27, 53]limit updates of important parameters for the prior tasks by adding corresponding regularization terms.

Original: While these methods reduce catastrophic forgetting to some extent, their performance deterioratesunder challenging settings [36], and on more complex benchmarks [46, 56].
Revised: While these methods reduce catastrophic forgetting to some extent, their performance deterioratesunder challenging settings [29], and on more complex benchmarks [38, 48].

Original: Rehearsal-basedmethods [13, 14, 22] save examples from previous tasks into a small-sized buffer to train the modeljointly with the current task.
Revised: Rehearsal-basedmethods [10, 11, 19] save examples from previous tasks into a small-sized buffer to train the modeljointly with the current task.

Original: Though simple in concept, the idea of rehearsal is very effective inpractice and has been adopted by many state-of-the-art methods [8, 11, 45].
Revised: Though simple in concept, the idea of rehearsal is very effectivein practice and has been adopted by many state-of-the-art methods [7, 8, 37].

Original: Architecture-basedmethods [38, 47, 53, 54, 57] isolate existing model parameters or assign additional parameters foreach task to reduce interference among tasks.
Revised: Architecture-basedmethods [31, 39, 45, 46, 49] isolate existing model parameters or assign additional parameters foreach task to reduce interference among tasks.

Original: As mentioned in Section 1, most of these methods usea dense model without consideration of efﬁciency and memory footprint, thusare not applicable toresource-limited settings.
Revised: As mentioned in Section 1, most of these methodsuse a dense model without consideration of efﬁciency and memory footprint, thus not applicable toresource-limited settings.

Original: Our work, orthogonal to these methods, serves as a general framework formaking these existing methods efﬁcient and enabling a broader deployment, e.g. , CL on edge devices.
Revised: Our work, orthogonal to these methods, serves as a general framework formaking these existing methods efﬁcient and enabling a broader deployment, e.g. , CL on edge devices.

Original: A limited number of works explore sparsity in CL, however, for different purposes.
Revised: A limited number of works explore sparsity in CL, however, for different purposes.

Original: Several methods[37, 38, 49, 53] incorporate the idea of weight pruning [21] to allocate a sparse sub-network foreach task to reduce inter-task interference.
Revised: Several methods[30, 31, 41, 45] incorporate the idea of weight pruning [18] to allocate a sparse sub-network foreach task to reduce inter-task interference.

Original: Nevertheless, these methods still reduce the full modelsparsity progressively for every task and ﬁnally end up with a much denser model.
Revised: Nevertheless, these methods still reduce the full modelsparsity progressively for every task and ﬁnally end up with a much denser model.

Original: On the contrary,
Revised: On the contrary,

Original: SparCL maintains a sparse network throughout the whole CL process, introducing great efﬁciencyand memory beneﬁts both during training and at the output model.
Revised: SparCL maintains a sparse network throughout the whole CL process, introducing great efﬁciencyand memory beneﬁts both during training and at the output model.

Original: A recent work [15] aims atdiscovering lottery tickets [20] under CL, but still does not address efﬁciency.
Revised: A recent work [12] aims atdiscovering lottery tickets [17] under CL, but still does not address efﬁciency.

Original: However, the existenceof lottery tickets in CL serves as a strong justiﬁcation for the outstanding performance of our SparCL.
Revised: However, the existenceof lottery tickets in CL serves as a strong justiﬁcation for the outstanding performance of our SparCL.

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: There are two main approaches for sparse training: ﬁxed-mask sparse training and dynamic sparsetraining.
Revised: There are two main approaches for sparse training: ﬁxed-mask sparse training and dynamic sparsetraining.

Original: Fixed-mask sparse training methods [31, 50, 52, 55] ﬁrst apply pruning, then executetraditional training on the sparse model with the obtained ﬁxed mask.
Revised: Fixed-mask sparse training methods [26, 42, 44, 47] ﬁrst apply pruning, then executetraditional training on the sparse model with the obtained ﬁxed mask.

Original: The pre-ﬁxed structure limitsthe accuracy performance, and the ﬁrst stage still causes huge computation and memory consumption.
Revised: The pre-ﬁxed structure limitsthe accuracy performance, and the ﬁrst stage still causes huge computation and memory consumption.

Original: To overcome these drawbacks, dynamic mask methods [4, 17, 19, 41, 42] adjust the sparsity topologyduring training while maintaining low memory footprint.
Revised: To overcome these drawbacks, dynamic mask methods [4, 14, 16, 34, 35] adjust the sparsity topologyduring training while maintaining low memory footprint.

Original: These methods start with a sparse modelstructure from an untrained dense model, then combine sparse topology exploration at the givensparsity ratio with the sparse model training.
Revised: These methods start with a sparse modelstructure from an untrained dense model, then combine sparse topology exploration at the givensparsity ratio with the sparse model training.

Original: Recent work [61] further considers to incorporate dataefﬁciency into sparse training for better training accelerations.
Revised: Recent work [52] further considers to incorporate dataefﬁciency into sparse training for better training accelerations.

Original: However, all prior sparse trainingworks are focused on the traditional training setting, while CL is a more complicated and difﬁcultscenario with inherent characteristics not explored by these works.
Revised: However, all prior sparse trainingworks are focused on the traditional training setting, while CL is a more complicated and difﬁcultscenario with inherent characteristics not explored by these works.

Original: In contrast to prior sparse trainingmethods, our work explores a new learning paradigm that introduces sparse training into CL forefﬁciency and also addresses key challenges in CL, mitigating catastrophic forgetting.
Revised: In contrast to prior sparse trainingmethods, our work explores a new learning paradigm that introduces sparse training into CL forefﬁciency and also addresses key challenges in CL, mitigating catastrophic forgetting.

Original:  Continual Learning Problem Setup
Revised:  Continual Learning Problem Setup

Original: In supervised CL, a model f ✓ learns from a sequence of tasks D
Revised: In supervised CL, a model f ✓ learns from a sequence of tasks D

Original: =
Revised: =

Original: {D 1 , . . .
Revised: {D 1 , . . .

Original: , D T } , where each task
Revised: , D T } , where each task

Original: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.
Revised: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.

Original: Tasksarrive sequentially, and the model must adapt to them.
Revised: Tasksarrive sequentially, and the model must adapt to them.

Original: At the t -th step, the model gains access todata from the t -th task.
Revised: At the t -th step, the model gains access todata from the t -th task.

Original: However, a small ﬁx-sized rehearsal buffer M is allowed to save data fromprior tasks.
Revised: However, a small ﬁx-sized rehearsal buffer M is allowed to save data fromprior tasks.

Original: At test time, the easiest setting is to assume task identity is known for each coming testexample, named task-incremental learning (Task-IL).
Revised: At test time, the easiest setting is to assume task identity is known for each coming testexample, named task-incremental learning (Task-IL).

Original: If this assumption does not hold, we have themore difﬁcult class-incremental learning (Class-IL) setting.
Revised: If this assumption does not hold, we have themore difﬁcult class-incremental learning (Class-IL) setting.

Original: In this work, we mainly focus on themore challenging Class-IL setting, and only report Task-IL performance for reference.
Revised: In this work, we mainly focus on themore challenging Class-IL setting, and only report Task-IL performance for reference.

Original: The goal of conventional CL is to train a model sequentially that performs well on all tasks at testtime.
Revised: The goal of conventional CL is to train a model sequentially that performs well on all tasks at testtime.

Original: The main evaluation metric is average test accuracy on all tasks.
Revised: The main evaluation metric is average test accuracy on all tasks.

Original: In real-world resourcelimited scenarios, we should further consider training efﬁciency of the model.
Revised: In real-world resourcelimited scenarios, we should further consider training efﬁciency of the model.

Original: Thus, we measure theperformance of the model more comprehensively by including training FLOPs and memory footprint.
Revised: Thus, we measure theperformance of the model more comprehensively by including training FLOPs and memory footprint.

Original:  Sparse Continual Learning (SparCL)
Revised:  Sparse Continual Learning (SparCL)

Original: Our method, Sparse Continual Learning, is a uniﬁed framework composed of three complementarycomponents: task-aware dynamic masking for weight sparsity, dynamic data removal for dataefﬁciency, and dynamic gradient masking for gradient sparsity.
Revised: Our method, Sparse Continual Learning, is a uniﬁed framework composed of three complementarycomponents: task-aware dynamic masking for weight sparsity, dynamic data removal for dataefﬁciency, and dynamic gradient masking for gradient sparsity.

Original: The entire framework is shown in
Revised: The entire framework is shown in

Original: Figure 2.
Revised: Figure 2.

Original: We will illustrate each component in detail in this section.
Revised: We will illustrate each component in detail in this section.

Original: 4.1 Task-aware Dynamic Masking
Revised: 4.1 Task-aware Dynamic Masking

Original: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamicstructure when learning a sequence of tasks, such that it not only achieves high efﬁciency, but alsointelligently adapts to the data stream for better performance.
Revised: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamicstructure when learning a sequence of tasks, such that it not only achieves high efﬁciency, but alsointelligently adapts to the data stream for better performance.

Original: Speciﬁcally, we propose a strategynamed task-aware dynamic masking (TDM), which dynamically removes less important weightsand grows back unused weights for stronger representation power periodically by maintaining asingle binary weight mask throughout the CL process.
Revised: Speciﬁcally, we propose a strategynamed task-aware dynamic masking (TDM), which dynamically removes less important weights andgrows back unused weights for stronger representation power periodically.

Original: Different from typical sparse training work,which only leverages the weight magnitude [41] or the gradient w.r.t.
Revised: Different from typicalsparse training work, which only leverages the weight magnitude [34] or the gradient w.r.t.

Original: data from a single trainingtask [19, 61], TDM considers also the importance of weights w.r.t.
Revised: data froma single training task [16, 52], TDM considers also the importance of weights w.r.t.

Original: data saved in the rehearsal buffer,as well as the switch between CL tasks.
Revised: data saved in therehearsal buffer, as well as the switch between CL tasks.

Original: Speciﬁcally, TDM strategy starts from a randomly initialized binary mask M ✓ = M 0 , with a givensparsity constraint k M ✓ k 0 / k ✓ k 0 = 1  s , where s 2 [0 , 1] is the sparsity ratio.
Revised: Speciﬁcally, TDM strategy starts from a randomly initialized binary mask M ✓ = M 0 , with a givensparsity constraint k M ✓ k 0 / k ✓ k 0 = 1  s , where s 2 [0 , 1] is the sparsity ratio.

Original: Moreover, it makes 149 different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on theircontinual weight importance (CWI).
Revised: Moreover, it makes 148 different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on theircontinual weight importance (CWI).

Original: We summarize the process of task-aware dynamic masking in
Revised: We summarize the process of task-aware dynamic masking in

Original: Algorithm 1 and elaborate its key components in detail below.
Revised: Algorithm 1 and elaborate its key components in detail below.

Original: Continual weight importance (CWI).
Revised: Continual weight importance (CWI).

Original: For a model f ✓ parameterized by ✓ , the CWI of weight w ⇢ ✓is deﬁned as follows:
Revised: For a model f ✓ parameterized by ✓ , the CWI of weight w ⇢ ✓is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and ↵ ,  arecoefﬁcients to control the inﬂuence of current and buffered data, respectively.
Revised: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and ↵ ,  arecoefﬁcients to control the inﬂuence of current and buffered data, respectively.

Original: Moreover, L representsthe cross-entropy loss for classiﬁcation, while ˜ L is the single-head [1] version of the cross-entropyloss, which only considers classes from the current task by masking out the logits of other classes.
Revised: Moreover, L representsthe cross-entropy loss for classiﬁcation, while ˜ L is the single-head [1] version of the cross-entropyloss, which only considers classes from the current task.

Original: Intuitively, CWI ensures we keep(1) weights of larger magnitude for output stability, (2) weightsimportant for the current task for learning capacity, and(3) weights important for past data to mitigatecatastrophic forgetting.
Revised: Intuitively, CWI ensures we keep weights of larger magnitude for output stability, weights importantfor the current task for learning capacity, and weights important for past data to mitigate catastrophicforgetting.

Original: Moreover, inspired by the classiﬁcation bias in CL [1], we use the single-headcross-entropy loss when calculating importance score w.r.t.
Revised: Moreover, inspired by the classiﬁcation bias in CL [1], we use the single-head crossentropy loss when calculating importance score w.r.t.

Original: the current task to make the importanceestimation more accurate.
Revised: the current task to make the importanceestimation more accurate.

Revised: Intra-task adjustment.

Revised: When training the t -th task, a natural assumption is that the data distributionis consistent inside the task, thus we would like to update the sparse model in a relatively stable way

Original: Input : Model weight ✓ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M ✓ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval  k
Revised: Input : Model weight ✓ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M ✓ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval  k

Original: Initialize: ✓ , M ✓ , s.t.
Revised: Initialize: ✓ , M ✓ , s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original:  s for t = 1 , . .
Revised:  s for t = 1 , . .

Original: , T do for e = 1 , . . .
Revised: , T do for e = 1 , . . .

Original: , K T do if t > 1 then /
Revised: , K T do if t > 1 then /

Original: * Inter-task adjustment */ Expand M ✓ by randomly adding unused weights, s.t.
Revised: * Inter-task adjustment */ Expand M ✓ by randomly adding unused weights, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 1  ( s  p inter ) if e =  k then
Revised: k M ✓ k 0 / k ✓ k 0 = 1  ( s  p inter ) if t =  k then

Original: Shrink M ✓ by removing the least important weights according to equation 1, s.t.
Revised: Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original:  s end endif e mod  k = 

Revised:  s else if t mod  k =

Original: then /
Revised: then /

Original: * Intra-task adjustment */ Shrink M ✓ by removing the least important weights according to equation 1, s.t.
Revised: * Intra-task adjustment */ Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 1  ( s + p intra ) Expand M
Revised: k M ✓ k 0 / k ✓ k 0 = 1  ( s + p intra ) Expand M

Original: ✓ by randomly adding unused weights, s.t. k M ✓ k 0 / k ✓ k 0 = 1  s end
Revised: ✓ by randomly adding unused weights, s.t. k M ✓ k 0 / k ✓ k 0 = 1  s end

Original: Update ✓  M ✓ via backpropagation end end Intra-task adjustment.
Revised: Update ✓  M ✓ via backpropagation end end while keeping its ﬂexibility.

Original: When training the t -th task, a natural assumption is that the data distributionis consistent inside the task, thus we would like to update the sparse model in a relatively stable waywhile keeping its ﬂexibility.

Original: Thus, in Algorithm 1, we choose to update the sparsity mask M ✓ in ashrink-and-expand way every  k epochs.
Revised: Thus, in Algorithm 1, we choose to update the sparsity mask M ✓ in ashrink-and-expand way every  k epochs.

Original: We ﬁrst remove p intra of the weights of least CWI to retainlearned knowledge so far.
Revised: We ﬁrst remove p intra of the weights of least CWI to retainlearned knowledge so far.

Original: Then we randomly select unused weights to recover the learning capacityfor the model and keep the sparsity ratio s unchanged.
Revised: Then we randomly select unused weights to recover the learning capacityfor the model and keep the sparsity ratio s unchanged.

Original: Inter-task adjustment.
Revised: Inter-task adjustment.

Original: When tasks switches, on the contrary, we assume data distribution shiftsimmediately.
Revised: When tasks switches, on the contrary, we assume data distribution shiftsimmediately.

Original: Ideally, we would like the model to keep the knowledge learned from old tasks as muchas possible, and to have enough learning capacity to accommodate the new task.
Revised: Ideally, we would like the model to keep the knowledge learned from old tasks as muchas possible, and to have enough learning capacity to accommodate the new task.

Original: Thus, instead ofthe shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.
Revised: Thus, instead ofthe shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.

Original: Speciﬁcally, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly addinga proportion of p inter unused weights.
Revised: Speciﬁcally, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly addinga proportion of p inter unused weights.

Original: Intuitively, the additional learning capacity facilitates fastadoption of new knowledge and reduces interference with learned knowledge.
Revised: Intuitively, the additional learning capacity facilities fastadoption of new knowledge and reduces interference with learned knowledge.

Original: We allow our model tohave smaller sparsity( i.e. , larger learning capacity) temporarily for the ﬁrst  k epochs as a warmup period, and then remove the p inter weights with least CWI, following the same process in theintra-task case, to satisfy the sparsity constraint.
Revised: We allow our modelto have smaller sparsity temporarily for the ﬁrst  k epochs as a warm-up period, and then removethe p inter weights with least CWI, following the same process in the intra-task case, to satisfy thesparsity constraint.

Original: 4.2 Dynamic Data Removal
Revised: 4.2 Dynamic Data Removal

Original: In addition to weight sparsity, decreasing the amount of training data can be directly translated intothe saving of training time without any requirements for hardware support.
Revised: In addition to weight sparsity, decreasing the amount of training data can be directly translated intothe saving of training time without any requirements for hardware support.

Original: Thus, we would also liketo explore data efﬁciency to reduce the training workload.
Revised: Thus, we would also liketo explore data efﬁciency to reduce the training workload.

Original: Some prior CL works select informativeexamples to construct the rehearsal buffer [3, 6, 59].
Revised: Some prior CL works select informativeexamples to construct the rehearsal buffer [3, 6, 51].

Original: However, the main purpose of them is nottraining acceleration, thus they either introduce excessive computational cost or consider differentproblem settings.
Revised: However, the main purpose of them is nottraining acceleration, thus they either introduce excessive computational cost or consider differentproblem settings.

Original: By considering the features of CL, we present a simple yet effective strategy,dynamic data removal (DDR), to reduce training data for further acceleration.
Revised: By considering the features of CL, we present a simple yet effective strategy,dynamic data removal (DDR), to reduce training data for further acceleration.

Original: We measure the importance of each training example by the occurrence of misclassiﬁcation [51, 61]during CL.
Revised: We measure the importance of each training example by the occurrence of misclassiﬁcation [43, 52]during CL.

Original: In TDM, the sparse structure of our model updates periodically every  k epochs, so wealign our data removal process with the update of weight mask for further efﬁciency and training stability.
Revised: In TDM, the sparse structure of our model updates periodically every  k epochs, so wealign our data removal process with the update of weight mask for further efﬁciency and trainingstability.

Original: In Section 4.1, we have partitioned the training process for the t -th task into N t =
Revised: In Section 4.1, we have partitioned the training process for the t -th task into N t =

Original: K t /  kstages based on the dynamic mask update.
Revised: K t /  kstages based on the dynamic mask update.

Original: Therefore, we gradually remove training data at the end ofi -th stage, based on the following policy: 1) Calculate the total number of misclassiﬁcations f i ( x j )for each training example during the i -th stage.
Revised: Therefore, we gradually remove training data at the end ofi -th stage, based on the following policy: 1) Calculate the total number of misclassiﬁcations f i ( x j )for each training example during the i -th stage.

Original: 2) Remove a proportion of ⇢ i training samples withthe least number of misclassiﬁcations.
Revised: 2) Remove a proportion of ⇢ i training samples with the least number of misclassiﬁcations.

Original: Although our main purpose is to keep the “harder” examplesto learn to consolidate the sparse model, we can get further beneﬁts for better CL result.
Revised: Although our main purpose is to keep the “harder” examplesto learn to consolidate the sparse model, we can get further beneﬁts for better CL result.

Original: First, theremoval of “easier” examples increases the probability that “harder” examples to be saved to therehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples.
Revised: First, theremoval of “easier” examples increases the probability that “harder” examples to be saved to therehearsal buffer, given the common strategy, e.g. reservoir sampling [11], to buffer examples.

Original: Thus,we construct a more informative buffer in a implicit way without heavy computation.
Revised: Thus,we construct a more informative buffer in a implicit way without heavy computation.

Original: Moreover, sincethe buffer size is much smaller than the training set size of each task, the data from the buffer and thenew task is highly imbalanced, dynamic data removal also relieves the data imbalance issue.
Revised: Moreover, sincethe buffer size is much smaller than the training set size of each task, the data from the buffer and thenew task is highly imbalanced, dynamic data removal also relieves the data imbalance issue.

Original: Formally, we set the data removal proportion for each task as ⇢ 2 [0 , 1] , and a cutoff stage, such that:
Revised: Formally, we set the data removal proportion for each task as ⇢ 2 [0 , 1] , and a cutoff stage, such that:

Original: [Equation]
Revised: [Equation]

Original: The cutoff stage controls the trade-off between efﬁciency and accuracy: When we set the cutoff stageearlier, we reduce the training time for all the following stages; however, when the cutoff stage is settoo early, the model might underﬁt the removed training data.
Revised: The cutoff stage controls the trade-off between efﬁciency and accuracy: When we set the cutoff stageearlier, we reduce the training time for all the following stages; however, when the cutoff stage is settoo early, the model might underﬁt the removed training data.

Original: Note that when we set ⇢ i = 0 for alli = 1 , 2 , . . .
Revised: Note that when we set ⇢ i = 0 for alli = 1 , 2 , . . .

Original: , N t and cutoff =
Revised: , N t and cutoff =

Original: N t , we simply recover the vanilla setting without any data efﬁciencyconsiderations.
Revised: N t , we simply recover the vanilla setting without any data efﬁciencyconsiderations.

Original: In our experiments, we assume ⇢ i = ⇢ / cutoff , i.e., removing equal proportion ofdata at the end of every stage, for simplicity.
Revised: In our experiments, we assume ⇢ i = ⇢ / cutoff , i.e., removing equal proportion ofdata at the end of every stage, for simplicity.

Original: We also conduct comprehensive exploration study for ⇢and the selection of the cutoff stage in Section 5.3 and Appendix D.3.
Revised: We also conduct comprehensive exploration study for ⇢and the selection of the cutoff stage in Section 5.3 and Appendix D.3.

Original: 4.3 Dynamic Gradient Masking
Revised: 4.3 Dynamic Gradient Masking

Original: With TDM and DDR, we can already achieve bi-level efﬁciency during training.
Revised: With TDM and DDR, we can already achieve bi-level efﬁciency during training.

Original: To further boosttraining efﬁciency, we explore sparsity in gradient and propose dynamic gradient masking (DGM)for CL.
Revised: To further boosttraining efﬁciency, we explore sparsity in gradient and propose dynamic gradient masking (DGM)for CL.

Original: Our method focuses on reducing computational cost by only applying the most importantgradients onto the corresponding unpruned model parameters via a gradient mask.
Revised: Our method focuses on reducing computational cost by only applying the most importantgradients onto the corresponding unpruned model parameters via a gradient mask.

Original: The gradient maskis also dynamically updated along with the weight mask deﬁned in Section 4.1.
Revised: The gradient maskis also dynamically updated along with the weight mask deﬁned in Section 4.1.

Original: Intuitively, whiletargeting for better training efﬁciency, DGM also promotes the preservation of past knowledge bypreventing a fraction of weights from update.
Revised: Intuitively, whiletargeting for better training efﬁciency, DGM also promotes the preservation of past knowledge bypreventing a fraction of weights from update.

Original: Formally, our goal here is to ﬁnd a subset of unpruned parameters (or, equivalently, a gradient mask
Revised: Formally, our goal here is to ﬁnd a subset of unpruned parameters (or, equivalently, a gradient mask

Original: M G ) to update over multiple training iterations.
Revised: M G ) to update over multiple training iterations.

Original: For a model f ✓ parameterized by ✓ , we have thecorresponding gradient matrix G calculated during each iteration.
Revised: For a model f ✓ parameterized by ✓ , we have thecorresponding gradient matrix G calculated during each iteration.

Original: To prevent the pruned weightsfrom updating, the weight mask M ✓ will be applied onto the gradient matrix G as G  M ✓ duringbackpropagation.
Revised: To prevent the pruned weightsfrom updating, the weight mask M ✓ will be applied onto the gradient matrix G as G  M ✓ duringbackpropagation.

Original: Besides the gradients of pruned weights, we in addition consider to remove lessimportant gradients for faster training.
Revised: Besides the gradients of pruned weights, we in addition consider to remove lessimportant gradients for faster training.

Original: To achieve this, we introduce the continual gradient importance(CGI) based on the CWI to measure the importance of weight gradients.
Revised: To achieve this, we introduce the continual gradient importance(CGI) based on the CWI to measure the importance of weight gradients.

Original: [Equation]
Revised: [Equation]

Original: We remove a proportion q of non-zero gradients from G with less importance measured by CGIand we have k M G k 0 / k ✓ k 0 = 1  ( s + q ) .
Revised: We remove a proportion q of non-zero gradients from G with less importance measured by CGIand we have k M G k 0 / k ✓ k 0 = 1  ( s + q ) .

Original: The gradient mask M G is then applied onto the gradient 226 matrix G .
Revised: The gradient mask M G is then applied onto the gradient 225 matrix G .

Original: During the entire training process, the gradient mask M G is updated with a ﬁxed interval.
Revised: During the entire training process, the gradient mask M G is updated with a ﬁxed interval.

Original:  Experiment
Revised:  Experiment

Original: 5.1 Experiment Setting
Revised: 5.1 Experiment Setting

Original: Datasets.
Revised: Datasets.

Original: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [29]and Split Tiny-ImageNet [16] to verify the efﬁcacy of SparCL.
Revised: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [24]and Split Tiny-ImageNet [13] to verify the efﬁcacy of SparCL.

Original: In particular, we follow [8, 62] bysplitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classesrespectively.
Revised: In particular, we follow [7, 53] bysplitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classesrespectively.

Original: Dataset licensing information can be found in Appendix C.
Revised: Dataset licensing information can be found in Appendix C.

Original: Comparing methods.
Revised: Comparing methods.

Original: In particular, we select CL methods of all kinds including regularizationbased (EWC [28], LwF [33]), architecture-based (PackNet [38], LPS [53]), and rehearsal-based
Revised: In particular, we select CL methods of all kinds including regularizationbased (EWC [23], LwF [27]), architecture-based (PackNet [31], LPS [45]), and rehearsal-

Original: [Table]
Revised: [Table]

Original: (A-GEM [13], iCaRL [40], FDR [5], ER [14], DER++ [8]) methods. Note that PackNet and LPSare only compatible with task-incremental learning.
Revised: Variants of our method.

Original: We also adapt representative sparse trainingmethods (SNIP [31], RigL [19]) to the CL setting by combining them with DER++ (SNIP-DER++,
Revised: To show the generality of SparCL [Table] we combine it with DER (one of the SOTA CL methods)

Original: RigL-DER++).
Revised: [Table]

Original: Variants of our method. To show the generality of SparCL, we combine it with DER++ (one ofthe SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER ,respectively.
Revised: and ER (simple and widely-used) as SparCL-DER and SparCL-ER , respectively.

Original: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensiveevaluation.
Revised: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensive evaluation.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use the average accuracy on all tasks to evaluate the performance of the ﬁnalmodel.
Revised: We use the average accuracy on all tasks to evaluate the performance of theﬁnal model.

Original: Moreover, we evaluate the training FLOPs [19], and memory footprint [61] (including featuremap pixels and model parameters during training) to demonstrate the efﬁciency of each method.
Revised: Moreover, we evaluate the training FLOPs [16] [Table] and memory footprint [52] (includingfeature map pixels and model parameters) to demonstrate the efﬁciency of each method.

Original: Please see Appendix D.1 for detailed deﬁnitions of these metrics.
Revised: Please see Appendix D.1 for detailed deﬁnitions of these metrics.

Original: Experiment details.
Revised: Experiment details.

Original: For fair comparison, we strictly follow the settings in prior CL work [8, 26].
Revised: For fair comparison, we strictly follow the settings in prior CL work [7, 21].

Original: Wesets the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively,with a batch size of 32 .
Revised: Wesets the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively,with a batch size of 32 .

Original: For the model architecture, We follow [8, 46] and adopt the ResNet-18 [23]without any pre-training.
Revised: For the model architecture, We follow [7, 38] and adopt the ResNet-18 [20]without any pre-training.

Original: We also use the best hyperparameter setting reported in [8, 53] for CLmethods, and in [19, 31] for CL-adapted sparse training methods.
Revised: We also use the best hyperparameter setting reported in [7, 45] for CLmethods, and in [16, 26] for CL-adapted sparse training methods.

Original: For SparCL and its competing
Revised: For SparCL and its competing

Original: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.
Revised: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.

Original: Please see Appendix D for other details.
Revised: Please see Appendix D for other details.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: Comparison with CL methods.
Revised: Comparison with CL methods.

Original: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.
Revised: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.

Original: From
Revised: From

Original: Table 1, we can clearly tell that SparCL signiﬁcantly improves ER and DER++, while also outperformsother CL baselines, in terms of training efﬁciency (measured in FLOPs).
Revised: Table 1, we can clearly tell that SparCL signiﬁcantly improves ER and DER, while also outperformsother CL baselines, in terms of training efﬁciency (measured in FLOPs).

Original: With higher sparsity ratio,
Revised: With higher sparsity ratio,

Original: SparCL leads to less training FLOPs.
Revised: SparCL leads to less training FLOPs.

Original: Notably, SparCL achieves 23 ⇥ training efﬁciency improvementupon DER++ with a sparsity ratio of 0.95.
Revised: Notably, SparCL achieves 23 ⇥ training efﬁciency improvementupon DER with a sparsity ratio of 0.95.

Original: On the other hand, our framework also improves theaverage accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90,[Table] [Table] and only slight performance drop when sparsity gets larger as 0.95.
Revised: On the other hand, our framework also improves the averageaccuracy of ER and DER consistently under all cases with a sparsity ratio of 0.75 and 0.90, and onlyslight performance drop when sparsity gets larger as 0.95.

Original: In particular, SparCL-DER++with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.
Revised: In particular, SparCL-DER with 0.75sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.

Original: Theoutstanding performance of SparCL indicates that our proposed strategies successfully preserveaccuracy by further mitigating catastrophic forgetting with a much sparser model. 
Revised: The outstandingperformance of SparCL indicates that our proposed strategies successfully preserve accuracy byfurther mitigating catastrophic forgetting with a much sparser model. Moreover, the improvement

Original: Moreover, theimprovement that SparCL brings to two different existing CL methods shows the generalizability of SparCL as a uniﬁed framework, i.e. , it has the potential to be combined with a wide array of existingmethods.
Revised: [Table] [Table] that SparCL brings to two different existing CL methods shows the generalizability of SparCL as auniﬁed framework, i.e. , it has the potential to be combined with a wide array of existing methods.

Original: We would also like to take a closer look at PackNet and LPS, which also leverage the idea of sparsityto split the model by different tasks, a different motivation from training efﬁciency.
Revised: We would also like to take a closer look at PackNet and LPS, which also leverage the idea of sparsityto split the model by different tasks, a different motivation from training efﬁciency.

Original: Firstly, they areonly compatible with the Task-IL setting, since they leverage task identity at both training and testtime.
Revised: Firstly, they areonly compatible with the Task-IL setting, since they leverage task identity at both training and testtime.

Original: Moreover, the model sparsity of these methods reduces with the increasing number of tasks,which still leads to much larger overall training FLOPs than that of SparCL.
Revised: Moreover, the model sparsity of these methods reduces with the increasing number of tasks,which still leads to much larger overall training FLOPs than that of SparCL.

Original: This further demonstratesthe importance of keeping a sparse model without permanent expansion throughout the CL process.
Revised: This further demonstratesthe importance of keeping a sparse model without permanent expansion throughout the CL process.

Original: Comparison with CL-adapted sparse training methods.
Revised: Comparison with CL-adapted sparse training methods.

Original: Table 2 shows the result under themore difﬁcult Class-IL setting.
Revised: Table 2 shows the result under themore difﬁcult Class-IL setting.

Original: SparCL outperforms all CL-adapted sparse training methods in bothaccuracy and training FLOPs.
Revised: SparCL outperforms all CL-adapted sparse training methods inboth accuracy and training FLOPs.

Original: The performance gap between SparCL-DER++ and other methodsgets larger with a higher sparsity.
Revised: The performance gap between SparCL-DER and other methodsgets larger with a higher sparsity.

Original: SNIP-
Revised: SNIP-

Original: and RigL-DER++ achieve training acceleration at the cost ofcompromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existingsparse training methods under the CL setting.
Revised: and RigL-DER achieve training acceleration at the cost ofcompromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existingsparse training methods under the CL setting.

Original: SNIP generates the static initial mask after networkinitialization which does not consider the structure suitability among tasks.
Revised: SNIP generates the static initial mask after networkinitialization which does not consider the structure suitability among tasks.

Original: Though RigL adopts adynamic mask, the lack of task-aware strategy prevents it from generalizing well to the CL setting.
Revised: Though RigL adopts adynamic mask, the lack of task-aware strategy prevents it from generalizing well to the CL setting.

Original: 5.3 Effectiveness of Key Components
Revised: 5.3 Effectiveness of Key Components

Original: Ablation study.
Revised: Ablation study.

Original: We provide a comprehensive ablation study in Table 3 using SparCL-DER++ with0.75 sparsity on Split CIFAR10.
Revised: We provide a comprehensive ablation study in Table 3 using SparCL-DER with 0.75sparsity on Split CIFAR10.

Original: Table 3 demonstrates that all components of our method contributeto both efﬁciency and accuracy improvements.
Revised: Table 3 demonstrates that all components of our method contributes toboth efﬁciency and accuracy improvements.

Original: Comparing row 1 and 2, we can see that the majorityof FLOPs decrease results from TDM.
Revised: Comparing row 1 and 2, we can see that the majority of FLOPs decrease results from TDM.

Original: Interestingly, TDM leads to an increase in accuracy, indicating
Revised: Interestingly, TDM leads to an increase in accuracy, indicating

Original: TDM generates a sparse model that is even more suitable for learning all tasks than then full densemodel.
Revised: TDM generates a sparse model that is even more suitable for learning all tasks than then full densemodel.

Original: Comparing row 2 and 3, we can see that DDR indeed further accelerates training by removingless informative examples.
Revised: Comparing row 2 and 3, we can see that DDR indeed further accelerates training by removingless informative examples.

Original: As discussed in Section 4.2, when we remove a certain number of samples(30% here), we achieve a point where we keep as much  informative samples as we need, and alsobalance the current and buffered data.
Revised: As discussed in Section 4.2, when we remove a certain number of samples(30% here), we achieve a point where we keep as much as informative samples as we need, and alsobalance the current and buffered data.

Original: Comparing row 2 and 4, DGM reduce both training FLOPs andmemory footprint while improve the performance of the network.
Revised: Comparing row 2 and 4, DGM reduce both training FLOPs andmemory footprint while improve the performance of the network.

Original: Finally, the last row demonstratesthe collaborative performance of all components.
Revised: Finally, the last row demonstratesthe collaborative performance of all components.

Original: We also show the same ablation study with 0.90sparsity in Appendix D.4 for reference.
Revised: We also show the same ablation study with 0.90sparsity in Appendix D.4 for reference.

Original: Detail can be found in Appendix D.1.
Revised: Detail can be found in Appendix D.1.

Original: Exploration on DDR.
Revised: Exploration on DDR.

Original: To understand the inﬂuence of the data removal proportion ⇢ , and the cutoffstage for each task, we show corresponding experiment results in Figure 3 and Appendix D.3,respectively.
Revised: To understand the inﬂuence of the data removal proportion ⇢ , and the cutoffstage for each task, we show corresponding experiment results in Figure 3 and Appendix D.3, respectively.

Original: In Figure 3, we ﬁx cutoff = 4 , i.e. , gradually removing equal number of examplesevery 5 epochs until epoch 20, and vary ⇢ from 10% to 90% .
Revised: In Figure 3, we ﬁx cutoff = 4 , i.e. , gradually removing equal number of examplesevery 5 epochs until epoch 20, and vary ⇢ from 10% to 90% .

Original: We also compare DDR with One-shotremoval strategy [61], which removes all examples at once at cutoff .
Revised: We also compare DDR with One-shotremoval strategy [52], which removes all examples at once at cutoff .

Original: DDR outperforms One-shotconsistently with different ⇢ in average accuracy.
Revised: DDR outperforms One-shotconsistently with different ⇢ in average accuracy.

Original: Also note that since DDR removes the examplesgradually before the cutoff stage, DDR is more efﬁcient than One-shot.
Revised: Also note that since DDR removes the examplesgradually before the cutoff stage, DDR is more efﬁcient than One-shot.

Original: When ⇢  30% , we alsoobserve increased accuracy of DDR compared with the baseline without removing any data.
Revised: When ⇢  30% , we alsoobserve increased accuracy of DDR compared with the baseline without removing any data.

Original: When⇢  40% , the accuracy gets increasingly lower for both strategies.
Revised: When⇢  40% , the accuracy gets increasingly lower for both strategies.

Original: The intuition is that when DDRremoves a proper amount of data, it removes redundant information while keeps the most informativeexamples.
Revised: The intuition is that when DDRremoves a proper amount of data, it removes redundant information while keeps the most informativeexamples.

Original: Moreover, as discussed in Section 4.2, it balances the current and buffered data, whilealso leave informative samples in the buffer.
Revised: Moreover, as discussed in Section 4.2, it balances the current and buffered data, whilealso leave informative samples in the buffer.

Original: When DDR removes too much data, it will also loseinformative examples, thus the model has not learned these examples well before removal.
Revised: When DDR removes too much data, it will also loseinformative examples, thus the model has not learned these examples well before removal.

Original: Exploration on DGM.
Revised: Exploration on DGM.

Original: We test the efﬁcacy of DGM at different sparsity levels.
Revised: We test the efﬁcacy of DGM at different sparsity levels.

Original: Detailed exploratoryexperiments are shown in Appendix D.5 for reference.
Revised: Detailed exploratoryexperiments are shown in Appendix D.5 for reference.

Original: The results indicate that by setting theproportion q within an appropriate range, DGM can consistently improve the accuracy performanceregardless of the change of weight sparsity.
Revised: The results indicate that by setting theproportion q within an appropriate range, DGM can consistently improve the accuracy performanceregardless of the change of weight sparsity.

Original: 5.4 Mobile Device Results
Revised: 5.4 Mobile Device Results

Original: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S20smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo 

Revised: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S20smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo

Original: Octa-core CPU.
Revised: Octa-core CPU.

Original: We run each test on a batch of 32 images to denote the training speed.
Revised: We run each test on a batch of 32 images to denote the training speed.

Original: The detail ofon-mobile compiler-level optimizations for training acceleration can be found in Appendix E.1.
Revised: The detail ofon-mobile compiler-level optimizations for training acceleration can be found in Appendix E.1.

Original: The acceleration results are shown in Figure 4.
Revised: The acceleration results are shown in Figure 4.

Original: SparCL can achieve approximately 3.1 ⇥ and 2.3 ⇥training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.
Revised: SparCL can achieve approximately 3.1 ⇥ and 2.3 ⇥training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.

Original: Besides, our framework canalso save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.
Revised: Besides, our framework canalso save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.

Original: Furthermore, theobtained sparse models save the storage consumption by using compressed sparse row (CSR) storageand can be accelerated to speed up the inference on-the-edge.
Revised: Furthermore, theobtained sparse models save the storage consumption by using compressed sparse row (CSR) storageand can be accelerated to speed up the inference on-the-edge.

Original: We provide on-mobile inferenceacceleration results in Appendix E.2.
Revised: We provide on-mobile inferenceacceleration results in Appendix E.2.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presents a uniﬁed framework named SparCL for efﬁcient CL that achieves both learningacceleration and accuracy preservation.
Revised: This paper presents a uniﬁed framework named SparCL for efﬁcient CL that achieves both learningacceleration and accuracy preservation.

Original: It comprises three complementary strategies: task-awaredynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradientmasking for gradient sparsity.
Revised: It comprises three complementary strategies: task-awaredynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradientmasking for gradient sparsity.

Original: Extensive experiments on standard CL benchmarks and real-world edgedevice evaluations demonstrate that our method signiﬁcantly improves upon existing CL methods andoutperforms CL-adapted sparse training methods.
Revised: Extensive experiments on standard CL benchmarks and real-world edgedevice evaluations demonstrate that our method signiﬁcantly improves upon existing CL methods andoutperforms CL-adapted sparse training methods.

Original: We discuss the limitations and potential negativesocial impacts of our method in Appendix A and B, respectively.Checklist
Revised: We discuss the limitations and potential negativesocial impacts of our method in Appendix A and B, respectively.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] The claims match the experimental results and it isexpected to generalize according to the diverse experiments stated in our paper.

Original: We

Original: include all of our code, data, and models in the supplementary materials, which can

Original: reproduce our experimental results.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6 and

Original: Appendix A.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See

Original: Section 6 and Appendix B.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes] We have read the ethics review guidelines and ensured that our paper

Original: conforms to them.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: Our paper is

Original: based on the experimental results and we do not have any theoretical results.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: Our paper is based

Original: on the experimental results and we do not have any theoretical results.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section 5.1,

Original: Section 5.4 and we provide code to reproduce the main experimental results.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] See Section 5.1 and Section 5.4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Table 1, Table 2, ﬁg

Original: 1, ﬁg 3.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 5.1, Section 5.4.

Original: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We mentioned and

Original: [Table]

Original: cited the datasets (Split CIFAR-10 and Tiny-ImageNet), and all comparing methods

Original: [Table]

Original: with their paper and github in it.

Original: [Table]

Original: (b) Did you mention the license of the assets?

Original: [Yes] The licences of used datasets/models

Original: [Table]

Original: are provided in the cited references and we state them explicitly in Appendix C.

Original: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: [Table]

Original: We provide code for our proposed method in the supplement.

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?

Original: [N/A]

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?

Original: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?

Original: [N/A]

Original: [Table]