Original: SparCL: Sparse Continual Learning on the Edge
Revised: SparCL: Sparse Continual Learning on the Edge

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a newtask.
Revised: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a new task.

Original: However, the training efﬁciency of a CL system is under-investigated, whichlimits the real-world application of CL systems under resource-limited scenarios.
Revised: However, the training efﬁciency of a CL system is under-investigated, which limits the real-world application of CL systems under resource-limited scenarios.

Original: In this work, we propose a novel framework called Sparse Continual Learning(SparCL), which is the ﬁrst study that leverages sparsity to enable cost-effectivecontinual learning on edge devices.
Revised: In this work, we propose a novel framework called Sparse Continual Learning (SparCL), which is the ﬁrst study that leverages sparsity to enable cost-effective continual learning on edge devices.

Original: SparCL achieves both training accelerationand accuracy preservation through the synergy of three aspects: weight sparsity ,data efﬁciency , and gradient sparsity .
Revised: SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efﬁciency , and gradient sparsity .

Original: Speciﬁcally, we propose task-aware dynamicmasking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamicgradient masking (DGM) to sparsify the gradient updates.
Revised: Speciﬁcally, we propose task-aware dynamic masking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates.

Original: Each of them not onlyimproves efﬁciency, but also further mitigates catastrophic forgetting.
Revised: Each of them not only improves efﬁciency, but also further mitigates catastrophic forgetting.

Original: SparCLconsistently improves the training efﬁciency of existing state-of-the-art (SOTA) CLmethods by at most 23 ⇥ less training FLOPs, and, surprisingly, further improvesthe SOTA accuracy by at most 1 . 7% .
Revised: SparCL consistently improves the training efﬁciency of existing state-of-the-art (SOTA) CL methods by at most 23 ⇥ less training FLOPs, and, surprisingly, further improves the SOTA accuracy by at most 1 . 7% .

Original: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting inboth efﬁciency and accuracy.
Revised: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting in both efﬁciency and accuracy.

Original: We also evaluate the effectiveness of SparCL on areal mobile phone, further indicating the practical potential of our method.
Revised: We also evaluate the effectiveness of SparCL on a real mobile phone , further indicating the practical potential of our method.

Original: Sourcecode will be released.

Original:  Introduction
Revised:  Introduction

Original: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledgefrom a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [28].
Revised: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledge from a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [32].

Original: Recent methods mostly focus on addressing the catastrophic forgetting [39] problem – learningmodel tends to suffer performance deterioration on previously seen tasks.
Revised: Recent methods mostly focus on addressing the catastrophic forgetting [43] problem – learning model tends to suffer performance deterioration on previously seen tasks.

Original: However, in the real world,when the CL applications are deployed in resource-limited platforms [44] such as edge devices,the learning efﬁciency, w.r.t.
Revised: However, in the real world, when the CL applications are deployed in resource-limited platforms [48] such as edge devices, the learning efﬁciency, w.r.t.

Original: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.
Revised: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.

Original: Existing CL methods can be categorized into regularization-based [2, 28, 33, 62], rehearsal-based [8,12, 46, 56], and architecture-based [27, 38, 48, 53, 54].
Revised: Existing CL methods can be categorized into regularization-based [2, 32, 37, 68], rehearsal-based [8, 12, 50, 61], and architecture-based [31, 42, 52, 58, 59, 70].

Original: Both regularization- and rehearsal-basedmethods directly train a dense model, which might even be over-parametrized for the union of alltasks [18, 35]; Though several architecture-based methods [47, 53, 58] start with a sparse sub-networkfrom the dense model, they still grow the model size progressively to learn emerging tasks.
Revised: Both regularization- and rehearsal-based methods directly train a dense model, which might even be over-parametrized for the union of all tasks [19, 39]; Though several architecture-based methods [51, 57, 64] start with a sparse sub-network from the dense model, they still grow the model size progressively to learn emerging tasks.

Original: Theaforementioned methods, although striving for greater performance with less forgetting, still introducesigniﬁcant memory and computation overhead during the whole CL process.
Revised: The aforementioned methods, although striving for greater performance with less forgetting, still introduce signiﬁcant memory and computation overhead during the whole CL process.

Original: Recently, another stream of work, sparse training [4, 19, 31] has emerged as a new training trend toachieve training acceleration, which embraces the promising training-on-the-edge paradigm.
Revised: Recently, another stream of work, sparse training [4, 20, 35] has emerged as a new training trend to achieve training acceleration, which embraces the promising training-on-the-edge paradigm.

Original: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity, under the traditional i.i.d.
Revised: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity, under the traditional i.i.d.

Original: learning setting.
Revised: learning setting.

Original: Inspired by these sparse training methods, we naturallythink about introducing sparse training to the ﬁeld of CL.
Revised: Inspired by these sparse training methods, we naturally think about introducing sparse training to the ﬁeld of CL.

Original: A straightforward idea is to directly combineexisting sparse training methods, such as SNIP [31], RigL [19], with a rehearsal buffer under the
Revised: A straightforward idea is to directly combine existing sparse training methods, such as SNIP [35], RigL [20], with a rehearsal buffer under the

Original: CL setting.
Revised: CL setting.

Original: However, these methods fail to consider key challenges in CL to mitigate catastrophicforgetting, for example, properly handling transition between tasks.
Revised: However, these methods fail to consider key challenges in CL to mitigate catastrophic forgetting, for example, properly handling transition between tasks.

Original: As a result, these sparse trainingmethods, though enhancing training efﬁciency, cause signiﬁcant accuracy drop (see Section 5.2).
Revised: As a result, these sparse training methods, though enhancing training efﬁciency, cause signiﬁcant accuracy drop (see Section 5.2).

Original: Thus, we would like to explore a general strategy, which is orthogonal to existing CL methods, thatnot only leverages the idea of sparse training for efﬁciency, but also addresses key challenges in CLto preserve (or even improve) accuracy.
Revised: Thus, we would like to explore a general strategy, which is orthogonal to existing CL methods, that not only leverages the idea of sparse training for efﬁciency, but also addresses key challenges in CL to preserve (or even improve) accuracy.

Original: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effectivecontinual learning, aiming at enabling practical CL on edge devices.
Revised: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, aiming at enabling practical CL on edge devices.

Original: As shown in Figure 1 (left),
Revised: As shown in Figure 1 (left),

Original: SparCL achieves both learning acceleration and accuracy preservation through the synergy of threeaspects: weight sparsity , data efﬁciency , and gradient sparsity .
Revised: SparCL achieves both learning acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efﬁciency , and gradient sparsity .

Original: Speciﬁcally, to maintain a smalldynamic sparse network during the whole CL process, we develop a novel task-aware dynamicmasking (TDM) strategy to keep only important weights for both the current and past tasks, withspecial consideration during task transitions.
Revised: Speciﬁcally, to maintain a small dynamic sparse network during the whole CL process, we develop a novel task-aware dynamic masking (TDM) strategy to keep only important weights for both the current and past tasks, with special consideration during task transitions.

Original: Moreover, we propose a dynamic data removal (DDR)scheme, which progressively removes “easy-to-learn” examples from training iterations, whichfurther accelerates the training process and also improves accuracy of CL by balancing current andpast data and keeping more informative samples in the buffer.
Revised: Moreover, we propose a dynamic data removal (DDR) scheme, which progressively removes “easy-to-learn” examples from training iterations, which further accelerates the training process and also improves accuracy of CL by balancing current and past data and keeping more informative samples in the buffer.

Original: Finally, we provide an additionaldynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efﬁciencyand knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.
Revised: Finally, we provide an additional dynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efﬁciency and knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.

Original: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and signiﬁcantlyimproves efﬁciency over DER++ [8], one of the SOTA CL methods, under different sparsity ratios.
Revised: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and signiﬁcantly improves efﬁciency over DER++ [8], one of the SOTA CL methods, under different sparsity ratios.

Original: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, andefﬁcient under practical scenarios.
Revised: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, and efﬁcient under practical scenarios.

Original: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.
Revised: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.

Original: We show that SparCL works collaboratively withexisting CL methods, greatly accelerates the learning process under different sparsity ratios, andeven sometimes improves upon the state-of-the-art accuracy.
Revised: We show that SparCL works collaboratively with existing CL methods, greatly accelerates the learning process under different sparsity ratios, and even sometimes improves upon the state-of-the-art accuracy.

Original: We also establish competing baselinesby combining representative sparse training methods with advanced rehearsal-based CL methods.
Revised: We also establish competing baselines by combining representative sparse training methods with advanced rehearsal-based CL methods.

Original: SparCL again outperforms these baselines in terms of both efﬁciency and accuracy.
Revised: SparCL again outperforms these baselines in terms of both efﬁciency and accuracy.

Original: Most importantly,we evaluate our SparCL framework on real edge devices to demonstrate the practical potential ofour method.
Revised: Most importantly, we evaluate our SparCL framework on real edge devices to demonstrate the practical potential of our method.

Original: We are not aware of any prior CL works that explored this area and considered theconstraints of limited resources during training.
Revised: We are not aware of any prior CL works that explored this area and considered the constraints of limited resources during training.

Original: In summary, our work makes the following contributions:
Revised: In summary, our work makes the following contributions:

Original: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continuallearning, which achieves learning acceleration through the synergy of weight sparsity , data efﬁciency , and gradient sparsity .
Revised: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, which achieves learning acceleration through the synergy of weight sparsity , data efﬁ- ciency , and gradient sparsity .

Original: To the best of our knowledge, our work is the ﬁrst to introduce the idea of sparse training to enable efﬁcient CL on edge devices.
Revised: To the best of our knowledge, our work is the ﬁrst to introduce the idea of sparse training to enable efﬁcient CL on edge devices.

Revised: Our code is publicly available † .

Original: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted sparse training methods on all benchmark datasets, leading to at most 23 ⇥ less training FLOPs and, surprisingly, 1 . 7% improvement over SOTA accuracy.
Revised: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted sparse training methods on all benchmark datasets, leading to at most 23 ⇥ less training FLOPs and, surprisingly, 1 . 7% improvement over SOTA accuracy.

Original: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of ourmethod and also encouraging future research on CL on-the-edge.
Revised: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of our method and also encouraging future research on CL on-the-edge.

Original: The results indicate that ourframework can achieve at most 3 . 1 ⇥ training acceleration.
Revised: The results indicate that our framework can achieve at most 3 . 1 ⇥ training acceleration.

Original:  Related work2.1 Continual Learning
Revised:  Related work 2.1 Continual Learning

Original: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.
Revised: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.

Original: Existingmethods can be classiﬁed into three major categories.
Revised: Existing methods can be classiﬁed into three major categories.

Original: Regularization-based methods [2, 28, 33, 62]limit updates of important parameters for the prior tasks by adding corresponding regularization terms.
Revised: Regularization-based methods [2, 32, 37, 68] limit updates of important parameters for the prior tasks by adding corresponding regularization terms.

Original: While these methods reduce catastrophic forgetting to some extent, their performance deterioratesunder challenging settings [36], and on more complex benchmarks [46, 56].
Revised: While these methods reduce catastrophic forgetting to some extent, their performance deteriorates under challenging settings [40], and on more complex benchmarks [50, 61].

Original: Rehearsal-basedmethods [13, 14, 22] save examples from previous tasks into a small-sized buffer to train the modeljointly with the current task.
Revised: Rehearsal-based methods [13, 14, 25] save examples from previous tasks into a small-sized buffer to train the model jointly with the current task.

Original: Though simple in concept, the idea of rehearsal is very effective inpractice and has been adopted by many state-of-the-art methods [8, 11, 45].
Revised: Though simple in concept, the idea of rehearsal is very effective in practice and has been adopted by many state-of-the-art methods [8, 11, 49].

Original: Architecture-basedmethods [38, 47, 53, 54, 57] isolate existing model parameters or assign additional parameters foreach task to reduce interference among tasks.
Revised: Architecture-based methods [42, 51, 57, 59, 63] isolate existing model parameters or assign additional parameters for each task to reduce interference among tasks.

Original: As mentioned in Section 1, most of these methods usea dense model without consideration of efﬁciency and memory footprint, thus are not applicable toresource-limited settings.
Revised: As mentioned in Section 1, most of these methods use a dense model without consideration of efﬁciency and memory footprint, thus are not applicable to resource-limited settings.

Original: Our work, orthogonal to these methods, serves as a general framework formaking these existing methods efﬁcient and enabling a broader deployment, e.g. , CL on edge devices.
Revised: Our work, orthogonal to these methods, serves as a general framework for making these existing methods efﬁcient and enabling a broader deployment, e.g. , CL on edge devices.

Original: A limited number of works explore sparsity in CL, however, for different purposes.
Revised: A limited number of works explore sparsity in CL, however, for different purposes.

Original: Several methods[37, 38, 49, 53] incorporate the idea of weight pruning [21] to allocate a sparse sub-network foreach task to reduce inter-task interference.
Revised: Several methods [41, 42, 53, 57] incorporate the idea of weight pruning [24] to allocate a sparse sub-network for each task to reduce inter-task interference.

Original: Nevertheless, these methods still reduce the full modelsparsity progressively for every task and ﬁnally end up with a much denser model.
Revised: Nevertheless, these methods still reduce the full model sparsity progressively for every task and ﬁnally end up with a much denser model.

Original: On the contrary,
Revised: On the contrary,

Original: SparCL maintains a sparse network throughout the whole CL process, introducing great efﬁciencyand memory beneﬁts both during training and at the output model.
Revised: SparCL maintains a sparse network throughout the whole CL process, introducing great efﬁciency and memory beneﬁts both during training and at the output model.

Original: A recent work [15] aims atdiscovering lottery tickets [20] under CL, but still does not address efﬁciency.
Revised: A recent work [15] aims at discovering lottery tickets [21] under CL, but still does not address efﬁciency.

Original: However, the existenceof lottery tickets in CL serves as a strong justiﬁcation for the outstanding performance of our SparCL.
Revised: However, the existence of lottery tickets in CL serves as a strong justiﬁcation for the outstanding performance of our SparCL.

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: There are two main approaches for sparse training: ﬁxed-mask sparse training and dynamic sparsetraining.
Revised: There are two main approaches for sparse training: ﬁxed-mask sparse training and dynamic sparse training.

Original: Fixed-mask sparse training methods [31, 50, 52, 55] ﬁrst apply pruning, then executetraditional training on the sparse model with the obtained ﬁxed mask.
Revised: Fixed-mask sparse training methods [35, 54, 56, 60] ﬁrst apply pruning, then execute traditional training on the sparse model with the obtained ﬁxed mask.

Original: The pre-ﬁxed structure limitsthe accuracy performance, and the ﬁrst stage still causes huge computation and memory consumption.
Revised: The pre-ﬁxed structure limits the accuracy performance, and the ﬁrst stage still causes huge computation and memory consumption.

Original: To overcome these drawbacks, dynamic mask methods [4, 17, 19, 41, 42] adjust the sparsity topologyduring training while maintaining low memory footprint.
Revised: To overcome these drawbacks, dynamic mask methods [4, 17, 20, 45, 46] adjust the sparsity topology during training while maintaining low memory footprint.

Original: These methods start with a sparse modelstructure from an untrained dense model, then combine sparse topology exploration at the givensparsity ratio with the sparse model training.
Revised: These methods start with a sparse model structure from an untrained dense model, then combine sparse topology exploration at the given sparsity ratio with the sparse model training.

Original: Recent work [61] further considers to incorporate dataefﬁciency into sparse training for better training accelerations.
Revised: Recent work [67] further considers to incorporate data efﬁciency into sparse training for better training accelerations.

Original: However, all prior sparse trainingworks are focused on the traditional training setting, while CL is a more complicated and difﬁcultscenario with inherent characteristics not explored by these works.
Revised: However, all prior sparse training works are focused on the traditional training setting, while CL is a more complicated and difﬁcult scenario with inherent characteristics not explored by these works.

Original: In contrast to prior sparse trainingmethods, our work explores a new learning paradigm that introduces sparse training into CL forefﬁciency and also addresses key challenges in CL, mitigating catastrophic forgetting.
Revised: In contrast to prior sparse training methods, our work explores a new learning paradigm that introduces sparse training into CL for efﬁciency and also addresses key challenges in CL, mitigating catastrophic forgetting.

Original:  Continual Learning Problem Setup
Revised:  Continual Learning Problem Setup

Original: In supervised CL, a model f ✓ learns from a sequence of tasks D
Revised: In supervised CL, a model f ✓ learns from a sequence of tasks D

Original: =
Revised: =

Original: {D 1 , . . .
Revised: {D 1 , . . .

Original: , D T } , where each task
Revised: , D T } , where each task

Original: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.
Revised: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.

Original: Tasksarrive sequentially, and the model must adapt to them.
Revised: Tasks arrive sequentially, and the model must adapt to them.

Original: At the t -th step, the model gains access todata from the t -th task.
Revised: At the t -th step, the model gains access to data from the t -th task.

Original: However, a small ﬁx-sized rehearsal buffer M is allowed to save data fromprior tasks.
Revised: However, a small ﬁx-sized rehearsal buffer M is allowed to save data from prior tasks.

Original: At test time, the easiest setting is to assume task identity is known for each coming testexample, named task-incremental learning (Task-IL).
Revised: At test time, the easiest setting is to assume task identity is known for each coming test example, named task-incremental learning (Task-IL).

Original: If this assumption does not hold, we have themore difﬁcult class-incremental learning (Class-IL) setting.
Revised: If this assumption does not hold, we have the more difﬁcult class-incremental learning (Class-IL) setting.

Original: In this work, we mainly focus on themore challenging Class-IL setting, and only report Task-IL performance for reference.
Revised: In this work, we mainly focus on the more challenging Class-IL setting, and only report Task-IL performance for reference.

Original: The goal of conventional CL is to train a model sequentially that performs well on all tasks at testtime.
Revised: The goal of conventional CL is to train a model sequentially that performs well on all tasks at test time.

Original: The main evaluation metric is average test accuracy on all tasks.
Revised: The main evaluation metric is average test accuracy on all tasks.

Original: In real-world resourcelimited scenarios, we should further consider training efﬁciency of the model.
Revised: In real-world resourcelimited scenarios, we should further consider training efﬁciency of the model.

Original: Thus, we measure theperformance of the model more comprehensively by including training FLOPs and memory footprint.
Revised: Thus, we measure the performance of the model more comprehensively by including training FLOPs and memory footprint.

Original:  Sparse Continual Learning (SparCL)
Revised:  Sparse Continual Learning (SparCL)

Original: Our method, Sparse Continual Learning, is a uniﬁed framework composed of three complementarycomponents: task-aware dynamic masking for weight sparsity, dynamic data removal for dataefﬁciency, and dynamic gradient masking for gradient sparsity.
Revised: Our method, Sparse Continual Learning, is a uniﬁed framework composed of three complementary components: task-aware dynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradient masking for gradient sparsity.

Original: The entire framework is shown in
Revised: The entire framework is shown in

Original: Figure 2.
Revised: Figure 2.

Original: We will illustrate each component in detail in this section.
Revised: We will illustrate each component in detail in this section.

Original: 4.1 Task-aware Dynamic Masking
Revised: 4.1 Task-aware Dynamic Masking

Original: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamicstructure when learning a sequence of tasks, such that it not only achieves high efﬁciency, but alsointelligently adapts to the data stream for better performance.
Revised: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamic structure when learning a sequence of tasks, such that it not only achieves high efﬁciency, but also intelligently adapts to the data stream for better performance.

Original: Speciﬁcally, we propose a strategynamed task-aware dynamic masking (TDM), which dynamically removes less important weightsand grows back unused weights for stronger representation power periodically by maintaining asingle binary weight mask throughout the CL process.
Revised: Speciﬁcally, we propose a strategy named task-aware dynamic masking (TDM), which dynamically removes less important weights and grows back unused weights for stronger representation power periodically by maintaining a single binary weight mask throughout the CL process.

Original: Different from typical sparse training work,which only leverages the weight magnitude [41] or the gradient w.r.t.
Revised: Different from typical sparse training work, which only leverages the weight magnitude [45] or the gradient w.r.t.

Original: data from a single trainingtask [19, 61], TDM considers also the importance of weights w.r.t.
Revised: data from a single training task [20, 67], TDM considers also the importance of weights w.r.t.

Original: data saved in the rehearsal buffer,as well as the switch between CL tasks.
Revised: data saved in the rehearsal buffer, as well as the switch between CL tasks.

Original: Speciﬁcally, TDM strategy starts from a randomly initialized binary mask M ✓ = M 0 , with a givensparsity constraint k M ✓ k 0 / k ✓ k 0 = 1  s , where s 2 [0 , 1] is the sparsity ratio.
Revised: Speciﬁcally, TDM strategy starts from a randomly initialized binary mask M ✓ = M 0 , with a given sparsity constraint k M ✓ k 0 / k ✓ k 0 = 1  s , where s 2 [0 , 1] is the sparsity ratio.

Original: Moreover, it makes149 different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on theircontinual weight importance (CWI).
Revised: Moreover, it makes different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on their continual weight importance (CWI).

Original: We summarize the process of task-aware dynamic masking in
Revised: We summarize the process of task-aware dynamic masking in

Original: Algorithm 1 and elaborate its key components in detail below.
Revised: Algorithm 1 and elaborate its key components in detail below.

Original: Continual weight importance (CWI).
Revised: Continual weight importance (CWI).

Original: For a model f ✓ parameterized by ✓ , the CWI of weight w ⇢ ✓is deﬁned as follows:
Revised: For a model f ✓ parameterized by ✓ , the CWI of weight w ⇢ ✓ is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and ↵ ,  arecoefﬁcients to control the inﬂuence of current and buffered data, respectively.
Revised: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and ↵ ,  are coefﬁcients to control the inﬂuence of current and buffered data, respectively.

Original: Moreover, L representsthe cross-entropy loss for classiﬁcation, while ˜ L is the single-head [1] version of the cross-entropyloss, which only considers classes from the current task by masking out the logits of other classes.
Revised: Moreover, L represents the cross-entropy loss for classiﬁcation, while ˜ L is the single-head [1] version of the cross-entropy loss, which only considers classes from the current task by masking out the logits of other classes.

Original: Intuitively, CWI ensures we keep (1) weights of larger magnitude for output stability, (2) weightsimportant for the current task for learning capacity, and (3) weights important for past data to mitigatecatastrophic forgetting.

Original: Moreover, inspired by the classiﬁcation bias in CL [1], we use the single-headcross-entropy loss when calculating importance score w.r.t.

Original: the current task to make the importanceestimation more accurate.

Original: Input : Model weight ✓ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M ✓ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval  k
Revised: Input : Model weight ✓ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M ✓ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval  k

Original: Initialize: ✓ , M ✓ , s.t.
Revised: Initialize: ✓ , M ✓ , s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original:  s for t = 1 , . .
Revised:  s for t = 1 , . .

Original: , T do for e = 1 , . . .
Revised: , T do for e = 1 , . . .

Original: , K T do if t > 1 then /
Revised: , K T do if t > 1 then /

Original: * Inter-task adjustment */ Expand M ✓ by randomly adding unused weights, s.t.
Revised: * Inter-task adjustment */ Expand M ✓ by randomly adding unused weights, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 1  ( s  p inter ) if e =  k then
Revised: k M ✓ k 0 / k ✓ k 0 = 1  ( s  p inter ) if e =  k then

Original: Shrink M ✓ by removing the least important weights according to equation 1, s.t.
Revised: Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original:  s end endif e mod  k
Revised:  s end endif e mod  k

Original: = 

Revised: =

Original: then /
Revised: then /

Original: * Intra-task adjustment */ Shrink M ✓ by removing the least important weights according to equation 1, s.t.
Revised: * Intra-task adjustment */ Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: k M ✓ k 0 / k ✓ k 0 = 1  ( s + p intra ) Expand M
Revised: k M ✓ k 0 / k ✓ k 0 = 1  ( s + p intra ) Expand M

Original: ✓ by randomly adding unused weights, s.t. k M ✓ k 0 / k ✓ k 0 = 1  s end
Revised: ✓ by randomly adding unused weights, s.t. k M ✓ k 0 / k ✓ k 0 = 1  s end

Original: Update ✓  M ✓ via backpropagation end end
Revised: Update ✓  M ✓ via backpropagation end end

Revised: Intuitively, CWI ensures we keep (1) weights of larger magnitude for output stability, (2) weights important for the current task for learning capacity, and (3) weights important for past data to mitigate catastrophic forgetting.

Revised: Moreover, inspired by the classiﬁcation bias in CL [1], we use the single-head cross-entropy loss when calculating importance score w.r.t.

Revised: the current task to make the importance estimation more accurate.

Original: Intra-task adjustment.
Revised: Intra-task adjustment.

Original: When training the t -th task, a natural assumption is that the data distributionis consistent inside the task, thus we would like to update the sparse model in a relatively stable waywhile keeping its ﬂexibility.
Revised: When training the t -th task, a natural assumption is that the data distribution is consistent inside the task, thus we would like to update the sparse model in a relatively stable way while keeping its ﬂexibility.

Original: Thus, in Algorithm 1, we choose to update the sparsity mask M ✓ in ashrink-and-expand way every  k epochs.
Revised: Thus, in Algorithm 1, we choose to update the sparsity mask M ✓ in a shrink-and-expand way every  k epochs.

Original: We ﬁrst remove p intra of the weights of least CWI to retainlearned knowledge so far.
Revised: We ﬁrst remove p intra of the weights of least CWI to retain learned knowledge so far.

Original: Then we randomly select unused weights to recover the learning capacityfor the model and keep the sparsity ratio s unchanged.
Revised: Then we randomly select unused weights to recover the learning capacity for the model and keep the sparsity ratio s unchanged.

Original: Inter-task adjustment.
Revised: Inter-task adjustment.

Original: When tasks switches, on the contrary, we assume data distribution shiftsimmediately.
Revised: When tasks switches, on the contrary, we assume data distribution shifts immediately.

Original: Ideally, we would like the model to keep the knowledge learned from old tasks as muchas possible, and to have enough learning capacity to accommodate the new task.
Revised: Ideally, we would like the model to keep the knowledge learned from old tasks as much as possible, and to have enough learning capacity to accommodate the new task.

Original: Thus, instead ofthe shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.
Revised: Thus, instead of the shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.

Original: Speciﬁcally, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly addinga proportion of p inter unused weights.
Revised: Speciﬁcally, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly adding a proportion of p inter unused weights.

Original: Intuitively, the additional learning capacity facilitates fastadoption of new knowledge and reduces interference with learned knowledge.
Revised: Intuitively, the additional learning capacity facilitates fast adoption of new knowledge and reduces interference with learned knowledge.

Original: We allow our model tohave smaller sparsity ( i.e. , larger learning capacity) temporarily for the ﬁrst  k epochs as a warmup period, and then remove the p inter weights with least CWI, following the same process in theintra-task case, to satisfy the sparsity constraint.
Revised: We allow our model to have smaller sparsity ( i.e. , larger learning capacity) temporarily for the ﬁrst  k epochs as a warmup period, and then remove the p inter weights with least CWI, following the same process in the intra-task case, to satisfy the sparsity constraint.

Original: 4.2 Dynamic Data Removal
Revised: 4.2 Dynamic Data Removal

Original: In addition to weight sparsity, decreasing the amount of training data can be directly translated intothe saving of training time without any requirements for hardware support.
Revised: In addition to weight sparsity, decreasing the amount of training data can be directly translated into the saving of training time without any requirements for hardware support.

Original: Thus, we would also liketo explore data efﬁciency to reduce the training workload.
Revised: Thus, we would also like to explore data efﬁciency to reduce the training workload.

Original: Some prior CL works select informativeexamples to construct the rehearsal buffer [3, 6, 59].
Revised: Some prior CL works select informative examples to construct the rehearsal buffer [3, 6, 65].

Original: However, the main purpose of them is nottraining acceleration, thus they either introduce excessive computational cost or consider differentproblem settings.
Revised: However, the main purpose of them is not training acceleration, thus they either introduce excessive computational cost or consider different problem settings.

Original: By considering the features of CL, we present a simple yet effective strategy,dynamic data removal (DDR), to reduce training data for further acceleration.
Revised: By considering the features of CL, we present a simple yet effective strategy, dynamic data removal (DDR), to reduce training data for further acceleration.

Original: We measure the importance of each training example by the occurrence of misclassiﬁcation [51, 61]during CL.
Revised: We measure the importance of each training example by the occurrence of misclassiﬁcation [55, 67] during CL.

Original: In TDM, the sparse structure of our model updates periodically every  k epochs, so wealign our data removal process with the update of weight mask for further efﬁciency and training stability.
Revised: In TDM, the sparse structure of our model updates periodically every  k epochs, so we align our data removal process with the update of weight mask for further efﬁciency and training stability.

Original: In Section 4.1, we have partitioned the training process for the t -th task into N t =
Revised: In Section 4.1, we have partitioned the training process for the t -th task into N t =

Original: K t /  kstages based on the dynamic mask update.
Revised: K t /  k stages based on the dynamic mask update.

Original: Therefore, we gradually remove training data at the end ofi -th stage, based on the following policy: 1) Calculate the total number of misclassiﬁcations f i ( x j )for each training example during the i -th stage.
Revised: Therefore, we gradually remove training data at the end of i -th stage, based on the following policy: 1) Calculate the total number of misclassiﬁcations f i ( x j ) for each training example during the i -th stage.

Original: 2) Remove a proportion of ⇢ i training samples withthe least number of misclassiﬁcations.
Revised: 2) Remove a proportion of ⇢ i training samples with the least number of misclassiﬁcations.

Original: Although our main purpose is to keep the “harder” examplesto learn to consolidate the sparse model, we can get further beneﬁts for better CL result.
Revised: Although our main purpose is to keep the “harder” examples to learn to consolidate the sparse model, we can get further beneﬁts for better CL result.

Original: First, theremoval of “easier” examples increases the probability that “harder” examples to be saved to therehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples.
Revised: First, the removal of “easier” examples increases the probability that “harder” examples to be saved to the rehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples.

Original: Thus,we construct a more informative buffer in a implicit way without heavy computation.
Revised: Thus, we construct a more informative buffer in a implicit way without heavy computation.

Original: Moreover, sincethe buffer size is much smaller than the training set size of each task, the data from the buffer and thenew task is highly imbalanced, dynamic data removal also relieves the data imbalance issue.
Revised: Moreover, since the buffer size is much smaller than the training set size of each task, the data from the buffer and the new task is highly imbalanced, dynamic data removal also relieves the data imbalance issue.

Original: Formally, we set the data removal proportion for each task as ⇢ 2 [0 , 1] , and a cutoff stage, such that:
Revised: Formally, we set the data removal proportion for each task as ⇢ 2 [0 , 1] , and a cutoff stage, such that:

Original: [Equation]
Revised: [Equation]

Original: The cutoff stage controls the trade-off between efﬁciency and accuracy: When we set the cutoff stageearlier, we reduce the training time for all the following stages; however, when the cutoff stage is settoo early, the model might underﬁt the removed training data.
Revised: The cutoff stage controls the trade-off between efﬁciency and accuracy: When we set the cutoff stage earlier, we reduce the training time for all the following stages; however, when the cutoff stage is set too early, the model might underﬁt the removed training data.

Original: Note that when we set ⇢ i = 0 for alli = 1 , 2 , . . .
Revised: Note that when we set ⇢ i = 0 for all i = 1 , 2 , . . .

Original: , N t and cutoff =
Revised: , N t and cutoff =

Original: N t , we simply recover the vanilla setting without any data efﬁciencyconsiderations.
Revised: N t , we simply recover the vanilla setting without any data efﬁciency considerations.

Original: In our experiments, we assume ⇢ i = ⇢ / cutoff , i.e., removing equal proportion ofdata at the end of every stage, for simplicity.
Revised: In our experiments, we assume ⇢ i = ⇢ / cutoff , i.e., removing equal proportion of data at the end of every stage, for simplicity.

Original: We also conduct comprehensive exploration study for ⇢and the selection of the cutoff stage in Section 5.3 and Appendix D.3.
Revised: We also conduct comprehensive exploration study for ⇢ and the selection of the cutoff stage in Section 5.3 and Appendix B.3.

Original: 4.3 Dynamic Gradient Masking
Revised: 4.3 Dynamic Gradient Masking

Original: With TDM and DDR, we can already achieve bi-level efﬁciency during training.
Revised: With TDM and DDR, we can already achieve bi-level efﬁciency during training.

Original: To further boosttraining efﬁciency, we explore sparsity in gradient and propose dynamic gradient masking (DGM)for CL.
Revised: To further boost training efﬁciency, we explore sparsity in gradient and propose dynamic gradient masking (DGM) for CL.

Original: Our method focuses on reducing computational cost by only applying the most importantgradients onto the corresponding unpruned model parameters via a gradient mask.
Revised: Our method focuses on reducing computational cost by only applying the most important gradients onto the corresponding unpruned model parameters via a gradient mask.

Original: The gradient maskis also dynamically updated along with the weight mask deﬁned in Section 4.1.
Revised: The gradient mask is also dynamically updated along with the weight mask deﬁned in Section 4.1.

Original: Intuitively, whiletargeting for better training efﬁciency, DGM also promotes the preservation of past knowledge bypreventing a fraction of weights from update.
Revised: Intuitively, while targeting for better training efﬁciency, DGM also promotes the preservation of past knowledge by preventing a fraction of weights from update.

Original: Formally, our goal here is to ﬁnd a subset of unpruned parameters (or, equivalently, a gradient mask
Revised: Formally, our goal here is to ﬁnd a subset of unpruned parameters (or, equivalently, a gradient mask

Original: M G ) to update over multiple training iterations.
Revised: M G ) to update over multiple training iterations.

Original: For a model f ✓ parameterized by ✓ , we have thecorresponding gradient matrix G calculated during each iteration.
Revised: For a model f ✓ parameterized by ✓ , we have the corresponding gradient matrix G calculated during each iteration.

Original: To prevent the pruned weightsfrom updating, the weight mask M ✓ will be applied onto the gradient matrix G as G  M ✓ duringbackpropagation.
Revised: To prevent the pruned weights from updating, the weight mask M ✓ will be applied onto the gradient matrix G as G  M ✓ during backpropagation.

Original: Besides the gradients of pruned weights, we in addition consider to remove lessimportant gradients for faster training.
Revised: Besides the gradients of pruned weights, we in addition consider to remove less important gradients for faster training.

Original: To achieve this, we introduce the continual gradient importance(CGI) based on the CWI to measure the importance of weight gradients.
Revised: To achieve this, we introduce the continual gradient importance (CGI) based on the CWI to measure the importance of weight gradients.

Original: [Equation]
Revised: [Equation]

Original: We remove a proportion q of non-zero gradients from G with less importance measured by CGIand we have k M G k 0 / k ✓ k 0 = 1  ( s + q ) .
Revised: We remove a proportion q of non-zero gradients from G with less importance measured by CGI and we have k M G k 0 / k ✓ k 0 = 1  ( s + q ) .

Original: The gradient mask M G is then applied onto the gradient226 matrix G .
Revised: The gradient mask M G is then applied onto the gradient matrix G .

Original: During the entire training process, the gradient mask M G is updated with a ﬁxed interval.
Revised: During the entire training process, the gradient mask M G is updated with a ﬁxed interval.

Original:  Experiment 
Revised:  Experiment 5.1 Experiment Setting

Original: 5.1 Experiment Setting

Original: Datasets.
Revised: Datasets.

Original: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [29]and Split Tiny-ImageNet [16] to verify the efﬁcacy of SparCL.
Revised: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [33] and Split Tiny-ImageNet [16] to verify the efﬁcacy of SparCL.

Original: In particular, we follow [8, 62] bysplitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classesrespectively.
Revised: In particular, we follow [8, 68] by [Table] splitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classes respectively.

Original: Dataset licensing information can be found in Appendix C.
Revised: Dataset licensing information can be found in Appendix A.

Original: Comparing methods.
Revised: Comparing methods.

Original: In particular, we select CL methods of all kinds including regularizationbased (EWC [28], LwF [33]), architecture-based (PackNet [38], LPS [53]), and rehearsal-based
Revised: In particular [Table] we select CL methods of all kinds including regularizationbased (EWC [32], LwF [37]), architecture-based (PackNet [42], LPS [57])

Original: [Table]
Revised: [Table]

Original:  (A-GEM [13], iCaRL [40], FDR [5], ER [14], DER++ [8]) methods.
Revised: and rehearsal-based (A-GEM [13], iCaRL [44], FDR [5], ER [14], DER++ [8]) methods.

Original: Note that PackNet and LPSare only compatible with task-incremental learning.
Revised: Note that PackNet and LPS are only compatible with task-incremental learning.

Original: We also adapt representative sparse trainingmethods (SNIP [31], RigL [19]) to the CL setting by combining them with DER++ (SNIP-DER++,
Revised: We also adapt representative sparse training methods (SNIP [35], RigL [20]) to the CL setting by combining them with DER++ (SNIP-DER++,

Original: RigL-DER++).
Revised: RigL-DER++).

Original: Variants of our method.
Revised: Variants of our method.

Original: To show the generality of SparCL, we combine it with DER++ (one ofthe SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER ,respectively.
Revised: To show the generality of SparCL, we combine it with DER++ (one of the SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER , respectively.

Original: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensiveevaluation.
Revised: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensive evaluation.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use the average accuracy on all tasks to evaluate the performance of the ﬁnalmodel.
Revised: We use the average accuracy on all tasks to evaluate the performance of the ﬁnal model.

Original: Moreover, we evaluate the training FLOPs [19], and memory footprint [61] (including featuremap pixels and model parameters during training) to demonstrate the efﬁciency of each method.
Revised: Moreover, we evaluate the training FLOPs [20], and memory footprint [67] (including feature map pixels and model parameters during training) to demonstrate the efﬁciency of each method.

Original: Please see Appendix D.1 for detailed deﬁnitions of these metrics.
Revised: Please see Appendix B.1 for detailed deﬁnitions of these metrics.

Original: Experiment details.
Revised: Experiment details.

Original: For fair comparison, we strictly follow the settings in prior CL work [8, 26].
Revised: For fair comparison, we strictly follow the settings in prior CL work [8, 29].

Original: Wesets the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively,with a batch size of 32 .
Revised: We sets the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively, with a batch size of 32 .

Original: For the model architecture, We follow [8, 46] and adopt the ResNet-18 [23]without any pre-training.
Revised: For the model architecture, We follow [8, 50] and adopt the ResNet-18 [26] without any pre-training.

Original: We also use the best hyperparameter setting reported in [8, 53] for CLmethods, and in [19, 31] for CL-adapted sparse training methods.
Revised: We also use the best hyperparameter setting reported in [8, 57] for CL methods, and in [20, 35] for CL-adapted sparse training methods.

Original: For SparCL and its competing
Revised: For SparCL and its competing

Original: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.
Revised: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.

Original: Please see Appendix D for other details.
Revised: Please see Appendix B for other details.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: Comparison with CL methods.
Revised: Comparison with CL methods.

Original: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.
Revised: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.

Original: From
Revised: From

Original:  Table 1, we can clearly tell that SparCL signiﬁcantly improves ER and DER++, while also outperformsother CL baselines, in terms of training efﬁciency (measured in FLOPs).
Revised: [Table] [Table] Table 1, we can clearly tell that SparCL signiﬁcantly improves ER and DER++, while also outperforms other CL baselines, in terms of training efﬁciency (measured in FLOPs).

Original: With higher sparsity ratio,
Revised: With higher sparsity ratio,

Original: SparCL leads to less training FLOPs.
Revised: SparCL leads to less training FLOPs.

Original: Notably, SparCL achieves 23 ⇥ training efﬁciency improvementupon DER++ with a sparsity ratio of 0.95.
Revised: Notably, SparCL achieves 23 ⇥ training efﬁciency improvement upon DER++ with a sparsity ratio of 0.95.

Original: On the other hand, our framework also improves theaverage accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90,[Table] [Table] and only slight performance drop when sparsity gets larger as 0.95.
Revised: On the other hand, our framework also improves the average accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90, and only slight performance drop when sparsity gets larger as 0.95.

Original: In particular, SparCL-DER++with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.
Revised: In particular, SparCL-DER++ with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.

Original: Theoutstanding performance of SparCL indicates that our proposed strategies successfully preserveaccuracy by further mitigating catastrophic forgetting with a much sparser model.
Revised: The outstanding performance of SparCL indicates that our proposed strategies successfully preserve accuracy by further mitigating catastrophic forgetting with a much sparser model.

Original: Moreover, theimprovement that SparCL brings to two different existing CL methods shows the generalizability of
Revised: Moreover, the improvement that SparCL brings to two different existing CL methods shows the generalizability of

Original: SparCL as a uniﬁed framework, i.e. , it has the potential to be combined with a wide array of existingmethods.
Revised: SparCL as a uniﬁed framework, i.e. , it has the potential to be combined with a wide array of existing methods.

Original: We would also like to take a closer look at PackNet and LPS, which also leverage the idea of sparsityto split the model by different tasks, a different motivation from training efﬁciency.
Revised: We would also like to take a closer look at PackNet and LPS, which also leverage the idea of sparsity to split the model by different tasks, a different motivation from training efﬁciency.

Original: Firstly, they areonly compatible with the Task-IL setting, since they leverage task identity at both training and testtime.
Revised: Firstly, they are only compatible with the Task-IL setting, since they leverage task identity at both training and test time.

Original: Moreover, the model sparsity of these methods reduces with the increasing number of tasks,which still leads to much larger overall training FLOPs than that of SparCL.
Revised: Moreover, the model sparsity of these methods reduces with the increasing number of tasks, which still leads to much larger overall training FLOPs than that of SparCL.

Original: This further demonstratesthe importance of keeping a sparse model without permanent expansion throughout the CL process.
Revised: This further demonstrates the importance of keeping a sparse model without permanent expansion throughout the CL process.

Original: Comparison with CL-adapted sparse training methods.
Revised: Comparison with CL-adapted sparse training methods.

Original: Table 2 shows the result under themore difﬁcult Class-IL setting.
Revised: Table 2 shows the result under the more difﬁcult Class-IL setting.

Original: SparCL outperforms all CL-adapted sparse training methods in bothaccuracy and training FLOPs.
Revised: SparCL outperforms all CL-adapted sparse training methods in both accuracy and training FLOPs.

Original: The performance gap between SparCL-DER++ and other methodsgets larger with a higher sparsity.
Revised: The performance gap between SparCL-DER++ and other methods gets larger with a higher sparsity.

Original: SNIP-
Revised: SNIP-

Original: and RigL-DER++ achieve training acceleration at the cost ofcompromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existingsparse training methods under the CL setting.
Revised: and RigL-DER++ achieve training acceleration at the cost of compromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existing sparse training methods under the CL setting.

Original: SNIP generates the static initial mask after networkinitialization which does not consider the structure suitability among tasks.
Revised: SNIP generates the static initial mask after network initialization which does not consider the structure suitability among tasks.

Original: Though RigL adopts adynamic mask, the lack of task-aware strategy prevents it from generalizing well to the CL setting.
Revised: Though RigL adopts a dynamic mask, the lack of task-aware strategy prevents it from generalizing well to the CL setting.

Original: 5.3 Effectiveness of Key Components
Revised: 5.3 Effectiveness of Key Components

Original: Ablation study.
Revised: Ablation study.

Original: We provide a comprehensive ablation study in Table 3 using SparCL-DER++ with0.75 sparsity on Split CIFAR10.
Revised: We provide a comprehensive ablation study in Table 3 using SparCL-DER++ with 0.75 sparsity on Split CIFAR10.

Original: Table 3 demonstrates that all components of our method contributeto both efﬁciency and accuracy improvements.
Revised: Table 3 demonstrates that all components of our method contribute to both efﬁciency and accuracy improvements.

Original: Comparing row 1 and 2, we can see that the majorityof FLOPs decrease results from TDM.
Revised: Comparing row 1 and 2, we can see that the majority of FLOPs decrease results from TDM.

Original: Interestingly, TDM leads to an increase in accuracy, indicating
Revised: Interestingly, TDM leads to an increase in accuracy, indicating

Original: TDM generates a sparse model that is even more suitable for learning all tasks than then full densemodel.
Revised: TDM generates a sparse model that is even more suitable for learning all tasks than then full dense model.

Original: Comparing row 2 and 3, we can see that DDR indeed further accelerates training by removingless informative examples.
Revised: Comparing row 2 and 3, we can see that DDR indeed further accelerates training by removing less informative examples.

Original: As discussed in Section 4.2, when we remove a certain number of samples(30% here), we achieve a point where we keep as much informative samples as we need, and alsobalance the current and buffered data.
Revised: As discussed in Section 4.2, when we remove a certain number of samples (30% here), we achieve a point where we keep as much informative samples as we need, and also balance the current and buffered data.

Original: Comparing row 2 and 4, DGM reduce both training FLOPs andmemory footprint while improve the performance of the network.
Revised: Comparing row 2 and 4, DGM reduce both training FLOPs and memory footprint while improve the performance of the network.

Original: Finally, the last row demonstratesthe collaborative performance of all components.
Revised: Finally, the last row demonstrates the collaborative performance of all components.

Original: We also show the same ablation study with 0.90sparsity in Appendix D.4 for reference.
Revised: We also show the same ablation study with 0. sparsity in Appendix B.4 for reference.

Original: Detail can be found in Appendix D.1.
Revised: Detail can be found in Appendix B.1.

Original: Exploration on DDR.
Revised: Exploration on DDR.

Original: To understand the inﬂuence of the data removal proportion ⇢ , and the cutoffstage for each task, we show corresponding experiment results in Figure 3 and Appendix D.3,respectively.
Revised: To understand the inﬂuence of the data removal proportion ⇢ , and the cutoff stage for each task, we show corresponding experiment results in Figure 3 and Appendix B.3, respectively.

Original: In Figure 3, we ﬁx cutoff = 4 , i.e. , gradually removing equal number of examplesevery 5 epochs until epoch 20, and vary ⇢ from 10% to 90% .
Revised: In Figure 3, we ﬁx cutoff = 4 , i.e. , gradually removing equal number of examples every 5 epochs until epoch 20, and vary ⇢ from 10% to 90% .

Original: We also compare DDR with One-shotremoval strategy [61], which removes all examples at once at cutoff .
Revised: We also compare DDR with One-shot removal strategy [67], which removes all examples at once at cutoff .

Original: DDR outperforms One-shotconsistently with different ⇢ in average accuracy.
Revised: DDR outperforms One-shot consistently with different ⇢ in average accuracy.

Original: Also note that since DDR removes the examplesgradually before the cutoff stage, DDR is more efﬁcient than One-shot.
Revised: Also note that since DDR removes the examples gradually before the cutoff stage, DDR is more efﬁcient than One-shot.

Original: When ⇢  30% , we alsoobserve increased accuracy of DDR compared with the baseline without removing any data.
Revised: When ⇢  30% , we also observe increased accuracy of DDR compared with the baseline without removing any data.

Original: When⇢  40% , the accuracy gets increasingly lower for both strategies.
Revised: When ⇢  40% , the accuracy gets increasingly lower for both strategies.

Original: The intuition is that when DDRremoves a proper amount of data, it removes redundant information while keeps the most informativeexamples.
Revised: The intuition is that when DDR removes a proper amount of data, it removes redundant information while keeps the most informative examples.

Original: Moreover, as discussed in Section 4.2, it balances the current and buffered data, whilealso leave informative samples in the buffer.
Revised: Moreover, as discussed in Section 4.2, it balances the current and buffered data, while also leave informative samples in the buffer.

Original: When DDR removes too much data, it will also loseinformative examples, thus the model has not learned these examples well before removal.
Revised: When DDR removes too much data, it will also lose informative examples, thus the model has not learned these examples well before removal.

Original: Exploration on DGM.
Revised: Exploration on DGM.

Original: We test the efﬁcacy of DGM at different sparsity levels.
Revised: We test the efﬁcacy of DGM at different sparsity levels.

Original: Detailed exploratoryexperiments are shown in Appendix D.5 for reference.
Revised: Detailed exploratory experiments are shown in Appendix B.5 for reference.

Original: The results indicate that by setting theproportion q within an appropriate range, DGM can consistently improve the accuracy performanceregardless of the change of weight sparsity.
Revised: The results indicate that by setting the proportion q within an appropriate range, DGM can consistently improve the accuracy performance regardless of the change of weight sparsity.

Original: 5.4 Mobile Device Results
Revised: 5.4 Mobile Device Results

Original: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S20smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo[-
-]
Revised: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo[-
-]

Original: Octa-core CPU.
Revised: Octa-core CPU.

Original: We run each test on a batch of 32 images to denote the training speed.
Revised: We run each test on a batch of 32 images to denote the training speed.

Original: The detail ofon-mobile compiler-level optimizations for training acceleration can be found in Appendix E.1.
Revised: The detail of on-mobile compiler-level optimizations for training acceleration can be found in Appendix C.1.

Original: The acceleration results are shown in Figure 4.
Revised: The acceleration results are shown in Figure 4.

Original: SparCL can achieve approximately 3.1 ⇥ and 2.3 ⇥training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.
Revised: SparCL can achieve approximately 3.1 ⇥ and 2.3 ⇥ training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.

Original: Besides, our framework canalso save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.
Revised: Besides, our framework can also save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.

Original: Furthermore, theobtained sparse models save the storage consumption by using compressed sparse row (CSR) storageand can be accelerated to speed up the inference on-the-edge.
Revised: Furthermore, the obtained sparse models save the storage consumption by using compressed sparse row (CSR) storage and can be accelerated to speed up the inference on-the-edge.

Original: We provide on-mobile inferenceacceleration results in Appendix E.2.
Revised: We provide on-mobile inference acceleration results in Appendix C.2.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presents a uniﬁed framework named SparCL for efﬁcient CL that achieves both learningacceleration and accuracy preservation.
Revised: This paper presents a uniﬁed framework named SparCL for efﬁcient CL that achieves both learning acceleration and accuracy preservation.

Original: It comprises three complementary strategies: task-awaredynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradientmasking for gradient sparsity.
Revised: It comprises three complementary strategies: task-aware dynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradient masking for gradient sparsity.

Original: Extensive experiments on standard CL benchmarks and real-world edgedevice evaluations demonstrate that our method signiﬁcantly improves upon existing CL methods andoutperforms CL-adapted sparse training methods.
Revised: Extensive experiments on standard CL benchmarks and real-world edge device evaluations demonstrate that our method signiﬁcantly improves upon existing CL methods and outperforms CL-adapted sparse training methods.

Original: We discuss the limitations and potential negativesocial impacts of our method in Appendix A and B, respectively.  
Revised: We discuss the limitations and potential negative social impacts of our method in Section 7 and 8, respectively.  Limitations

Revised: One limitation of our method is that we assume a rehearsal buffer is available throughout the CL process.

Revised: Although the assumption is widely-accepted, there are still situations that a rehearsal buffer is not allowed.

Revised: However, as a framework targeting for efﬁciency, our work has the potential to accelerate all types of CL methods.

Original: Checklist
Revised: For example, simply removing the terms related to rehearsal buffer in equation 1 and equation 3 could serve as a naive variation of our method that is compatible with other non-rehearsal methods. It is interesting to further improve SparCL to be more generic for all kinds of CL methods. Moreover, the benchmarks we use are limited to vision domain. Although using vision-based benchmarks has been a common practice in the CL community, we believe evaluating our method, as well as other CL methods, on datasets from other domains such as NLP will lead to a more comprehensive and reliable conclusion.

Revised: We will keep track of newer CL benchmarks from different domains and further improve our work correspondingly.

Revised:  Potential Negative Societal Impact

Original:  For all authors...
Revised: Although SparCL is a general framework to enhance efﬁciency for various CL methods, we still need to be aware of its potential negative societal impact. For example, we need to be very careful about the trade-off between accuracy and efﬁciency when using SparCL. If one would like to pursue efﬁciency by setting the sparsity ratio too high, then even SparCL will result in signiﬁcant accuracy drop, since the over-sparsiﬁed model does not have enough representation power. Thus, we should pay much attention when applying SparCL on accuracy-sensitive applications such as healthcare [66].

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: Another example is that, SparCL as a powerful tool to make CL methods efﬁcient, can also strengthen models for malicious applications [7]. Therefore, we encourage the community to come up with more strategies and regulations to prevent malicious use of artiﬁcial intelligence  Acknowledgement The authors gratefully acknowledge support by the National Science Foundation under grants CCF1937500 and CNS-

Original:  contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] The claims match the experimental results and it isexpected to generalize according to the diverse experiments stated in our paper.
Revised: [Yes] The claims match the experimental results and it is expected to generalize according to the diverse experiments stated in our paper.

Original: We include all of our code, data, and models in the supplementary materials, which can reproduce our experimental results.
Revised: We include all of our code, data, and models in the supplementary materials, which can reproduce our experimental results.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6 and Appendix A.
Revised: [Yes] See Section 6 and Appendix 7.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 6 and Appendix B.
Revised: [Yes] See Section 6 and Appendix 8.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
Revised: [Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: Our paper is based on the experimental results and we do not have any theoretical results.
Revised: Our paper is based on the experimental results and we do not have any theoretical results.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: Our paper is based on the experimental results and we do not have any theoretical results.
Revised: Our paper is based on the experimental results and we do not have any theoretical results.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section 5.1, Section 5.4 and we provide code to reproduce the main experimental results.
Revised: [Yes] See Section 5.1, Section 5.4 and we provide code to reproduce the main experimental results.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1 and Section 5.4.
Revised: [Yes] See Section 5.1 and Section 5.4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Table 1, Table 2, ﬁg 1, ﬁg 3.
Revised: [Yes] See Table 1, Table 2, ﬁg 1, ﬁg 3.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 5.1, Section 5.4. 
Revised: [Yes] See Section 5.1, Section 5.4. 4.

Original: [Table] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...[Table] (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We mentioned and

Original: [Table]

Original:  cited the datasets (Split CIFAR-10 and Tiny-ImageNet), and all comparing methods[Table] with their paper and github in it.[Table]
Revised: [Yes] We mentioned and cited the datasets (Split CIFAR-10 and Tiny-ImageNet), and all comparing methods with their paper and github in it.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The licences of used datasets/models

Original: [Table] are provided in the cited references and we state them explicitly in Appendix C. [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] The licences of used datasets/models are provided in the cited references and we state them explicitly in Appendix A. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes][Table] We provide code for our proposed method in the supplement.[Table]
Revised: [Yes] We provide code for our proposed method in the supplement.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?[N/A]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable[Table] information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] [Table]
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable?

Original: [N/A]

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]