Original: SparCL: Sparse Continual Learning on the Edge
Revised: SparCL: Sparse Continual Learning on the Edge

Original: Abstract
Revised: Abstract

Original: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a new task.
Revised: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e. , model performance deterioration on past tasks when learning a new task.

Original: However, the training efficiency of a CL system is under-investigated, which limits the real-world application of CL systems under resource-limited scenarios.
Revised: However, the training efﬁciency of a CL system is under-investigated, which limits the real-world application of CL systems under resource-limited scenarios.

Original: In this work, we propose a novel framework called Sparse Continual Learning (SparCL), which is the first study that leverages sparsity to enable cost-effective continual learning on edge devices.
Revised: In this work, we propose a novel framework called Sparse Continual Learning (SparCL), which is the ﬁrst study that leverages sparsity to enable cost-effective continual learning on edge devices.

Original: SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efficiency , and gradient sparsity .
Revised: SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efﬁciency , and gradient sparsity .

Original: Specifically, we propose task-aware dynamic masking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates.
Revised: Speciﬁcally, we propose task-aware dynamic masking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates.

Original: Each of them not only improves efficiency, but also further mitigates catastrophic forgetting.
Revised: Each of them not only improves efﬁciency, but also further mitigates catastrophic forgetting.

Original: SparCL consistently improves the training efficiency of existing state-of-the-art (SOTA) CL methods by at most 23 × less training FLOPs, and, surprisingly, further improves the SOTA accuracy by at most 1 . 7% .
Revised: SparCL consistently improves the training efﬁciency of existing state-of-the-art (SOTA) CL methods by at most 23 ⇥ less training FLOPs, and, surprisingly, further improves the SOTA accuracy by at most 1 . 7% .

Original: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting in both efficiency and accuracy.
Revised: SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting in both efﬁciency and accuracy.

Original: We also evaluate the effectiveness of SparCL on a real mobile phone, further indicating the practical potential of our method.
Revised: We also evaluate the effectiveness of SparCL on a real mobile phone , further indicating the practical potential of our method.

Original:  Introduction
Revised:  Introduction

Original: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledge from a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [32].
Revised: The objective of Continual Learning (CL) is to enable an intelligent system to accumulate knowledge from a sequence of tasks, such that it exhibits satisfying performance on both old and new tasks [32].

Original: Recent methods mostly focus on addressing the catastrophic forgetting [43] problem – learning model tends to suffer performance deterioration on previously seen tasks.
Revised: Recent methods mostly focus on addressing the catastrophic forgetting [43] problem – learning model tends to suffer performance deterioration on previously seen tasks.

Original: However, in the real world, whenthe CL applications are deployed in resource-limited platforms [48] such as edge devices, the learning efficiency, w.r.t.
Revised: However, in the real world, when CL applications are deployed in resource-limited platforms [48] such as edge devices, the learning efﬁciency, w.r.t.

Original: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.
Revised: both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.

Original: Existing CL methods can be categorized into regularization-based [2, 32, 37, 68], rehearsal-based [8, 12, 50, 61], and architecture-based [31, 42, 52, 58, 59, 70].
Revised: Existing CL methods can be categorized into regularization-based [2, 32, 37, 68], rehearsal-based [8, 12, 50, 61], and architecture-based [31, 42, 52, 58, 59, 70].

Original: Both regularization- and rehearsal-based methods directly train a dense model, which mighteven be over-parametrized for the union of all tasks [19, 39]; Though several architecture-based methods [51, 57, 64] start with a sparse sub-network from the dense model, they still grow the model size progressively to learn emerging tasks.
Revised: Both regularization- and rehearsal-based methods directly train a dense model, which might be over-parameterized even for the union of all tasks [19, 39]. Though several architecture-based methods [51, 57, 64] start with a sparse sub-network from the dense model, they still grow the model size progressively to learn emerging tasks.

Original: The aforementioned methods, although striving for greater performance with less forgetting, still introduce significant memory and computation overhead during the whole CL process.
Revised: The aforementioned methods, although striving for greater performance with less forgetting, still introduce signiﬁcant memory and computation overhead during the whole CL process.

Original: Recently, another stream of work, sparse training [4, 20, 35] has emerged as a new training trend to achieve training acceleration, which embraces the promising training-on-the-edge paradigm.
Revised: Recently, another stream of work, sparse training [4, 20, 35], has emerged as a new training trend to achieve training acceleration, which embraces the promising training-on-the-edge paradigm.

Original: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity, under the traditional i.i.d.
Revised: With sparse training, each iteration takes less time with the reduction in computation achieved by sparsity.

Original: learning setting. Inspired by these sparse training methods,  we naturally think about introducing sparse training to the field of CL.
Revised: Inspired by these sparse training methods, under the traditional i.i.d. learning setting, we naturally think about introducing sparse training to the ﬁeld of CL.

Original: A straightforward idea is to directly combine existing sparse training methods, such as SNIP [35], RigL [20], with a rehearsal buffer under the
Revised: A straightforward idea is to directly combine existing sparse training methods, such as SNIP [35], RigL [20], with a rehearsal buffer under the

Original: CL setting.
Revised: CL setting.

Original: However, these methods fail to consider key challenges in CL to mitigate catastrophic forgetting, for example, properly handling transition between tasks.
Revised: However, these methods fail to consider key challenges in CL to mitigate catastrophic forgetting, for example, properly handling transition between tasks.

Original: As a result, these sparse training methods, though enhancing training efficiency, cause significant accuracy drop (see Section 5.2).
Revised: As a result, these sparse training methods, though enhancing training efﬁciency, cause signiﬁcant accuracy drop (see Section 5.2).

Original: Thus, we would like to explore a general strategy,which is orthogonal to existing CL methods, that not only leverages the idea of sparse training for efficiency, but also addresses key challenges in CL to preserve (or even improve) accuracy.
Revised: Thus, we would like to explore a general strategy, orthogonal to existing CL methods, that not only leverages the idea of sparse training for efﬁciency, but also addresses key challenges in CL to preserve (or even improve) accuracy.

Original: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, aiming at enabling practical CL on edge devices.
Revised: In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, aiming at enabling practical CL on edge devices.

Original: As shown in Figure 1 (left),
Revised: As shown in Figure 1 (left),

Original: SparCL achieves both learning acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efficiency , and gradient sparsity .
Revised: SparCL achieves both learning acceleration and accuracy preservation through the synergy of three aspects: weight sparsity , data efﬁciency , and gradient sparsity .

Original: Specifically, to maintain a small dynamic sparse network during the whole CL process, we develop a novel task-aware dynamic masking (TDM) strategy to keep only important weights for both the current and past tasks, with special consideration during task transitions.
Revised: Speciﬁcally, to maintain a small dynamic sparse network during the whole CL process, we develop a novel task-aware dynamic masking (TDM) strategy to keep only important weights for both the current and past tasks, with special consideration during task transitions.

Original: Moreover, we propose a dynamic data removal (DDR) scheme, which progressively removes “easy-to-learn” examples from training iterations, which further accelerates the training process and also improves accuracy of CL by balancing current and past data and keeping more informative samples in the buffer.
Revised: Moreover, we propose a dynamic data removal (DDR) scheme, which progressively removes “easy-to-learn” examples from training iterations, which further accelerates the training process and also improves accuracy of CL by balancing current and past data and keeping more informative samples in the buffer.

Original: Finally, we provide an additional dynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efficiency and knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.
Revised: Finally, we provide an additional dynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efﬁciency and knowledge preservation of learned tasks, such that only a subset of sparse weights are updated.

Original: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and significantly improves efficiency over DER++ [8], one of the SOTA CL methods, under different sparsity ratios.
Revised: Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and signiﬁcantly improves efﬁciency over DER++ [8], one of the SOTA CL methods, under different sparsity ratios.

Original: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, and efficient under practical scenarios.
Revised: SparCL is simple in concept, compatible with various existing rehearsal-based CL methods, and efﬁcient under practical scenarios.

Original: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.
Revised: We conduct comprehensive experiments on multiple CL benchmarks to evaluate the effectiveness of our method.

Original: We show that SparCL works collaboratively with existing CL methods, greatly accelerates the learning process under different sparsity ratios, and even sometimes improves upon the state-of-the-art accuracy.
Revised: We show that SparCL works collaboratively with existing CL methods, greatly accelerates the learning process under different sparsity ratios, and even sometimes improves upon the state-of-the-art accuracy.

Original: We also establish competing baselines by combining representative sparse training methods with advanced rehearsal-based CL methods.
Revised: We also establish competing baselines by combining representative sparse training methods with advanced rehearsal-based CL methods.

Original: SparCL again outperforms these baselines in terms of both efficiency and accuracy.
Revised: SparCL again outperforms these baselines in terms of both efﬁciency and accuracy.

Original: Most importantly, we evaluate our SparCL framework on real edge devices to demonstrate the practical potential of our method.
Revised: Most importantly, we evaluate our SparCL framework on real edge devices to demonstrate the practical potential of our method.

Original: We are not aware of any prior CL works that explored this area and considered the constraints of limited resources during training.
Revised: We are not aware of any prior CL works that explored this area and considered the constraints of limited resources during training.

Original: In summary, our work makes the following contributions:
Revised: In summary, our work makes the following contributions:

Original: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, which achieves learning acceleration through the synergy of weight sparsity , data effi-
Revised: • We propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, which achieves learning acceleration through the synergy of weight sparsity , data efﬁ-

Original: ciency , and gradient sparsity .
Revised: ciency , and gradient sparsity .

Original: To the best of our knowledge, our work is the first to introduce the idea of sparse training to enable efficient CL on edge devices.
Revised: To the best of our knowledge, our work is the ﬁrst to introduce the idea of sparse training to enable efﬁcient CL on edge devices.

Original: Our code is publicly available † .
Revised: Our code is publicly available ⇤ .

Original: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted sparse training methods on all benchmark datasets, leading to at most 23 × less training FLOPs and, surprisingly,  1 . 7% improvement over SOTA accuracy.
Revised: • SparCL shows superior performance compared to both conventional CL methods and CL-adapted sparse training methods on all benchmark datasets, leading to at most 23 ⇥ less training FLOPs and, surprisingly, an 1 . 7% improvement over SOTA accuracy.

Original: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of our method and also encouraging future research on CL on-the-edge.
Revised: • We evaluate SparCL on a real mobile edge device, demonstrating the practical potential of our method and also encouraging future research on CL on-the-edge.

Original: The results indicate that our framework can achieve at most 3 . 1 × training acceleration.
Revised: The results indicate that our framework can achieve at most 3 . 1 ⇥ training acceleration.

Original:  Related work 2.1 Continual Learning
Revised:  Related work 2.1 Continual Learning

Original: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.
Revised: The main focus in continual learning (CL) has been mitigating catastrophic forgetting.

Original: Existing methods can be classified into three major categories.
Revised: Existing methods can be classiﬁed into three major categories.

Original: Regularization-based methods [2, 32, 37, 68] limit updates of important parameters for the prior tasks by adding corresponding regularization terms.
Revised: Regularization-based methods [2, 32, 37, 68] limit updates of important parameters for the prior tasks by adding corresponding regularization terms.

Original: While these methods reduce catastrophic forgetting to some extent, their performance deteriorates under challenging settings [40], and on more complex benchmarks [50, 61].
Revised: While these methods reduce catastrophic forgetting to some extent, their performance deteriorates under challenging settings [40], and on more complex benchmarks [50, 61].

Original: Rehearsal-based methods [13, 14, 25] save examples from previous tasks into a small-sized buffer to train the model jointly with the current task.
Revised: Rehearsal-based methods [13, 14, 25] save examples from previous tasks into a small-sized buffer to train the model jointly with the current task.

Original: Though simple in concept, the idea of rehearsal is very effective in practice and has been adopted by many state-of-the-art methods [8, 11, 49].
Revised: Though simple in concept, the idea of rehearsal is very effective in practice and has been adopted by many state-of-the-art methods [8, 11, 49].

Original: Architecture-based methods [42, 51, 57, 59, 63] isolate existing model parameters or assign additional parameters for each task to reduce interference among tasks.
Revised: Architecture-based methods [42, 51, 57, 59, 63] isolate existing model parameters or assign additional parameters for each task to reduce interference among tasks.

Original: As mentioned in Section 1, most of these methods use a dense model without consideration of efficiency and memory footprint, thus are not applicable to resource-limited settings.
Revised: As mentioned in Section 1, most of these methods use a dense model without consideration of efﬁciency and memory footprint, thus are not applicable to resource-limited settings.

Original: Our work, orthogonal to these methods, serves as a general framework for making these existing methods efficient and enabling a broader deployment, e.g. , CL on edge devices.
Revised: Our work, orthogonal to these methods, serves as a general framework for making these existing methods efﬁcient and enabling a broader deployment, e.g. , CL on edge devices.

Original: A limited number of works explore sparsity in CL, however, for different purposes.
Revised: A limited number of works explore sparsity in CL, however, for different purposes.

Original: Several methods [41, 42, 53, 57] incorporate the idea of weight pruning [24] to allocate a sparse sub-network for each task to reduce inter-task interference.
Revised: Several methods [41, 42, 53, 57] incorporate the idea of weight pruning [24] to allocate a sparse sub-network for each task to reduce inter-task interference.

Original: Nevertheless, these methods still reduce the full model sparsity progressively for every task and finally end up with a much denser model.
Revised: Nevertheless, these methods still reduce the full model sparsity progressively for every task and ﬁnally end up with a much denser model.

Original: On the contrary,
Revised: On the contrary,

Original: SparCL maintains a sparse network throughout the whole CL process, introducing great efficiency and memory benefits both during training and at the output model.
Revised: SparCL maintains a sparse network throughout the whole CL process, introducing great efﬁciency and memory beneﬁts both during training and at the output model.

Original: A recent work [15] aims at discovering lottery tickets [21] under CL, but still does not address efficiency.
Revised: A recent work [15] aims at discovering lottery tickets [21] under CL, but still does not address efﬁciency.

Original: However, the existence of lottery tickets in CL serves as a strong justification for the outstanding performance of our SparCL.
Revised: However, the existence of lottery tickets in CL serves as a strong justiﬁcation for the outstanding performance of our SparCL.

Original: 2.2 Sparse Training
Revised: 2.2 Sparse Training

Original: There are two main approaches for sparse training: fixed-mask sparse training and dynamic sparse training.
Revised: There are two main approaches for sparse training: ﬁxed-mask sparse training and dynamic sparse training.

Original: Fixed-mask sparse training methods [35, 54, 56, 60] first apply pruning, then execute traditional training on the sparse model with the obtained fixed mask.
Revised: Fixed-mask sparse training methods [35, 54, 56, 60] ﬁrst apply pruning, then execute traditional training on the sparse model with the obtained ﬁxed mask.

Original: The pre-fixed structure limits the accuracy performance, and the first stage still causes huge computation and memory consumption.
Revised: The pre-ﬁxed structure limits the accuracy performance, and the ﬁrst stage still causes huge computation and memory consumption.

Original: To overcome these drawbacks, dynamic mask methods [4, 17, 20, 45, 46] adjust the sparsity topology during training while maintaining low memory footprint.
Revised: To overcome these drawbacks, dynamic mask methods [4, 17, 20, 45, 46] adjust the sparsity topology during training while maintaining low memory footprint.

Original: These methods start with a sparse model structure from an untrained dense model, then combine sparse topology exploration at the given sparsity ratio with the sparse model training.
Revised: These methods start with a sparse model structure from an untrained dense model, then combine sparse topology exploration at the given sparsity ratio with the sparse model training.

Original: Recent work [67] further considers to incorporate data efficiency into sparse training for better training accelerations.
Revised: Recent work [67] further considers incorporating data efﬁciency into sparse training for better training accelerations.

Original: However, all prior sparse training works are focused on the traditional training setting, while CL is a more complicated and difficult scenario with inherent characteristics not explored by these works.
Revised: However, all prior sparse training works are focused on the traditional training setting, while CL is a more complicated and difﬁcult scenario with inherent characteristics not explored by these works.

Original: In contrast to prior sparse training methods, our work explores a new learning paradigm that introduces sparse training into CL for efficiency and also addresses key challenges in CL, mitigating catastrophic forgetting.
Revised: In contrast to prior sparse training methods, our work explores a new learning paradigm that introduces sparse training into CL for efﬁciency and also addresses key challenges in CL, mitigating catastrophic forgetting.

Original:  Continual Learning Problem Setup
Revised:  Continual Learning Problem Setup

Original: In supervised CL, a model f θ learns from a sequence of tasks D
Revised: In supervised CL, a model f ✓ learns from a sequence of tasks D

Original: = {D 1 , . . .
Revised: = {D 1 , . . .

Original: , D T } , where each task
Revised: , D T } , where each task

Original: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.
Revised: D t = { ( x t,i , y t,i ) } n t i =1 consists of input-label pairs, and each task has a disjoint set of classes.

Original: Tasks 
Revised: Tasks ⇤

Original: [Figure] Ep o c h s arrive sequentially, and the model must adapt to them.
Revised: -spiral/SparCL arrive sequentially, and the model must adapt to them.

Original: At the t -th step, the model gains access to data from the t -th task.
Revised: At the t -th step, the model gains access to data from the t -th task.

Original: However, a small fix-sized rehearsal buffer M is allowed to save data from prior tasks.
Revised: However, a small ﬁx-sized rehearsal buffer M is allowed to save data from prior tasks.

Original: At test time, the easiest setting is to assume task identity is known for each coming test example, named task-incremental learning (Task-IL).
Revised: At test time, the easiest setting is to assume task identity is known for each coming test example, named task-incremental learning (Task-IL).

Original: If this assumption does not hold, we have the more difficult class-incremental learning (Class-IL) setting.
Revised: If this assumption does not hold, we have the more difﬁcult class-incremental learning (Class-IL) setting.

Original: In this work, we mainly focus on the more challenging Class-IL setting, and only report Task-IL performance for reference.
Revised: In this work, we mainly focus on the more challenging Class-IL setting, and only report Task-IL performance for reference.

Original: The goal of conventional CL is to train a model sequentially that performs well on all tasks at test time.
Revised: The goal of conventional CL is to train a model sequentially that performs well on all tasks at test time.

Original: The main evaluation metric is average test accuracy on all tasks.
Revised: The main evaluation metric is average test accuracy on all tasks.

Original: In real-world resourcelimited scenarios, we should further consider training efficiency of the model.
Revised: In real-world resourcelimited scenarios, we should further consider training efﬁciency of the model.

Original: Thus, we measure the performance of the model more comprehensively by including training FLOPs and memory footprint.
Revised: Thus, we measure the performance of the model more comprehensively by including training FLOPs and memory footprint.

Original:  Sparse Continual Learning (SparCL)
Revised:  Sparse Continual Learning (SparCL)

Original: Our method, Sparse Continual Learning, is a unified framework composed of three complementary components: task-aware dynamic masking for weight sparsity, dynamic data removal for data efficiency, and dynamic gradient masking for gradient sparsity.
Revised: Our method, Sparse Continual Learning, is a uniﬁed framework composed of three complementary components: task-aware dynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradient masking for gradient sparsity.

Original: The entire framework is shown in
Revised: The entire framework is shown in

Original: Figure 2.
Revised: Figure 2.

Original: We will illustrate each component in detail in this section.
Revised: We will illustrate each component in detail in this section.

Original: 4.1 Task-aware Dynamic Masking
Revised: 4.1 Task-aware Dynamic Masking

Original: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamic structure when learning a sequence of tasks, such that it not only achieves high efficiency, but also intelligently adapts to the data stream for better performance.
Revised: To enable cost-effective CL in resource limited scenarios, SparCL is designed to maintain a dynamic structure when learning a sequence of tasks, such that it not only achieves high efﬁciency, but also intelligently adapts to the data stream for better performance.

Original: Specifically, we propose a strategy named task-aware dynamic masking (TDM), which dynamically removes less important weights and grows back unused weights for stronger representation power periodically by maintaining a single binary weight mask throughout the CL process.
Revised: Speciﬁcally, we propose a strategy named task-aware dynamic masking (TDM), which dynamically removes less important weights and grows back unused weights for stronger representation power periodically by maintaining a single binary weight mask throughout the CL process.

Original: Different from typical sparse training work, which only leverages the weight magnitude [45] or the gradient w.r.t.
Revised: Different from typical sparse training work, which only leverages the weight magnitude [45] or the gradient w.r.t.

Original: data from a single training task [20, 67], TDM considers also the importance of weights w.r.t.
Revised: data from a single training task [20, 67], TDM considers also the importance of weights w.r.t.

Original: data saved in the rehearsal buffer, as well as the switch between CL tasks.
Revised: data saved in the rehearsal buffer, as well as the switch between CL tasks.

Original: Specifically, TDM strategy starts from a randomly initialized binary mask M θ = M 0 , with a given sparsity constraint ∥ M θ ∥ 0 / [-∥ θ ∥ 
-] = 1 − s , where s ∈ [0 , 1] is the sparsity ratio.
Revised: Speciﬁcally, TDM strategy starts from a randomly initialized binary mask M ✓ = M 0 , with a given sparsity constraint k M ✓ k 0 / [-∥ θ ∥ 
-]k ✓ k 0 = 1  s , where s 2 [0 , 1] is the sparsity ratio.

Original: Moreover, it makes different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on their continual weight importance (CWI).
Revised: Moreover, it makes different intra- and inter-task adjustments to keep a dynamic sparse set of weights based on their continual weight importance (CWI).

Original: We summarize the process of task-aware dynamic masking in
Revised: We summarize the process of task-aware dynamic masking in

Original: Algorithm 1 and elaborate its key components in detail below.
Revised: Algorithm 1 and elaborate its key components in detail below.

Original: Continual weight importance (CWI).
Revised: Continual weight importance (CWI).

Original: For a model f θ parameterized by θ , the CWI of weight w ⊂ θ is defined as follows:
Revised: For a model f ✓ parameterized by ✓ , the CWI of weight w ⇢ ✓ is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and α , β are coefficients to control the influence of current and buffered data, respectively.
Revised: where D t denotes the training data from the t -th task, M is the current rehearsal buffer, and ↵ ,  are coefﬁcients to control the inﬂuence of current and buffered data, respectively.

Original: Moreover, L represents the cross-entropy loss for classification, while ˜ L is the single-head [1] version of the cross-entropy loss, which only considers classes from the current task by masking out the logits of other classes.
Revised: Moreover, L represents the cross-entropy loss for classiﬁcation, while ˜ L is the single-head [1] version of the cross-entropy loss, which only considers classes from the current task by masking out the logits of other classes.

Original: Input : Model weight θ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M θ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval δk
Revised: Input : Model weight ✓ , number of tasks T , training epochs of the t -th task K t , binary sparse mask M ✓ , sparsity ratio s , intra-task adjustment ratio p intra , inter-task adjustment ratio p inter , update interval  k

Original: Initialize: θ , M θ , s.t.
Revised: Initialize: ✓ , M ✓ , s.t.

Original: ∥ M θ ∥ 0 / ∥ θ ∥ 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original: = 1 − s for t = 1 , . ..
Revised:  s for t = 1 , . .

Original: , T do for e = 1 , . . .
Revised: , T do for e = 1 , . . .

Original: , K T do if t > 1 then /
Revised: , K T do if t > 1 then /

Original: * Inter-task adjustment */ Expand M θ by randomly adding unused weights, s.t.
Revised: * Inter-task adjustment */ Expand M ✓ by randomly adding unused weights, s.t.

Original: ∥ M θ ∥ 0 / [-∥ θ ∥ 
-] = 1 − ( s − p inter ) if e = δk then
Revised: k M ✓ k 0 / [-∥ θ ∥ 
-]k ✓ k 0 = 1  ( s  p inter ) if e =  k then

Original: Shrink M θ by removing the least important weights according to equation 1, s.t.
Revised: Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: ∥ M θ ∥ 0 / ∥ θ ∥ 

Revised: k M ✓ k 0 / k ✓ k 0 =

Original: = 1 − s end endif e mod δk = 

Revised:  s end endif e mod  k =

Original: then /
Revised: then /

Original: * Intra-task adjustment */ Shrink M θ by removing the least important weights according to equation 1, s.t.
Revised: * Intra-task adjustment */ Shrink M ✓ by removing the least important weights according to equation 1, s.t.

Original: ∥ M θ ∥ 0 / [-∥ θ ∥ 
-] = 1 − ( s + p intra ) 
Revised: k M ✓ k 0 / [-∥ θ ∥ 
-]k ✓ k 0 = 1  ( s + p intra ) Expand M

Original: Expand M θ by randomly adding unused weights, s.t. ∥ M θ ∥ 0 / 
Revised: ✓ by randomly adding unused weights, s.t. k M ✓ k 0 / k ✓ k 0 = 1  s end

Original: [-∥ θ ∥ 
 = 1 − s end-] Update θ ⊙ M θ via backpropagation end end
Revised: [-∥ θ ∥ 
 = 1 − s end-] Update ✓  M ✓ via backpropagation end end

Original: Intuitively, CWI ensures we keep (1) weights of larger magnitude for output stability, (2) weights important for the current task for learning capacity, and (3) weights important for past data to mitigate catastrophic forgetting.
Revised: Intuitively, CWI ensures we keep (1) weights of larger magnitude for output stability, (2) weights important for the current task for learning capacity, and (3) weights important for past data to mitigate catastrophic forgetting.

Original: Moreover, inspired by the classification bias in CL [1], we use the single-head cross-entropy loss when calculating  importance score w.r.t.
Revised: Moreover, inspired by the classiﬁcation bias in CL [1], we use the singlehead cross-entropy loss when calculating the importance score w.r.t.

Original: the current task to make the importance estimation more accurate.
Revised: the current task to make the importance estimation more accurate.

Original: Intra-task adjustment.
Revised: Intra-task adjustment.

Original: When training the t -th task, a natural assumption is that the data distribution is consistent inside the task, thus we would like to update the sparse model in a relatively stable way while keeping its flexibility.
Revised: When training the t -th task, a natural assumption is that the data distribution is consistent inside the task. As a result, we would like to update the sparse model in a relatively stable way while keeping its ﬂexibility.

Original: Thus, in Algorithm 1, we choose to update the sparsity mask M θ in a shrink-and-expand way every δk epochs.
Revised: Thus, in Algorithm 1, we choose to update the sparsity mask M ✓ in a shrink-and-expand way every  k epochs.

Original: We first remove p intra of the weights of least CWI to retain learned knowledge so far.
Revised: We ﬁrst remove p intra of the weights of least CWI to retain learned knowledge so far.

Original: Then we randomly select unused weights to recover the learning capacity for the model and keep the sparsity ratio s unchanged.
Revised: Then we randomly select unused weights to recover the learning capacity for the model and keep the sparsity ratio s unchanged.

Original: Inter-task adjustment.
Revised: Inter-task adjustment.

Original: When tasks switches, on the contrary, we assume  data distribution shifts immediately.
Revised: When tasks switch, on the contrary, we assume that the data distribution shifts immediately.

Original: Ideally, we would like the model to keep the knowledge learned from old tasks as much as possible, and to have enough learning capacity to accommodate the new task.
Revised: Ideally, we would like the model to keep the knowledge learned from old tasks as much as possible, and to have enough learning capacity to accommodate the new task.

Original: Thus, instead of the shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.
Revised: Thus, instead of the shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shrink scheme.

Original: Specifically, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly adding a proportion of p inter unused weights.
Revised: Speciﬁcally, at the beginning of the ( t + 1) -th task, we expand the sparse model by randomly adding a proportion of p inter unused weights.

Original: Intuitively, the additional learning capacity facilitates fast adoption of new knowledge and reduces interference with learned knowledge.
Revised: Intuitively, the additional learning capacity facilitates fast adoption of new knowledge and reduces interference with learned knowledge.

Original: We allow our model to have smaller sparsity ( i.e. , larger learning capacity) temporarily for the first δk epochs as a warmup period, and then remove the p inter weights with least CWI, following the same process in the intra-task case, to satisfy the sparsity constraint.
Revised: We allow our model to have smaller sparsity ( i.e. , larger learning capacity) temporarily for the ﬁrst  k epochs as a warm-up period, and then remove the p inter weights with least CWI, following the same process in the intra-task case, to satisfy the sparsity constraint.

Original: 4.2 Dynamic Data Removal
Revised: 4.2 Dynamic Data Removal

Original: In addition to weight sparsity, decreasing the amount of training data can be directly translated intothe saving of training time without any requirements for hardware support.
Revised: In addition to weight sparsity, decreasing the amount of training data can be directly translated into training time savings.

Original: Thus, we would also like to explore data efficiency to reduce the training workload.
Revised: Thus, we would also like to explore data efﬁciency to reduce the training workload.

Original: Some prior CL works select informative examples to construct the rehearsal buffer [3, 6, 65].
Revised: Some prior CL works select informative examples to construct the rehearsal buffer [3, 6, 65].

Original: However, the main purposeof them is not training acceleration, thus they either introduce excessive computational cost or consider different problem settings.
Revised: However, their main purpose is not training acceleration; thus, they either introduce excessive computational cost or consider different problem settings.

Original: By considering the features of CL, we present a simple yet effective strategy, dynamic data removal (DDR), to reduce training data for further acceleration.
Revised: By considering the features of CL, we present a simple yet effective strategy, dynamic data removal (DDR), to reduce training data for further acceleration.

Original: We measure the importance of each training example by the occurrence of misclassification [55, 67] during CL.
Revised: We measure the importance of each training example by the occurrence of misclassiﬁcation [55, 67] during CL.

Original: In TDM, the sparse structure of our model updates periodically every δk epochs, so we align our data removal process withthe update of weight mask  for further efficiency and training stability.
Revised: In TDM, the sparse structure of our model updates periodically every  k epochs, so we align our data removal process with weight mask updates for further efﬁciency and training stability.

Original: In Section 4.1, we have partitioned the training process for the t -th task into N t =
Revised: In Section 4.1, we have partitioned the training process for the t -th task into N t =

Original: K t /δk stages based on the dynamic mask update.
Revised: K t /  k stages based on the dynamic mask update.

Original: Therefore, we gradually remove training data at the end of i -th stage, based on the following policy: 1) Calculate the total number of misclassifications f i ( x j ) for each training example during the i -th stage.
Revised: Therefore, we gradually remove training data at the end of i -th stage, based on the following policy: 1) Calculate the total number of misclassiﬁcations f i ( x j ) for each training example during the i -th stage.

Original: 2) Remove a proportion of ρ i training samples with the least number of misclassifications.
Revised: 2) Remove a proportion of ⇢ i training samples with the least number of misclassiﬁcations.

Original: Although our main purpose is to keep the “harder” examples to learn to consolidate the sparse model, we can get further benefits for better CL result.
Revised: Although our main purpose is to keep the “harder” examples to learn to consolidate the sparse model, we can get further beneﬁts for better CL results.

Original: First, the removal of “easier” examples increases the probability that “harder” examples to be saved to the rehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples.
Revised: First, the removal of “easier” examples increases the probability that “harder” examples to be saved to the rehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples.

Original: Thus, we construct a more informative buffer in a implicit way without heavy computation.
Revised: Thus, we construct a more informative buffer in a implicit way without heavy computation.

Original: Moreover, since the buffer size is much smaller than the training set size of each task, the data from the buffer and the new task is highly imbalanced, dynamic data removal also relieves the data imbalance issue.
Revised: Moreover, since the buffer size is much smaller than the training set size of each task, the data from the buffer and the new task is highly imbalanced; dynamic data removal also relieves this data imbalance issue.

Original: Formally, we set the data removal proportion for each task as ρ ∈ [0 , 1] , and a cutoff stage, such that:
Revised: Formally, we set the data removal proportion for each task as ⇢ 2 [0 , 1] , and a cutoff stage, such that:

Original: [Equation]
Revised: [Equation]

Original: The cutoff stage controls the trade-off between efficiency and accuracy: When we set the cutoff stage earlier, we reduce the training time for all the following stages; however, when the cutoff stage is set too early, the model might underfit the removed training data.
Revised: The cutoff stage controls the trade-off between efﬁciency and accuracy: when we set the cutoff stage earlier, we reduce the training time for all the following stages; however, when the cutoff stage is set too early, the model might underﬁt the removed training data.

Original: Note that when we set ρ i = 0 for all i = 1 , 2 , . . .
Revised: Note that when we set ⇢ i = 0 for all i = 1 , 2 , . . .

Original: , N t and cutoff =
Revised: , N t and cutoff =

Original: N t , we simply recover the vanilla setting without any data efficiency considerations.
Revised: N t , we simply recover the vanilla setting without any data efﬁciency considerations.

Original: In our experiments, we assume ρ i = ρ/ cutoff , i.e., removing equal proportion of data at the end of every stage, for simplicity.
Revised: In our experiments, we assume ⇢ i = ⇢ / cutoff , i.e., removing equal proportion of data at the end of every stage, for simplicity.

Original: We also conduct  comprehensive exploration study for ρ and the selection of the cutoff stage in Section 5.3 and Appendix B.3.
Revised: We also conduct a comprehensive exploration study of ⇢ and the selection of the cutoff stage in Section 5.3 and Appendix B.3.

Original: 4.3 Dynamic Gradient Masking
Revised: 4.3 Dynamic Gradient Masking

Original: With TDM and DDR, we can already achieve bi-level efficiency during training.
Revised: With TDM and DDR, we can already achieve weight efﬁciency and data efﬁciency during training.

Original: To further boost training efficiency, we exploresparsity in gradient  and propose dynamic gradient masking (DGM) for CL.
Revised: To further boost training efﬁciency, we explore gradient sparsity and propose dynamic gradient masking (DGM) for CL.

Original: Our method focuses on reducing computational cost by only applying the most important gradients onto the corresponding unpruned model parameters via a gradient mask.
Revised: Our method focuses on reducing computational costs by only applying the most important gradients onto the corresponding unpruned model parameters via a gradient mask.

Original: The gradient mask is also dynamically updated along with the weight mask defined in Section 4.1.
Revised: The gradient mask is also dynamically updated along with the weight mask deﬁned in Section 4.1.

Original: Intuitively, while targeting for better training efficiency, DGM also promotes the preservation of past knowledge by preventing a fraction of weights from update.
Revised: Intuitively, while targeting for better training efﬁciency, DGM also promotes the preservation of past knowledge by preventing a fraction of weights from updating.

Original: Formally, our goal here is to find a subset of unpruned parameters (or, equivalently, a gradient mask
Revised: Formally, our goal here is to ﬁnd a subset of unpruned parameters (or, equivalently, a gradient mask

Original: M G ) to update over multiple training iterations.
Revised: M G ) to update over multiple training iterations.

Original: For a model f θ parameterized by θ , we have the corresponding gradient matrix G calculated during each iteration.
Revised: For a model f ✓ parameterized by ✓ , we have the corresponding gradient matrix G calculated during each iteration.

Original: To prevent the pruned weights from updating, the weight mask M θ will be applied onto the gradient matrix G as G ⊙ M θ during backpropagation.
Revised: To prevent the pruned weights from updating, the weight mask M ✓ will be applied onto the gradient matrix G as G  M ✓ during backpropagation.

Original: Besides the gradients of pruned weights, we in addition consider to remove less important gradients for faster training.
Revised: Besides the gradients of pruned weights, we in addition consider to remove less important gradient coefﬁcients for faster training.

Original: To achieve this, we introduce the continual gradient importance (CGI) based on the CWI to measure the importance of weight gradients.
Revised: To achieve this, we introduce the continual gradient importance (CGI) based on the CWI to measure the importance of weight gradients:

Original: [Equation]
Revised: [Equation]

Original: We remove a proportion q of non-zero gradients from G with less importance measured by CGI and we have ∥ M G ∥ 0 / [-∥ θ ∥ 
-] = 1 − ( s + q ) .
Revised: We remove a proportion q of non-zero gradients from G with less importance measured by CGI and we have k M G k 0 / [-∥ θ ∥ 
-]k ✓ k 0 = 1  ( s + q ) .

Original: The gradient mask M G is then applied onto the gradient matrix G .
Revised: The gradient mask M G is then applied onto the gradient matrix G .

Original: During the entire training process, the gradient mask M G is updated with a fixed interval.
Revised: During the entire training process, the gradient mask M G is updated with a ﬁxed interval.

Original:  Experiment 5.1 Experiment Setting
Revised:  Experiment 5.1 Experiment Setting

Original: Datasets.
Revised: Datasets.

Original: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [33] and Split Tiny-ImageNet [16] to verify the efficacy of SparCL.
Revised: We evaluate our SparCL on two representative CL benchmarks, Split CIFAR-10 [33] and Split Tiny-ImageNet [16] to verify the efﬁcacy of SparCL.

Original: In particular, we follow [8, 68] by
Revised: In particular, we follow [8, 68] by

Original: [Table]
Revised: [Table]

Original: splitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classes respectively.
Revised: splitting CIFAR-10 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classes respectively.

Original: Dataset licensing information can be found in Appendix A.
Revised: Dataset licensing information can be found in Appendix A.

Original: Comparing methods.
Revised: Comparing methods.

Original: In particular, we select  CL methodsof all kinds including regularizationbased (EWC [32], LwF [37]), architecture-based (PackNet [42], LPS [57])
Revised: We select several CL methods including regularization-based (EWC [32], LwF [37]), architecture-based (PackNet [42], LPS [57])

Original: [Table]
Revised: [Table]

Original: and rehearsal-based (A-GEM [13], iCaRL [44], FDR [5], ER [14], DER++ [8]) methods.
Revised: and rehearsal-based (A-GEM [13], iCaRL [44], FDR [5], ER [14], DER++ [8]) methods.

Original: Note that PackNet and LPS are only compatible with task-incremental learning.
Revised: Note that PackNet and LPS are only compatible with task-incremental learning.

Original: We also adapt representative sparse training methods (SNIP [35], RigL [20]) to the CL setting by combining them with DER++ (SNIP-DER++, RigL-DER++).
Revised: We also adapt representative sparse training methods (SNIP [35], RigL [20]) to the CL setting by combining them with DER++ (SNIP-DER++, RigL-DER++).

Original: Variants of our method.
Revised: Variants of our method.

Original: To show the generality of SparCL, we combine it with DER++ (one of the SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER , respectively.
Revised: To show the generality of SparCL, we combine it with DER++ (one of the SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER , respectively.

Original: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensive evaluation.
Revised: We also vary the weight sparsity ratio ( 0 . 75 , 0 . 90 , 0 . 95 ) of SparCL for a comprehensive evaluation.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use the average accuracy on all tasks to evaluate the performance of the final model.
Revised: We use the average accuracy on all tasks to evaluate the performance of the ﬁnal model.

Original: Moreover, we evaluate the training FLOPs [20], and memory footprint [67] (including feature map pixels and model parameters during training) to demonstrate the efficiency of each method.
Revised: Moreover, we measure the training FLOPs [20], and memory footprint [67] (including feature map pixels and model parameters during training) to demonstrate the efﬁciency of each method.

Original: Please see Appendix B.1 for detailed definitions of these metrics.
Revised: Please see Appendix B.1 for detailed deﬁnitions of these metrics.

Original: Experiment details.
Revised: Experiment details.

Original: For fair comparison, we strictly follow the settings in prior CL work [8, 29].
Revised: For fair comparison, we strictly follow the settings in prior CL work [8, 29].

Original: We sets the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively, with a batch size of 32 .
Revised: We set the per task training epochs to 50 and 100 for Split CIFAR-10 and Tiny-ImageNet, respectively, with a batch size of 32 .

Original: For the model architecture, We follow [8, 50] and adopt the ResNet-18 [26] without any pre-training.
Revised: For the model architecture, we follow [8, 50] and adopt the ResNet-18 [26] without any pre-training.

Original: We also use the best hyperparameter setting reported in [8, 57] for CL methods, and in [20, 35] for CL-adapted sparse training methods.
Revised: We also use the best hyperparameter setting reported in [8, 57] for CL methods, and in [20, 35] for CL-adapted sparse training methods.

Original: For SparCL and its competing
Revised: For SparCL and its competing

Original: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.
Revised: CL-adapted sparse training methods, we adopt a uniform sparsity ratio for all convolutional layers.

Original: Please see Appendix B for other details.
Revised: Please see Appendix B for further details.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: Comparison with CL methods.
Revised: Comparison with CL methods.

Original: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.
Revised: Table 1 summarizes the results on Split CIFAR-10 and TinyImageNet, under both class-incremental (Class-IL) and task-incremental (Task-IL) settings.

Original: From
Revised: From

Revised: Table 1, we can clearly tell that SparCL signiﬁcantly improves upon ER and DER++, while also

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 1, we can clearly tell that SparCL significantly improves ER and DER++, while also outperforms other CL baselines, in terms of training efficiency (measured in FLOPs).
Revised: outperforming other CL baselines, in terms of training efﬁciency (measured in FLOPs).

Original: With higher sparsity ratio, SparCL leads to less training FLOPs.
Revised: With higher sparsity ratio, SparCL leads to fewer training FLOPs.

Original: Notably, SparCL achieves 23 × training efficiency improvement upon DER++ with a sparsity ratio of 0.95.
Revised: Notably, SparCL achieves 23 ⇥ training efﬁciency improvement upon DER++ with a sparsity ratio of 0.95.

Original: On the other hand, our framework also improves the average accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90, and only  slight performance drop when sparsity gets larger as 0.95.
Revised: On the other hand, our framework also improves the average accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90, and only a slight performance drop when sparsity gets larger as 0.95.

Original: In particular, SparCL-DER++ with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.
Revised: In particular, SparCL-DER++ with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks.

Original: The outstanding performance of SparCL indicates that our proposed strategies successfully preserve accuracy by further mitigating catastrophic forgetting with a much sparser model.
Revised: The outstanding performance of SparCL indicates that our proposed strategies successfully preserve accuracy by further mitigating catastrophic forgetting with a much sparser model.

Original: Moreover, the improvement that SparCL brings to two different existing CL methods shows the generalizability of SparCL as a unified framework, i.e. , it has the potential to be combined with a wide array of existing methods.
Revised: Moreover, the improvement that SparCL brings to two different existing CL methods shows the generalizability of SparCL as a uniﬁed framework, i.e. , it has the potential to be combined with a wide array of existing methods.

Original: Wewould alsolike to take a closer look at PackNet and LPS, which also leverage the idea of sparsity to split the model by different tasks, a different motivation from training efficiency.
Revised: We also take a closer look at PackNet and LPS, which also leverage the idea of sparsity to split the model by different tasks, a different motivation from training efﬁciency.

Original: Firstly, they are only compatible with the Task-IL setting, since they leverage task identity at both training and test time.
Revised: Firstly, they are only compatible with the Task-IL setting, since they leverage task identity at both training and test time.

Original: Moreover, the model sparsity of these methods reduces with the increasing number of tasks, which still leads to much larger overall training FLOPs than that of SparCL.
Revised: Moreover, the model sparsity of these methods reduces with the increasing number of tasks, which still leads to much larger overall training FLOPs than that of SparCL.

Original: This further demonstrates the importance of keeping a sparse model without permanent expansion throughout the CL process.
Revised: This further demonstrates the importance of keeping a sparse model without permanent expansion throughout the CL process.

Original: Comparison with CL-adapted sparse training methods.
Revised: Comparison with CL-adapted sparse training methods.

Original: Table 2 shows the result under the more difficult Class-IL setting.
Revised: Table 2 shows results under the more difﬁcult Class-IL setting.

Original: SparCL outperforms all CL-adapted sparse training methods in both accuracy and training FLOPs.
Revised: SparCL outperforms all CL-adapted sparse training methods in both accuracy and training FLOPs.

Original: The performance gap between SparCL-DER++ and other methods gets larger with a higher sparsity.
Revised: The performance gap between SparCL-DER++ and other methods gets larger with a higher sparsity.

Original: SNIP-
Revised: SNIP-

Original: and RigL-DER++ achieve training acceleration at the cost of compromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existing sparse training methods under the CL setting.
Revised: and RigL-DER++ achieve training acceleration at the cost of compromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existing sparse training methods under the CL setting.

Original: SNIP generates the static initial mask after network initialization which does not consider the structure suitability among tasks.
Revised: SNIP generates the static initial mask after network initialization which does not consider the structure suitability among tasks.

Original: Though RigL adopts a dynamic mask, the lack of task-aware strategy prevents it from generalizing well to the CL setting.
Revised: Though RigL adopts a dynamic mask, the lack of task-awareness prevents it from generalizing well to the CL setting.

Original: 5.3 Effectiveness of Key Components
Revised: 5.3 Effectiveness of Key Components

Original: Ablation study.
Revised: Ablation study.

Original: We provide a comprehensive ablation study in Table 3 using SparCL-DER++ with 0.75 sparsity on Split CIFAR10.
Revised: We provide a comprehensive ablation study in Table 3 using SparCL-DER++ with 0.75 sparsity on Split CIFAR10.

Original: Table 3 demonstrates that all components of our method contribute to both efficiency and accuracy improvements.
Revised: Table 3 demonstrates that all components of our method contribute to both efﬁciency and accuracy improvements.

Original: Comparing row 1 and 2, we can see that the majority of FLOPs decrease results from TDM.
Revised: Comparing rows 1 and 2, we can see that the majority of FLOPs decrease results from TDM.

Original: Interestingly, TDM leads to an increase in accuracy, indicating
Revised: Interestingly, TDM leads to an increase in accuracy, indicating

Original: TDM generates a sparse model that is even more suitable for learning all tasks than then full dense model.
Revised: TDM generates a sparse model that is even more suitable for learning all tasks than then full dense model.

Original: Comparing row 2 and 3, we can see that DDR indeed further accelerates training by removing less informative examples.
Revised: Comparing rows 2 and 3, we can see that DDR indeed further accelerates training by removing less informative examples.

Original: As discussed in Section 4.2, when we remove a certain number of samples (30% here), we achieve a point where we keep as much informative samples as we need, and also balance the current and buffered data.
Revised: As discussed in Section 4.2, when we remove a certain number of samples (30% here), we achieve a point where we keep as much informative samples as we need, and also balance the current and buffered data.

Original: Comparing row 2 and 4, DGM reduce both training FLOPs and memory footprint while improve the performance of the network.
Revised: Comparing rows 2 and 4, DGM reduce both training FLOPs and memory footprint while improve the performance of the network.

Original: Finally, the last row demonstrates the collaborative performance of all components.
Revised: Finally, the last row demonstrates the collaborative performance of all components.

Original: We also show the same ablation study with 0.
Revised: We also show the same ablation study with 0.

Original: sparsity in Appendix B.4 for reference.
Revised: sparsity in Appendix B.4 for reference.

Original: Detail can be found in Appendix B.1.
Revised: Details can be found in Appendix B.1.

Original: Exploration on DDR.
Revised: Exploration on DDR.

Original: To understand the influence of the data removal proportion ρ , and the cutoff stage for each task, we show corresponding experiment results in Figure 3 and Appendix B.3, respectively.
Revised: To understand the inﬂuence of the data removal proportion ⇢ , and the cutoff stage for each task, we show corresponding experiment results in Figure 3 and Appendix B.3, respectively.

Original: In Figure 3, we fix cutoff = 4 , i.e. , gradually removing equal number of examples every 5 epochs until epoch 20, and vary ρ from 10% to 90% .
Revised: In Figure 3, we ﬁx cutoff = 4 , i.e. , gradually removing equal number of examples every 5 epochs until epoch 20, and vary ⇢ from 10% to 90% .

Original: We also compare DDR with One-shot removal strategy [67], which removes all examples at once at cutoff .
Revised: We also compare DDR with One-shot removal strategy [67], which removes all examples at once at cutoff .

Original: DDR outperforms One-shot consistently with different ρ in average accuracy.
Revised: DDR outperforms One-shot consistently with different ⇢ in average accuracy.

Original: Also note that since DDR removes the examples gradually before the cutoff stage, DDR is more efficient than One-shot.
Revised: Also note that since DDR removes the examples gradually before the cutoff stage, DDR is more efﬁcient than One-shot.

Original: When ρ ≤ 30% , we also observe increased accuracy of DDR compared with the baseline without removing any data.
Revised: When ⇢  30% , we also observe increased accuracy of DDR compared with the baseline without removing any data.

Original: When ρ ≥ 40% , the accuracy gets increasingly lower for both strategies.
Revised: When ⇢  40% , the accuracy gets increasingly lower for both strategies.

Original: The intuition is that when DDR removes a proper amount of data, it removes redundant information while keeps the most informative examples.
Revised: The intuition is that when DDR removes a proper amount of data, it removes redundant information while keeps the most informative examples.

Original: Moreover, as discussed in Section 4.2, it balances the current and buffered data, while also leave informative samples in the buffer.
Revised: Moreover, as discussed in Section 4.2, it balances the current and buffered data, while also leave informative samples in the buffer.

Original: When DDR removes too much data, it will also lose informative examples, thus the model has not learned these examples well before removal.
Revised: When DDR removes too much data, it will also lose informative examples, thus the model has not learned these examples well before removal.

Original: Exploration on DGM.
Revised: Exploration on DGM.

Original: We test the efficacy of DGM at different sparsity levels.
Revised: We test the efﬁcacy of DGM at different sparsity levels.

Original: Detailed exploratory experiments are shown in Appendix B.5 for reference.
Revised: Detailed exploratory experiments are shown in Appendix B.5 for reference.

Original: The results indicate that by setting the proportion q within an appropriate range, DGM can consistently improve the accuracy performance regardless of the change of weight sparsity.
Revised: The results indicate that by setting the proportion q within an appropriate range, DGM can consistently improve the accuracy performance regardless of the change of weight sparsity.

Original: 5.4 Mobile Device Results
Revised: 5.4 Mobile Device Results

Original: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo
Revised: The training acceleration results are measured on the CPU of an off-the-shelf Samsung Galaxy S smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo

Original: Octa-core CPU.
Revised: Octa-core CPU.

Original: We run each test on a batch of 32 images to denote the training speed.
Revised: We run each test on a batch of 32 images to denote the training speed.

Original: The detail of on-mobile compiler-level optimizations for training acceleration can be found in Appendix C.1.
Revised: The detail of on-mobile compiler-level optimizations for training acceleration can be found in Appendix C.1.

Original: The acceleration results are shown in Figure 4.
Revised: The acceleration results are shown in Figure 4.

Original: SparCL can achieve approximately 3.1 × and 2.3 × training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.
Revised: SparCL can achieve approximately 3.1 ⇥ and 2.3 ⇥ training acceleration with 0.95 sparsity and 0.90 sparsity, respectively.

Original: Besides, our framework can also save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.
Revised: Besides, our framework can also save 51% and 48% memory footprint when the sparsity is 0.95 and 0.90.

Original: Furthermore, the obtained sparse models save the storage consumption by using compressed sparse row (CSR) storage and can be accelerated to speed up the inference on-the-edge.
Revised: Furthermore, the obtained sparse models save the storage consumption by using compressed sparse row (CSR) storage and can be accelerated to speed up the inference on-the-edge.

Original: We provide on-mobile inference acceleration results in Appendix C.2.
Revised: We provide on-mobile inference acceleration results in Appendix C.2.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presents a unified framework named SparCL for efficient CL that achieves both learning acceleration and accuracy preservation.
Revised: This paper presents a uniﬁed framework named SparCL for efﬁcient CL that achieves both learning acceleration and accuracy preservation.

Original: It comprises three complementary strategies: task-aware dynamic masking for weight sparsity, dynamic data removal for data efficiency, and dynamic gradient masking for gradient sparsity.
Revised: It comprises three complementary strategies: task-aware dynamic masking for weight sparsity, dynamic data removal for data efﬁciency, and dynamic gradient masking for gradient sparsity.

Original: Extensive experiments on standard CL benchmarks and real-world edge device evaluations demonstrate that our method significantly improves upon existing CL methods and outperforms CL-adapted sparse training methods.
Revised: Extensive experiments on standard CL benchmarks and real-world edge device evaluations demonstrate that our method signiﬁcantly improves upon existing CL methods and outperforms CL-adapted sparse training methods.

Original: We discuss the limitations and potential negative social impacts of our method in Section 7 and 8, respectively.
Revised: We discuss the limitations and potential negative social impacts of our method in Sections 7 and 8, respectively.

Original:  Limitations
Revised:  Limitations

Original: One limitation of our method is that we assume a rehearsal buffer is available throughout the CL process.
Revised: One limitation of our method is that we assume a rehearsal buffer is available throughout the CL process.

Original: Although the assumption is widely-accepted, there are still situations that a rehearsal buffer is not allowed.
Revised: Although the assumption is widely-accepted, there are still situations that a rehearsal buffer is not allowed.

Original: However, as a framework targeting for efficiency, our work has the potential to accelerate all types of CL methods.
Revised: However, as a framework targeting for efﬁciency, our work has the potential to accelerate all types of CL methods.

Original: For example, simply removing the terms related to rehearsal buffer in equation 1 and equation 3 could serve as a naive variation of our method that is compatible with other non-rehearsal methods.
Revised: For example, simply removing the terms related to rehearsal buffer in equation 1 and equation 3 could serve as a naive variation of our method that is compatible with other non-rehearsal methods.

Original: It is interesting to further improve SparCL to be more generic for all kinds of CL methods.
Revised: It is interesting to further improve SparCL to be more generic for all kinds of CL methods.

Original: Moreover, the benchmarks we use are limited to vision domain.
Revised: Moreover, the benchmarks we use are limited to vision domain.

Original: Although using vision-based benchmarks has been a common practice in the CL community, we believe evaluating our method, as well as other CL methods, on datasets from other domains such as NLP will lead to a more comprehensive and reliable conclusion.
Revised: Although using vision-based benchmarks has been a common practice in the CL community, we believe evaluating our method, as well as other CL methods, on datasets from other domains such as NLP will lead to a more comprehensive and reliable conclusion.

Original: We will keep track of newer CL benchmarks from different domains and further improve our work correspondingly.
Revised: We will keep track of newer CL benchmarks from different domains and further improve our work correspondingly.

Original:  Potential Negative Societal Impact
Revised:  Potential Negative Societal Impact

Original: Although SparCL is a general framework to enhance efficiency for various CL methods, we still need to be aware of its potential negative societal impact.
Revised: Although SparCL is a general framework to enhance efﬁciency for various CL methods, we still need to be aware of its potential negative societal impact.

Original: For example, we need to be very careful about the trade-off between accuracy and efficiency when using SparCL.
Revised: For example, we need to be very careful about the trade-off between accuracy and efﬁciency when using SparCL.

Original: If one would like to pursue efficiency by setting the sparsity ratio too high, then even SparCL will result in significant accuracy drop, since the over-sparsified model does not have enough representation power.
Revised: If one would like to pursue efﬁciency by setting the sparsity ratio too high, then even SparCL will result in signiﬁcant accuracy drop, since the over-sparsiﬁed model does not have enough representation power.

Original: Thus, we should pay much attention when applying SparCL on accuracy-sensitive applications such as healthcare [66].
Revised: Thus, we should pay much attention when applying SparCL on accuracy-sensitive applications such as healthcare [66].

Original: Another example is that, SparCL as a powerful tool to make CL methods efficient, can also strengthen models for malicious applications [7].
Revised: Another example is that, SparCL as a powerful tool to make CL methods efﬁcient, can also strengthen models for malicious applications [7].

Original: Therefore, we encourage the community to come up with more strategies and regulations to prevent malicious use of artificial intelligence.
Revised: Therefore, we encourage the community to come up with more strategies and regulations to prevent malicious use of artiﬁcial intelligence

Original:  Acknowledgement
Revised:  Acknowledgement

Original: The authors gratefully acknowledge support by the National Science Foundation under grants CCF1937500, CCF-1919117 and CNS- A Dataset Licensing Information • CIFAR-10 [33] is licensed under the MIT license.
Revised: The authors gratefully acknowledge support by the National Science Foundation under grants CCF1937500, CCF-1919117 and CNS-2112471.

Original: • The licensing information of Tiny-ImageNet [34] is not available. However, the data is available for free to researchers for non-commercial use. B Additional Experiment Details and Results We set α = 0 .
Revised: Checklist

Original: 5 , β = 1 in equation 1 and equation 3. We also set δk = 5 , p inter = 0 . 01 , p intra = 0 . We also match different weight sparsity with gradient sparsity for best performance.
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: We sample 20% data from Split CIFAR-10 training set for validation, and we use grid-search on this validation set to help us select the mentioned best hyperparameters.

Original: We use the same set of hyperparameters for both datasets.

Original: For accurate evaluation, we repeat each experiments 3 times using different random seeds and report the average performance. During our experiments, we adopt unstructured sparsity type and uniform sparsity ratio ( 0 . 95 ) for all convolutional layers in the models. B.1 Evaluation Metrics Explanation
Revised: [Yes] The claims match the experimental results and it is expected to generalize according to the diverse experiments stated in our paper.

Original: Training FLOPs The FLOPs of a single forward pass is calculated by taking the sum of the number of multiplications and additions in each layer l for a given layer sparsity s l . Each iteration in the training process is composed of two phases, i.e., the forward propagation and backward propagation. The goal of the forward pass is to calculate the loss of the current set of parameters on a given batch of data.
Revised: We include all of our code, data, and models in the supplementary materials, which can reproduce our experimental results.

Original: It can be formulated as a l = σ ( z l ) = σ ( w l ∗ a l − 1 + b l ) for each layer l in the model. Here, w , b , and z represent the weights, biases, and output before activation, respectively; σ ( . ) denotes the activation function; a is the activations; ∗ means convolution operation.
Revised: (b) Did you describe the limitations of your work?

Original: The formulation indicates that the layer activations are calculated in sequence using the previous activations and the parameters of the layer. Activation of layers are stored in memory for the backward pass. As for the backward propogation, the objective is to back-propagate the error signal while calculating the gradients of the parameters. The two main calculation steps can be represented as:
Revised: [Yes] See Section 6 and Appendix 7.

Original: [Equation] where δ l is the error associated with the layer l , G l denotes the gradients, ⊙ represents Hadamard product, σ ′ ( . ) denotes the derivative of activation, and rotate 180 ° ( . ) means rotating the matrix by 180 ° is the matrix transpose operation.
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: During the backward pass, each layer l calculates two quantities, i.e., the gradient of the activations of the previous layer and the gradient of its parameters. Thus, the backward passes are counted as twice the computation expenses of the forward pass [20]. We omit the FLOPs needed for batch normalization and cross entropy.
Revised: [Yes] See Section 6 and Appendix 8.

Original: In our work, the total FLOPs introduced by TDM, DDR, and DGM on split CIFAR-10 is approximately 4 . 5 × 10 9 which is less than 0 . 0001% of total training FLOPs.
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: For split Tiny-ImageNet, the total FLOPs of them is approximately 1 . 8 × 10 10 , which is also less than 0 .

Original: 0001% of total training FLOPs.

Original: Therefore, the computation introduced by

Original: TDM, DDR, and DGM is negligible. Memory Footprints Following works [10, 67], the definition of memory footprints contain two parts: 1) activations (feature map pixels) during training phase, and 2) model parameters during training phase. For experiments, activations, model weights, and gradients are stored in 32-bit floating-point format for training. The memory footprint results are calculated with an approximate summation of them.
Revised: [Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.

Original: The memory footprint is composed of three parts: activations, model weights, and gradients. They are all represented as b w -bit numbers for training. The number of activations in the model is the sum of the activations in each layer.
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: Suppose that the output feature of the l -th layer with a batch size of B is represented as a l ∈ R B × O l × H l × W l , where O l is the number of channels and H l × W l is the feature size. The total number of activations of the model is thus B (cid:80) l
Revised: [N/A]

Original: O l H l W l . As for the model weights, our SparCL training a sparse model with a sparsity ratio s ∈ [0 , 1] from scratch. The sparse model is obtained from a dense model with a total of N weights. A higher value of s indicates fewer non-zero weights in the sparse model.
Revised: Our paper is based on the experimental results and we do not have any theoretical results.

Original: Compressed sparse row (CSR) format is commonly used for sparse storage, which greatly reduces the number of indices need to be stored for sparse matrices.

Original: As our SparCL adopt only one sparsity type and we use a low-bit format to store the indices, we omit the indices storage here.

Original: Therefore, the memory footprint for model representation is (1 − s )

Original: Nb w .

Original: Similar calculations can be applied for the gradient matrix.

Original: Besides the sparsity ratio s , additional q gradients are masked out from the gradient matrix, resulting a sparsity ratio s + q .

Original: Therefore, the storage of gradients can be approximated as (1 − ( s + q ))

Original: Nb w . Combining the activations, model representation, and gradients, the total memory footprint in SparCL can be represented as (2 B (cid:80) l O l H l W l + (1 − s ) N + (1 − ( s + q )) N ) b w . DDR requires store indices for the easier examples during the training process.
Revised: (b) Did you include complete proofs of all theoretical results?

Original: The number of training examples for Split CIFAR-10 and Split Tiny-ImageNet on each task is 10000.

Original: In our work, we only need about 3KB (remove 30% training data) for indices storage (in the int8 format) and the memory cost is negligible compared with the total memory footprint.

Original: B.3 Effect of Cutoff Stage

Original: [Table]
Revised: [N/A]

Original: To evaluate the effect of the cutoff stage, we use the same setting as in Figure 3 by setting the sparsity ratio to 0 .

Original: We keep the data removal proportion ρ = 30% , and only change cutoff .

Original: Table A1 shows the relationship between cutoff and the Class-IL average accuracy.

Original: Note that from the perspective of efficiency, we would like the cutoff stage as early as possible, so that the remaining epochs will have less examples.

Original: However, from Table A1, we can see that if we set it too early, i.e. , cutoff ≤ 3 , the accuracy drop is significant.

Original: This indicate that even for the “easy-to-learn” examples, removing them too early results in underfitting.

Original: As a balance point between accuracy andefficiency, we choose cutoff = 4 in our final version. B.4 Supplementary Ablation Study [Table] Similar to Table 3, we show ablation study with 0 .
Revised: Our paper is based on the experimental results and we do not have any theoretical results.

Original: 90 sparsity ratio in Table A2.

Original: Under a larger sparsity ratio, the conclusion that all components contribute to the final performance still holds. However, we can observe that the accuracy increase that comes from DDR and DGM is less than what we show in Table 3. We assume that larger sparsity ratio makes it more difficult for the model to retain good accuracy in CL. Similar results has also been observed in[67] under the usual i.i.d.
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: learning setting. [Table] B.5 Exploration on DGM We conduct further experiments to demonstrate the influence of gradient sparsity, and  the results are shown in Table A4.
Revised: [Yes] See Section 5.1, Section 5.4 and we provide code to reproduce the main experimental results.

Original: There are two sets of the experiments with different weight sparsity settings: 0.75 and 0.90. Within each set of the experiments (the weight sparsity is fixed), we vary the gradient sparsity values. From the results we can see that increasing the gradient sparsity can decrease the FLOPs and memory footprint.
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: However, the accuracy performance degrades more obvious when the gradient sparsity is too much for the weight sparsity. The results indicate that suitable gradient sparsity setting can bring further efficiency to the training process while boosting the accuracy performance. In the main results, the gradient sparsity is set as 0.80 for 0.75 weight sparsity, and set as 0.92 for 0.90 weight sparsity. C On-Mobile Compiler Optimizations and Inference Results C.1 Compiler Optimizations
Revised: [Yes] See Section 5.1 and Section 5.4.

Original: Each iteration in the training process is composed of two phases, i.e., the forward propagation and backward propagation. Prior works [18, 23, 27, 28, 30, 36, 38, 69] have proved that sparse weight matrices (tensors) can provide inference acceleration via reducing the number of multiplications in convolution operation. Therefore, the forward propagation phase, which is the same as inference, can be accelerated by the sparsity inherently. As for backward pass, both of the calculation steps are based on convolution, i.e., matrix multiplication.
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: Equation 4 uses sparse weight matrix (tensor) as the operand, thus can be accelerated in the same way as the forward propagation. Equation 5 allows a sparse output result since the gradient matrix is also sparse. Thus, both two steps have reduced computations, which are roughly proportional to the sparsity ratio, providing the acceleration for the backward propagation phase. Compiler optimizations are used to accelerate the inference in prior works [22, 47, 62].
Revised: [Yes] See Table 1, Table 2, ﬁg 1, ﬁg 3.

Original: In this work, we extend the compiler optimization techniques for accelerating the forward and backward pass during training on the edge devices.

Original: Our compiler optimizations are general, support both sparse model training and inference accelerations on mobile platforms. The optimizations include1) the supports for sparse models; 2) an auto-tuning process to determine the best-suited configurations of parameters for different mobile CPUs. The details of our compiler optimizations are presented as follows. C.1.1 Supports for Sparse Models
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: Our framework supports sparse model training and inference accelerations with unstructured pruning. For the sparse (pruned) model, the framework first compacts the model storage with a compression format called Compressed Sparse Row (CSR) format, and then performs computation reordering to reduce the branches within each thread and eliminates the load imbalance among threads. A row reordering optimization is also included to further improve the regularity of the weight matrix. After this reordering, the continuous rows with identical or similar numbers of non-zero weights are processed by multi-threads simultaneously, thus eliminating thread divergence and achieving load balance.
Revised: [Yes] See Section 5.1, Section 5.4. 4.

Original: Each thread processes more than one rows, thus eliminating branches and improving instruction-level parallelism. Moreover, a similar optimization flow (i.e., model compaction and computation reorder and other optimizations) is employed to support all compiler optimizations for sparsity as PatDNN [47]. C.1.2 Auto-Tuning for Different Mobile CPUs During DNN sparse training and inference execution, there are many tuning parameters, e.g., matrix tiling sizes, loop unrolling factors, and data placement on memory, that influence the performance.
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: It is hard to determine the best-suited configuration of these parameters manually. To alleviate this problem, our compiler incorporates an auto-tuning approach for sparse (pruned) models. The Genetic Algorithm is leveraged to explore the best-suited configurations automatically.
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: It starts the parameter search process with an arbitrary number of chromosomes and explores the parallelism better. Acceleration codes for different DNN models and different mobile CPUs can be generated efficiently and quickly through this auto-tuning process. C.2 Inference Acceleration Results On Mobile [Figure]
Revised: [Yes] We mentioned and cited the datasets (Split CIFAR-10 and Tiny-ImageNet), and all comparing methods with their paper and github in it.

Original: Besides accelerating the training process, SparCL also possesses the advantages ofproviding a sparse model as the output for faster inference. To demonstrate this, we show the inference acceleration results of SparCL with different sparsity ratio settings on mobile in Figure 5. The inference time is measured on the CPU of an off-the-shelf Samsung Galaxy S20 smartphone.
Revised: (b) Did you mention the license of the assets?

Original: Each test takes 50 runs on different inputs with 8 threads on CPU. As different runs do not vary greatly, only the average time is reported. From the results we can see that the obtained sparse model from SparCL can significantly accelerate the inference on both Split-CIFAR-10 and Tiny-ImageNet dataset compared to the model obtained by traditional CL training. For ResNet-18 on Split-CIFAR-10, the model obtained by traditional CL training, which is a dense model, takes 18.53ms for inference.
Revised: [Yes] The licences of used datasets/models are provided in the cited references and we state them explicitly in Appendix A. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: The model provided by SparCL can achieve an inference time of 14.01ms, 8.30ms, and 5.85ms with sparsity ratio of 0.75, 0.90, and 0.95, respectively. The inference latency of the dense ResNet-18 obtained by traditional CL training on Tiny-ImageNet is 39.64 ms. While the sparse models provided by SparCL with sparsity ratio settings as 0.75, 0.90, and 0.95 reach inference speed of 33.06ms, 20.37ms, and 15.49ms, respectively, on Tiny-ImageNet. D Comparison with Buffer Selection Methods
Revised: [Yes] We provide code for our proposed method in the supplement.

Original: In this section, we compare DDR and GSS [3] or Loss-Aware Reservoir Sampling (LARS) DDR aims at removing training examples for efficiency, while GSS and LARS put the focus on selecting more informative examples that are saved in the buffer. Technically, DDR removes less informative training examples at certain epochs (and thus indirectly affects samples saved in the buffer), while GSS and LARS directly replaces less informative buffered examples in the buffer. Thus, the original GSS and LARS are not directly comparable to DDR.
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: However, we can actually use the example importance criteria used in GSS and LARS to remove less informative training examples as well. We replace the misclassification rate in DDR by the gradient-based (GSS) and loss-based criteria (LARS) objectives and get two variants of our approach, DDR-GSS and DDR-LARS, respectively. For fair comparison, we fix all other parameters used in DDR the same for all methods (sparisity 0.75, remove 30% training data, with TDM only). Since all variants of DDR already remove training examples for efficiency, we mainly focus on their accuracy performance here.
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: The final results on

Original: Split-CIFAR10 is shown in the table below:

Original: [Table]
Revised: [N/A] 5.

Original: In this work, we conduct uniform pruning (i.e., each layer has the same pruning ratio) across different CONV layers as mentioned before in experimental details.
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: The usage of uniform pruning ratio is to match the single-instruction multiple-data (SIMD) [47] architecture of embedded CPU/GPU processors for efficient hardware accelerations. To observe the pruning pattern, we also experimented with setting an overall pruning ratio as 95% for the entire network, allowing each layer to have different pruning ratios by ranking CWI for the whole model. According to the results, earlier CONV layers tend to have a smaller pruning ratio, which is only around 25% − 30% , while the pruning ratios for the latter CONV layers can reach 99% . The results are reasonable, as latter layers are more redundant with a larger amount of parameters.
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: In addition, the weights in earlier layers might be more important for keeping high accuracy, but take a large portion of the computation. Therefore, though slightly improving the accuracy performance to 72 .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: 45% compared to the uniform pruning ratio setting, allowing different pruning ratios across different layers yields worse acceleration (drop to 2 . 2 × compared with 3 . 1 × when adopting the uniform pruning ratio) on the hardware. As our purpose is to facilitate the efficiency of the CL-system, we adopt the uniform pruning ratio setting.
Revised: [N/A]