Original: O N THE C ONVERGENCE OF N ONCONVEX C ONTINUAL L EARNING WITH A DAPTIVE L EARNING R ATES
Revised: O N THE C ONVERGENCE OF N ONCONVEX C ONTINUAL L EARNING WITH A DAPTIVE L EARNING R ATES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially.
Revised: One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially.

Original: The memory based continual learning stores a small subset of the data for previous tasks and applies various methods such as quadratic programming and sample selection.
Revised: The memory based continual learning stores a small subset of the data for previous tasks and applies various methods such as quadratic programming and sample selection.

Original: Some memory-based approaches are formulated as a constrained optimization problem and rephrase constraints on the objective for memory as the inequalities on gradients.
Revised: Some memory-based approaches are formulated as a constrained optimization problem and rephrase constraints on the objective for memory as the inequalities on gradients.

Original: However, there have been little theoretical results on the convergence of continual learning.
Revised: However, there have been little theoretical results on the convergence of continual learning.

Original: In this paper, we propose a theoretical convergence analysis of memory-based continual learning with stochastic gradient descent.
Revised: In this paper, we propose a theoretical convergence analysis of memory-based continual learning with stochastic gradient descent.

Original: The proposed method called nonconvex continual learning (NCCL) adapts the learning rates of both previous and current tasks with the gradients.
Revised: The proposed method called nonconvex continual learning (NCCL) adapts the learning rates of both previous and current tasks with the gradients.

Original: The proposed method can achieve the same convergence rate as the SGD method for a single task when the catastrophic forgetting term which we deﬁne in the paper is suppressed at each iteration.
Revised: The proposed method can achieve the same convergence rate as the SGD method for a single task when the catastrophic forgetting term which we deﬁne in the paper is suppressed at each iteration.

Original: It is also shown that memory-based approaches inherently overﬁt to memory, which degrades the performance on previously learned tasks.
Revised: It is also shown that memory-based approaches inherently overﬁt to memory, which degrades the performance on previously learned tasks.

Original: Experiments show that the proposed algorithm improves the performance of continual learning over existing methods for several image classiﬁcation tasks.
Revised: Experiments show that the proposed algorithm improves the performance of continual learning over existing methods for several image classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning new tasks without forgetting previously learned tasks is a key aspect of artiﬁcial intelligence to be as versatile as humans.
Revised: Learning new tasks without forgetting previously learned tasks is a key aspect of artiﬁcial intelligence to be as versatile as humans.

Original: Unlike the conventional deep learning that observes tasks from an i.i.d.
Revised: Unlike the conventional deep learning that observes tasks from an i.i.d.

Original: distribution, continual learning train sequentially a model on a non-stationary stream of data (Ring, 1995; Thrun, 1994).
Revised: distribution, continual learning train sequentially a model on a non-stationary stream of data (Ring, 1995; Thrun, 1994).

Original: The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted (French & Chater, 2002).
Revised: The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted (French & Chater, 2002).

Original: To overcome catastrophic forgetting, continual learning algorithms introduce novel methods such as a replay memory to store and replay the previously learned examples (Lopez-Paz & Ranzato, 2017; Aljundi et al., 2019; Chaudhry et al., 2019a), regularization methods that penalize neural networks (Kirkpatrick et al., 2017; Zenke et al., 2017), Bayesian methods that utilize the uncertainty of parameters or data points (Nguyen et al., 2018; Ebrahimi et al., 2020), and other recent approaches (Yoon et al., 2018;
Revised: To overcome catastrophic forgetting, continual learning algorithms introduce novel methods such as a replay memory to store and replay the previously learned examples (Lopez-Paz & Ranzato, 2017; Aljundi et al., 2019; Chaudhry et al., 2019a), regularization methods that penalize neural networks (Kirkpatrick et al., 2017; Zenke et al., 2017), Bayesian methods that utilize the uncertainty of parameters or data points (Nguyen et al., 2018; Ebrahimi et al., 2020), and other recent approaches (Yoon et al., 2018;

Original: Lee et al., 2019).
Revised: Lee et al., 2019).

Original: In this paper, we focus on the online continual learning with replay memory.
Revised: In this paper, we focus on the online continual learning with replay memory.

Original: The learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep the model staying in a feasible region corresponding to moderate suboptimal region.
Revised: The learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep the model staying in a feasible region corresponding to moderate suboptimal region.

Original: Gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017)
Revised: Gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017)

Original: ﬁrst formulated the replay based continual learning as a constrained optimization problem.
Revised: ﬁrst formulated the replay based continual learning as a constrained optimization problem.

Original: This formulation rephrases the constraints on objectives for previous tasks as the inequalities based on the inner product of loss gradient vectors for previous tasks and a current task.
Revised: This formulation rephrases the constraints on objectives for previous tasks as the inequalities based on the inner product of loss gradient vectors for previous tasks and a current task.

Original: However, the theoretical convergence analysis of the performance of previously learned tasks, which implies a measure of catastrophic forgetting, has not been rigorously studied in the literature.
Revised: However, the theoretical convergence analysis of the performance of previously learned tasks, which implies a measure of catastrophic forgetting, has not been rigorously studied in the literature.

Original: Without convergence analysis, this intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting.
Revised: Without convergence analysis, this intuitive reformulation of constrained optimization does not provide theoretical guarantee to prevent catastrophic forgetting.

Original: Nonconvex ﬁnite-sum optimization problem offers a solution to analyze catastrophic forgetting by measuring the convergence of previously learned tasks, which is related to the performance.
Revised: Nonconvex ﬁnite-sum optimization problem offers a solution to analyze catastrophic forgetting by measuring the convergence of previously learned tasks, which is related to the performance.

Original: Most deep learning problems are deﬁned as nonconvex optimization, and the target objective is composed of the sum of objectives for each data point.
Revised: Most deep learning problems are deﬁned as nonconvex optimization, and the target objective is composed of the sum of objectives for each data point.

Original: Now we express our continual learning problem of the form
Revised: Now we express our continual learning problem of the form

Original: [Equation]
Revised: [Equation]

Original: where we assume that each objective f i ( x ) with a model x and a data point i is nonconvex with
Revised: where we assume that each objective f i ( x ) with a model x and a data point i is nonconvex with

Original: Lipschitz gradient.
Revised: Lipschitz gradient.

Original: Here, we expect that a stochastic gradient descent based algorithm reach a stationary point instead of the global minimum in the nonconvex optimization.
Revised: Here, we expect that a stochastic gradient descent based algorithm reach a stationary point instead of the global minimum in the nonconvex optimization.

Original: This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for a nonconvex setting (Zhou & Gu, 2019;
Revised: This generic form is well studied to demonstrate the convergence and complexity of stochastic gradient methods for a nonconvex setting (Zhou & Gu, 2019;

Original: Lei et al., 2017; Reddi et al., 2016a;
Revised: Lei et al., 2017; Reddi et al., 2016a;

Original: Zaheer et al., 2018).
Revised: Zaheer et al., 2018).

Original: Unlike the convex case, the convergence is generally measured by the expectation of the squared norm of the gradient E (cid:107)∇ f ( x ) (cid:107) 2 .
Revised: Unlike the convex case, the convergence is generally measured by the expectation of the squared norm of the gradient E (cid:107)∇ f ( x ) (cid:107) 2 .

Original: The theoretical complexity is derived from the (cid:15) -accurate solution, which is also known as a stationary point with E (cid:107)∇ f ( x ) (cid:107) 2 ≤ (cid:15) .
Revised: The theoretical complexity is derived from the (cid:15) -accurate solution, which is also known as a stationary point with E (cid:107)∇ f ( x ) (cid:107) 2 ≤ (cid:15) .

Original: Suppose we divide the entire sum of objectives into two terms for previous tasks and current tasks, and measure the convergence on each term.
Revised: Suppose we divide the entire sum of objectives into two terms for previous tasks and current tasks, and measure the convergence on each term.

Original: Then, we can observe the transition of convergences on the previous and current tasks respectively while learning sequentially from a data stream.
Revised: Then, we can observe the transition of convergences on the previous and current tasks respectively while learning sequentially from a data stream.

Original: We consider this transition of convergence on the previous task as catastrophic forgetting if E (cid:107)∇ f P ( x ) (cid:107) 2 with the set of data points from previous tasks P increases over iterations.
Revised: We consider this transition of convergence on the previous task as catastrophic forgetting if E (cid:107)∇ f P ( x ) (cid:107) 2 with the set of data points from previous tasks P increases over iterations.

Original: In this work, we formulate continual learning problem as a nonconvex ﬁnite-sum optimization with stochastic gradient descent algorithm that updates both previously learned tasks from replay memory and the current task simultaneously, and present a theoretical convergence analysis of continual learning problems by leveraging the replay-based update method.
Revised: In this work, we formulate continual learning problem as a nonconvex ﬁnite-sum optimization with stochastic gradient descent algorithm that updates both previously learned tasks from replay memory and the current task simultaneously, and present a theoretical convergence analysis of continual learning problems by leveraging the replay-based update method.

Original: This extends the continual learning algorithm such as ER-Reservoir (experience replay with reservoir sampling) with ﬁxed and same learning rates on both two tasks to our adaptive method which controls the relative importance between tasks at each step with theoretical guarantee.
Revised: This extends the continual learning algorithm such as ER-Reservoir (experience replay with reservoir sampling) with ﬁxed and same learning rates on both two tasks to our adaptive method which controls the relative importance between tasks at each step with theoretical guarantee.

Original: In addition, the replay-based continual learning has the critical limitation of overﬁtting to memory, which also degrades the performance of previously learned tasks like catastrophic forgetting by gradients from the current tasks.
Revised: In addition, the replay-based continual learning has the critical limitation of overﬁtting to memory, which also degrades the performance of previously learned tasks like catastrophic forgetting by gradients from the current tasks.

Original: It is known that choosing the perfect memory for continual learning to prevent catastrophic forgetting is an NP-hard problem (Knoblauch et al., 2020).
Revised: It is known that choosing the perfect memory for continual learning to prevent catastrophic forgetting is an NP-hard problem (Knoblauch et al., 2020).

Original: We also present that the inductive bias by replay memory is inevitable and analyze the relation between the generalization on previous tasks and the sample batch size from memory empirically.
Revised: We also present that the inductive bias by replay memory is inevitable and analyze the relation between the generalization on previous tasks and the sample batch size from memory empirically.

Original:  B ACKGROUNDS
Revised:  B ACKGROUNDS

Original: The continual learning algorithm with an replay memory with size m cannot access the whole dataset of the previously learned tasks with n f samples, but use limited samples in the memory when a learner trains on the current task.
Revised: The continual learning algorithm with an replay memory with size m cannot access the whole dataset of the previously learned tasks with n f samples, but use limited samples in the memory when a learner trains on the current task.

Original: This limited access does not guarantee to completely prevent catastrophic forgetting, and causes the overﬁtting problem with the biased gradient on a memory.
Revised: This limited access does not guarantee to completely prevent catastrophic forgetting, and causes the overﬁtting problem with the biased gradient on a memory.

Original: In Section 3, we provide the convergence analysis of the previously learned tasks f ( x ) , which are vulnerable to catastrophic forgetting.
Revised: In Section 3, we provide the convergence analysis of the previously learned tasks f ( x ) , which are vulnerable to catastrophic forgetting.

Original: As we denote f i ( x ) as the component, which indicates the loss of sample i from the previously learned tasks with the model parameter x , we deﬁne ∇ f i ( x ) as its gradient.
Revised: As we denote f i ( x ) as the component, which indicates the loss of sample i from the previously learned tasks with the model parameter x , we deﬁne ∇ f i ( x ) as its gradient.

Original: We use I t , J t as the mini-batch of samples at iteration t and denote the mini-batch size | I t | and | J t | as b f , b g throughout the paper.
Revised: We use I t , J t as the mini-batch of samples at iteration t and denote the mini-batch size | I t | and | J t | as b f , b g throughout the paper.

Original: We also note that g j from the current task satisﬁes the above and following assumptions.
Revised: We also note that g j from the current task satisﬁes the above and following assumptions.

Original: To formulate the convergence over iterations, we introduce the Incremental First-order Oracle (IFO) framework (Ghadimi & Lan, 2013), which is deﬁned as a unit of cost by sampling the pair ( ∇ f i ( x ) , f i ( x )) .
Revised: To formulate the convergence over iterations, we introduce the Incremental First-order Oracle (IFO) framework (Ghadimi & Lan, 2013), which is deﬁned as a unit of cost by sampling the pair ( ∇ f i ( x ) , f i ( x )) .

Original: For example, a stochastic gradient descent algorithm requires the cost as much as the batch size b t at each step, and the total cost is the sum of batch sizes (cid:80)
Revised: For example, a stochastic gradient descent algorithm requires the cost as much as the batch size b t at each step, and the total cost is the sum of batch sizes (cid:80)

Original: Tt =1 b t .
Revised: Tt =1 b t .

Original: Let T ( (cid:15) ) be the minimum number of iterations to guarantee (cid:15) -accurate solutions.
Revised: Let T ( (cid:15) ) be the minimum number of iterations to guarantee (cid:15) -accurate solutions.

Original: Then the average bound of IFO complexity is less than or equal to (cid:80) T ( (cid:15) ) t =1 b t .
Revised: Then the average bound of IFO complexity is less than or equal to (cid:80) T ( (cid:15) ) t =1 b t .

Original: To analyze the convergence and compute the IFO complexity, we deﬁne the supremum of loss gap between a local optimal point x 0 and the global optimum x ∗ as
Revised: To analyze the convergence and compute the IFO complexity, we deﬁne the supremum of loss gap between a local optimal point x 0 and the global optimum x ∗ as

Original: [Equation]
Revised: [Equation]

Original: Suppose that sup f ( x 0 ) is same with f ( x ∗ ) , then we have ∆ f = 0 , which might be much smaller than the loss gap of general SGD.
Revised: Suppose that sup f ( x 0 ) is same with f ( x ∗ ) , then we have ∆ f = 0 , which might be much smaller than the loss gap of general SGD.

Original: Without the continual learning scenario, a general nonconvex SGD updates the parameters from an randomly initialized point, which is highly likely to have the large loss f ( x 0 ) .
Revised: Without the continual learning scenario, a general nonconvex SGD updates the parameters from an randomly initialized point, which is highly likely to have the large loss f ( x 0 ) .

Original: Then, ∆ f is the key constant to determine the IFO complexity for converengece as ∆ f is in the numerator of Equation 11.
Revised: Then, ∆ f is the key constant to determine the IFO complexity for converengece as ∆ f is in the numerator of Equation 11.

Original: However, a continual learning algorithm has already converged to a local optimal point x 0 for the previous task f ( x ) and might get a much smaller ∆ f than the general
Revised: However, a continual learning algorithm has already converged to a local optimal point x 0 for the previous task f ( x ) and might get a much smaller ∆ f than the general

Original: SGD.
Revised: SGD.

Original: It means that ∆ f for nonconvex continual learining in Equation 11 dose not have a large impact on the IFO complexity.
Revised: It means that ∆ f for nonconvex continual learining in Equation 11 dose not have a large impact on the IFO complexity.

Original: To generalize the theoretical result, we deine the worst local minimum to explain the upper bound of convergence rate.
Revised: To generalize the theoretical result, we deine the worst local minimum to explain the upper bound of convergence rate.

Original: This implies that ∆ f is not a reason for moving away from stationary points of f by catastrophic forgetting, which we will explain in Section 3.
Revised: This implies that ∆ f is not a reason for moving away from stationary points of f by catastrophic forgetting, which we will explain in Section 3.

Original: We also deﬁne σ f and σ g for f and g , respectively, as the upper bounds on the variance of the stochastic gradients of a given mini-batch.
Revised: We also deﬁne σ f and σ g for f and g , respectively, as the upper bounds on the variance of the stochastic gradients of a given mini-batch.

Original: For brevity, we write only one of them σ f ,
Revised: For brevity, we write only one of them σ f ,

Original: [Equation]
Revised: [Equation]

Original: Throughout the paper, we assume the L -smoothness.
Revised: Throughout the paper, we assume the L -smoothness.

Original: Assumption 1.
Revised: Assumption 1.

Original: f i is L -smooth that there exists a constant L > 0 such that for any x, y ∈ R d ,
Revised: f i is L -smooth that there exists a constant L > 0 such that for any x, y ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)·(cid:107) denotes the Euclidean norm.
Revised: where (cid:107)·(cid:107) denotes the Euclidean norm.

Original: Then the following inequality directly holds that
Revised: Then the following inequality directly holds that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We derive the inequality in Appendix B. With Assumption 1, we can successfully handle individual noncovex objectives for each data point.
Revised: We derive the inequality in Appendix B. With Assumption 1, we can successfully handle individual noncovex objectives for each data point.

Original: In the next section, we investigate nonconvex continual learning with adaptive learning rates to overcome catastrophic forgetting.
Revised: In the next section, we investigate nonconvex continual learning with adaptive learning rates to overcome catastrophic forgetting.

Original:  N ONCONVEX C ONTINUAL L EARNING
Revised:  N ONCONVEX C ONTINUAL L EARNING

Original: We ﬁrst present a theoretical convergence analysis of memory-based continual learning in nonconvex setting.
Revised: We ﬁrst present a theoretical convergence analysis of memory-based continual learning in nonconvex setting.

Original: We use the convergence rate of stochastic gradient methods, which denotes the IFO complexity to reach an (cid:15) -accurate solution for smooth nonconvex ﬁnite-sum problem (Reddi et al., 2016a).
Revised: We use the convergence rate of stochastic gradient methods, which denotes the IFO complexity to reach an (cid:15) -accurate solution for smooth nonconvex ﬁnite-sum problem (Reddi et al., 2016a).

Original: This generic form enables both deep learning and optimization communities to formulate various accelerated gradient methods with theoretical guarantee.
Revised: This generic form enables both deep learning and optimization communities to formulate various accelerated gradient methods with theoretical guarantee.

Original: We seek to understand why catastrophic forgetting happens in terms of the convergence rate, and propose non-convex continual learning (NCCL) algorithms with theoretical convergence analysis.
Revised: We seek to understand why catastrophic forgetting happens in terms of the convergence rate, and propose non-convex continual learning (NCCL) algorithms with theoretical convergence analysis.

Original: 3. P ROBLEM F ORMULATION
Revised: 3. P ROBLEM F ORMULATION

Original: Given two ﬁnite sets P and C at the initial time step t = 0 , we let two sets denote the sets of indices for previously learned data points and upcoming data points, respectively.
Revised: Given two ﬁnite sets P and C at the initial time step t = 0 , we let two sets denote the sets of indices for previously learned data points and upcoming data points, respectively.

Original: Note that the task description for a continual learner is two separate sets.
Revised: Note that the task description for a continual learner is two separate sets.

Original: In this section, we will show a convergence analysis of the model parameter that we has trained on P and starts to learn C .
Revised: In this section, we will show a convergence analysis of the model parameter that we has trained on P and starts to learn C .

Original: Thus, we simply denote a data stream of continual learning as two consecutive sets P and C .
Revised: Thus, we simply denote a data stream of continual learning as two consecutive sets P and C .

Original: We consider our goal as a smooth nonconvex ﬁnite-sum optimization problem with two objectives
Revised: We consider our goal as a smooth nonconvex ﬁnite-sum optimization problem with two objectives

Original: [Equation]
Revised: [Equation]

Original: where f i ( x ) and g j ( x ) denote the objectives of data points i ∈ P and j ∈ C , respectively.
Revised: where f i ( x ) and g j ( x ) denote the objectives of data points i ∈ P and j ∈ C , respectively.

Original: In addition, n f and n g are the numbers of elements for P and C .
Revised: In addition, n f and n g are the numbers of elements for P and C .

Original: To ease exposition, we use a different notation g j ( x ) for a data point j ∈ C , which is usually the same objective function for a data point i ∈ P .
Revised: To ease exposition, we use a different notation g j ( x ) for a data point j ∈ C , which is usually the same objective function for a data point i ∈ P .

Original: To formulate a theoretical convergence analysis of continual learning, we consider a replay memory based method of which memory is a subset of P ∪ C .
Revised: To formulate a theoretical convergence analysis of continual learning, we consider a replay memory based method of which memory is a subset of P ∪ C .

Original: Let a random variable M t ⊂ P ∪ C be the replay memory at time step t ∈ [0 , T ] , whose union is of the form M :=
Revised: Let a random variable M t ⊂ P ∪ C be the replay memory at time step t ∈ [0 , T ] , whose union is of the form M :=

Original: ∪ t M t .
Revised: ∪ t M t .

Original: We focus both the episodic memory and the the replay memory with dropping rule.
Revised: We focus both the episodic memory and the the replay memory with dropping rule.

Original: The episodic memory based methods include GEM (Aljundi et al., 2019), A-GEM (Chaudhry et al., 2019a), and ORTHOG-SUBSPACE (Chaudhry et al., 2020).
Revised: The episodic memory based methods include GEM (Aljundi et al., 2019), A-GEM (Chaudhry et al., 2019a), and ORTHOG-SUBSPACE (Chaudhry et al., 2020).

Original: ER-Reservoir (Chaudhry et al., 2019b) is a replay memory based method with dropping rule, which replaces the dropped sample d ∈ M t with a sample in the stream for C .
Revised: ER-Reservoir (Chaudhry et al., 2019b) is a replay memory based method with dropping rule, which replaces the dropped sample d ∈ M t with a sample in the stream for C .

Original: We now deﬁne the gradient update of continual learning
Revised: We now deﬁne the gradient update of continual learning

Original: [Equation]
Revised: [Equation]

Original: where I t ⊂ M t and J t ⊂ C denote the mini-batches from the replay memory and the current data stream, respectively.
Revised: where I t ⊂ M t and J t ⊂ C denote the mini-batches from the replay memory and the current data stream, respectively.

Original: Here, H t is the union of I t and J t .
Revised: Here, H t is the union of I t and J t .

Original: The adaptive learning rates of ∇ f I t ( x t ) and ∇ g J t ( x t ) are denoted by α H t and β H t which are the functions of H t .
Revised: The adaptive learning rates of ∇ f I t ( x t ) and ∇ g J t ( x t ) are denoted by α H t and β H t which are the functions of H t .

Original: Strictly speaking, the mini-batch I t from M t might contain a datapoint d ∈ C for ER-Reservoir.
Revised: Strictly speaking, the mini-batch I t from M t might contain a datapoint d ∈ C for ER-Reservoir.

Original: We describe the details of the problem in Appendix B and assume that the notation I t indicates a subset of P for convenience.
Revised: We describe the details of the problem in Appendix B and assume that the notation I t indicates a subset of P for convenience.

Original: Equation 7 is a generalized version of continual learning algorithms, which is our novelty to prove the convergence rates in the nonconvex setting for the proposed method, A-GEM, and ER-Reservoir later.
Revised: Equation 7 is a generalized version of continual learning algorithms, which is our novelty to prove the convergence rates in the nonconvex setting for the proposed method, A-GEM, and ER-Reservoir later.

Original: 3. M EMORY - BASED N ONCONVEX C ONTINUAL L EARNING
Revised: 3. M EMORY - BASED N ONCONVEX C ONTINUAL L EARNING

Original: Unlike the conventional smooth nonconvex ﬁnite-sum problem where each mini-batch is iid-sampled from the dataset P ∪ C , the replay memory based continual learning encounters a non-iid stream of data C and has access to a small sized memory M .
Revised: Unlike the conventional smooth nonconvex ﬁnite-sum problem where each mini-batch is iid-sampled from the dataset P ∪ C , the replay memory based continual learning encounters a non-iid stream of data C and has access to a small sized memory M .

Original: Algorithm 1 presents the pseudocode of NCCL by Equation 7.
Revised: Algorithm 1 presents the pseudocode of NCCL by Equation 7.

Original: By the limited access to P , the gradient update for f ( x ) in Equation 7 is a biased estimate of the gradient ∇ f ( x t ) .
Revised: By the limited access to P , the gradient update for f ( x ) in Equation 7 is a biased estimate of the gradient ∇ f ( x t ) .

Original: Speciﬁcally, at the timestep t ,
Revised: Speciﬁcally, at the timestep t ,

Original: [Equation]
Revised: [Equation]

Original: where e t and e M t denote the error term ∇ f I t ( x t ) − ∇ f ( x t ) and the expectation value over I t given
Revised: where e t and e M t denote the error term ∇ f I t ( x t ) − ∇ f ( x t ) and the expectation value over I t given

Original: M t , respectively.
Revised: M t , respectively.

Original: We note that the given replay memory M t with small size at timestep t induces the inevitable overﬁtting bias.
Revised: We note that the given replay memory M t with small size at timestep t induces the inevitable overﬁtting bias.

Original: We ﬁrst state an intermediate result for a single gradient update of NCCL.
Revised: We ﬁrst state an intermediate result for a single gradient update of NCCL.

Original: For ease of exposition, we deﬁne the overﬁtting term B t and the catastrophic forgetting term Γ t as follows:
Revised: For ease of exposition, we deﬁne the overﬁtting term B t and the catastrophic forgetting term Γ t as follows:

Original: [Equation]
Revised: [Equation]

Original: where L is a constant for Lipschitz smoothness.
Revised: where L is a constant for Lipschitz smoothness.

Original: Under Assumption 1, a single gradient update by
Revised: Under Assumption 1, a single gradient update by

Original: Equation 7 satisﬁes the following bound by letting x ← x t +1 and y ← x t :
Revised: Equation 7 satisﬁes the following bound by letting x ← x t +1 and y ← x t :

Original: [Equation]
Revised: [Equation]

Original: This reveals the basic qualitative difference between the conventional nonconvex SGD and NCCL in the convergence rate.
Revised: This reveals the basic qualitative difference between the conventional nonconvex SGD and NCCL in the convergence rate.

Original: Compared to the nonconvex SGD, there exist two terms B t and Γ t in Equation 8.
Revised: Compared to the nonconvex SGD, there exist two terms B t and Γ t in Equation 8.

Original: We group the terms containing e t as B t and the other terms as Γ t .
Revised: We group the terms containing e t as B t and the other terms as Γ t .

Original: We note that the catastrophic forgetting term Γ t has (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) , which is the key aspect of interference and transer (Riemer et al., 2018), and B t includes the error term between the batch of M and the entire dataset
Revised: We note that the catastrophic forgetting term Γ t has (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) , which is the key aspect of interference and transer (Riemer et al., 2018), and B t includes the error term between the batch of M and the entire dataset

Original: P .
Revised: P .

Original: Then, we can quantify the amount of overﬁtting by tracing B t .
Revised: Then, we can quantify the amount of overﬁtting by tracing B t .

Original: To compute the expectation over the stochasticity of NCCL, we derive the expectation of ∇ f M t ( x t ) over the sampling rule of M t .
Revised: To compute the expectation over the stochasticity of NCCL, we derive the expectation of ∇ f M t ( x t ) over the sampling rule of M t .

Original: The episodic memory M t = M 0 for all t is uniformly sampled once from the random sequence of P , and ER-reservoir iteratively samples the replay memory M t by the selection probability
Revised: The episodic memory M t = M 0 for all t is uniformly sampled once from the random sequence of P , and ER-reservoir iteratively samples the replay memory M t by the selection probability

Original: P ( M t | M t − 1 ) .
Revised: P ( M t | M t − 1 ) .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let the history of M t be M [0: t ]
Revised: Let the history of M t be M [0: t ]

Original: =
Revised: =

Original: ( M 0 , · · · , M t ) .
Revised: ( M 0 , · · · , M t ) .

Original: If M 0 is uniformly sampled from P , then both episodic memory and ER-reservoir satisﬁes
Revised: If M 0 is uniformly sampled from P , then both episodic memory and ER-reservoir satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We provide the detailed proof in Appendix B. Note that taking expectation iteratively with respect to the history M [0: t ] is needed to compute the expected value of gradients for M t .
Revised: We provide the detailed proof in Appendix B. Note that taking expectation iteratively with respect to the history M [0: t ] is needed to compute the expected value of gradients for M t .

Original: Since taking the expectation over the stochasticity of NCCL implies the total expectation
Revised: Since taking the expectation over the stochasticity of NCCL implies the total expectation

Original: [Equation]
Revised: [Equation]

Original: we know that E [ B t ] is also 0.
Revised: we know that E [ B t ] is also 0.

Original: Our ﬁrst main result is the following lemma that provides the stepwise change of upper bound.
Revised: Our ﬁrst main result is the following lemma that provides the stepwise change of upper bound.

Original: Input: Previous task P , K task data stream { D 1 , · · · D K } , initial model x 0 , memory M 0 ⊂ P for k = 1 to K do for t = 0 to T − 1 do
Revised: Input: Previous task P , K task data stream { D 1 , · · · D K } , initial model x 0 , memory M 0 ⊂ P for k = 1 to K do for t = 0 to T − 1 do

Original: Uniformly sample a mini-batch I t ⊂ M t with | I t | =
Revised: Uniformly sample a mini-batch I t ⊂ M t with | I t | =

Original: b f
Revised: b f

Original: Uniformly sample a mini-batch J t ⊂ D k with | J t | =
Revised: Uniformly sample a mini-batch J t ⊂ D k with | J t | =

Original: b g
Revised: b g

Original: Compute learning rates α H t and β H t with ∇ f I t ( x t ) and ∇ g J t ( x t )
Revised: Compute learning rates α H t and β H t with ∇ f I t ( x t ) and ∇ g J t ( x t )

Original: x t +1 ← x t − α H t ∇ f I t ( x t ) − β H t ∇ g J t ( x t )
Revised: x t +1 ← x t − α H t ∇ f I t ( x t ) − β H t ∇ g J t ( x t )

Original: Store J t into M t +1 by the rule of replay memory scheme end for x 0 ← x T −
Revised: Store J t into M t +1 by the rule of replay memory scheme end for x 0 ← x T −

Original: P ← P ∪ (cid:83) t J t
Revised: P ← P ∪ (cid:83) t J t

Original: M 0 ← M T − 1 end for
Revised: M 0 ← M T − 1 end for

Original: Lemma 2.
Revised: Lemma 2.

Original: Suppose that Assumption 1 holds and 0 ≤ α H t ≤ 2 L .
Revised: Suppose that Assumption 1 holds and 0 ≤ α H t ≤ 2 L .

Original: Then for x t updated by Algorithm 1, we have the following bound
Revised: Then for x t updated by Algorithm 1, we have the following bound

Original: [Equation]
Revised: [Equation]

Original: where the effect of B t vanishes by E [ B t ] = 0 .
Revised: where the effect of B t vanishes by E [ B t ] = 0 .

Original: The proof is presented in Appendix B. Surprisingly, taking the expectation over M 0 ⊂ P in Equation 10 to handle the stochasticity of choosing M 0 allows us to analyze the convergence of f ( x ) in Section 3.3.
Revised: The proof is presented in Appendix B. Surprisingly, taking the expectation over M 0 ⊂ P in Equation 10 to handle the stochasticity of choosing M 0 allows us to analyze the convergence of f ( x ) in Section 3.3.

Original: We also note that the individual trial with a randomly given M 0 still has the degradation of upper bound by the overﬁtting term B t , although its total expectation over the whole possible trials E [ B t ] is zero.
Revised: We also note that the individual trial with a randomly given M 0 still has the degradation of upper bound by the overﬁtting term B t , although its total expectation over the whole possible trials E [ B t ] is zero.

Original: More speciﬁcally, the worst case sup M 0 (cid:107)∇ f ( x t ) (cid:107) 2 deﬁnitely contains the non-zero value of
Revised: More speciﬁcally, the worst case sup M 0 (cid:107)∇ f ( x t ) (cid:107) 2 deﬁnitely contains the non-zero value of

Original: B t in the upper bound term.
Revised: B t in the upper bound term.

Original: We discuss the more details of the overﬁtting to memory in Appendix D.
Revised: We discuss the more details of the overﬁtting to memory in Appendix D.

Original: 3. T HEORETICAL C ONVERGENCE A NALYSIS
Revised: 3. T HEORETICAL C ONVERGENCE A NALYSIS

Original: We now describe our convergence analysis of the stochastic gradient descent by Equation 7.
Revised: We now describe our convergence analysis of the stochastic gradient descent by Equation 7.

Original: Without any continual learning scheme, the model x t eventually converges to M ∪ C , not P ∪ C .
Revised: Without any continual learning scheme, the model x t eventually converges to M ∪ C , not P ∪ C .

Original: A local optimal point x 0 for P at time step t = 0 , which we have already trained before the continual learning step starts, loses the performance on P very quickly over time.
Revised: A local optimal point x 0 for P at time step t = 0 , which we have already trained before the continual learning step starts, loses the performance on P very quickly over time.

Original: This phenomenon is called catastrophic forgetting.
Revised: This phenomenon is called catastrophic forgetting.

Original: We also note that the overﬁtting to replay memory is another reason of degrading performance for each trial when the model trains on samples in M with replacement after learning on P for previous tasks.
Revised: We also note that the overﬁtting to replay memory is another reason of degrading performance for each trial when the model trains on samples in M with replacement after learning on P for previous tasks.

Original: We obtain the convergence rate of NCCL to a stationary point of f as the following theorem.
Revised: We obtain the convergence rate of NCCL to a stationary point of f as the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let α H t
Revised: Let α H t

Original: = α = c √ T for some 0 < c ≤ 2 √ TL and t ∈ { 0 , · · · , T − 1 } .
Revised: = α = c √ T for some 0 < c ≤ 2 √ TL and t ∈ { 0 , · · · , T − 1 } .

Original: By Lemma 2, the iterates of NCCL satisfy
Revised: By Lemma 2, the iterates of NCCL satisfy

Original: [Equation]
Revised: [Equation]

Original: where A = 1 / (1 − Lα/ 2) .
Revised: where A = 1 / (1 − Lα/ 2) .

Original: For completeness we present a proof in Appendix A.
Revised: For completeness we present a proof in Appendix A.

Original: The loss gap ∆ f and the variance of gradients σ f are ﬁxed values.
Revised: The loss gap ∆ f and the variance of gradients σ f are ﬁxed values.

Original: We ﬁrst note that there exists the cumulative sum of E [Γ t ] unlike the convergence rate of SGD.
Revised: We ﬁrst note that there exists the cumulative sum of E [Γ t ] unlike the convergence rate of SGD.

Original: Most of novelty in our analysis lies in dealing with (cid:80) E
Revised: Most of novelty in our analysis lies in dealing with (cid:80) E

Original: [Γ t ] .
Revised: [Γ t ] .

Original: The (cid:80) T − 1 t =

Revised: The (cid:80) T − 1 t =

Original: E [Γ t ] / √ T is not guaranteed to converge to 0.
Revised: E [Γ t ] / √ T is not guaranteed to converge to 0.

Original: This fact gives rise to catastrophic forgetting in terms of the nondecreasing upper bound.
Revised: This fact gives rise to catastrophic forgetting in terms of the nondecreasing upper bound.

Original: Before showing a bound on IFO calls made by NCCL, we present the convergence rate for g ( x ) .
Revised: Before showing a bound on IFO calls made by NCCL, we present the convergence rate for g ( x ) .

Original: To simplify the proof, we assume that learning rates α H t , β H t are a same ﬁxed value β = c (cid:48) /
Revised: To simplify the proof, we assume that learning rates α H t , β H t are a same ﬁxed value β = c (cid:48) /

Original: √ T.
Revised: √ T.

Original: The assumption is reasonable, because we can observe that RHS of Equation 10 is not perturbed drastically by the learning rates with small value 0 <
Revised: The assumption is reasonable, because we can observe that RHS of Equation 10 is not perturbed drastically by the learning rates with small value 0 <

Original: α H t , β H t ≤ 2 /L (cid:28) 1 .
Revised: α H t , β H t ≤ 2 /L (cid:28) 1 .

Original: Then we have the following lemma.
Revised: Then we have the following lemma.

Original: Lemma 3.
Revised: Lemma 3.

Original: Suppose that I t ∩ J t = ∅ , and the datapoints d ∈ M ∩ P use the same objective function g d = f d .
Revised: Suppose that I t ∩ J t = ∅ , and the datapoints d ∈ M ∩ P use the same objective function g d = f d .

Original: Taking expectation over I t ⊂ M t and J t ⊂ C , we have
Revised: Taking expectation over I t ⊂ M t and J t ⊂ C , we have

Original: [Equation]
Revised: [Equation]

Original: where ∆ g and σ g is the version of loss gap and the variance for g on M ∪ C , respectively.
Revised: where ∆ g and σ g is the version of loss gap and the variance for g on M ∪ C , respectively.

Original: In fact, we note that the convergence rate of g is on M ∪ C .
Revised: In fact, we note that the convergence rate of g is on M ∪ C .

Original: Despite the convergence rate is not on C , it also converges to C trivially.
Revised: Despite the convergence rate is not on C , it also converges to C trivially.

Original: Now we can get the upper bound of (cid:80) E
Revised: Now we can get the upper bound of (cid:80) E

Original: [Γ t ] by Lemma 3.
Revised: [Γ t ] by Lemma 3.

Original: Then we have the following main result.
Revised: Then we have the following main result.

Original: Corollary 1.
Revised: Corollary 1.

Original: The IFO complexity of Algorithm 1 to obtain an (cid:15) -accurate
Revised: The IFO complexity of Algorithm 1 to obtain an (cid:15) -accurate

Original: solution is:
Revised: solution is:

Original: [Equation]
Revised: [Equation]

Original: The proofs of both Lemma 3 and Corollary 1 are presented in Appendix B.
Revised: The proofs of both Lemma 3 and Corollary 1 are presented in Appendix B.

Original: Here, we denote ∞ as the impossibility of convergence.
Revised: Here, we denote ∞ as the impossibility of convergence.

Original: This result reveals that catastrophic forgetting is inevitable without minimizing the cumulative sum (cid:80) E
Revised: This result reveals that catastrophic forgetting is inevitable without minimizing the cumulative sum (cid:80) E

Original: [Γ t ] .
Revised: [Γ t ] .

Original: With replay memory, we know that the inner product in Γ t can be negative by the result of A-GEM (Chaudhry et al., 2019a).
Revised: With replay memory, we know that the inner product in Γ t can be negative by the result of A-GEM (Chaudhry et al., 2019a).

Original: This observation shows that there exists the chance that Γ t becomes negative, and
Revised: This observation shows that there exists the chance that Γ t becomes negative, and

Original: (cid:80) E
Revised: (cid:80) E

Original: [Γ t ] can converge to a constant O (1) as t → ∞ .
Revised: [Γ t ] can converge to a constant O (1) as t → ∞ .

Original: Now we have the proper convergence rate of f ( x ) , and the model can keep the performance on P after learning on C .
Revised: Now we have the proper convergence rate of f ( x ) , and the model can keep the performance on P after learning on C .

Original: On the other hand, when α H t = 0 for all time step, E [Γ t ] is always larger than 0.
Revised: On the other hand, when α H t = 0 for all time step, E [Γ t ] is always larger than 0.

Original: Intuitively, this case implies SGD on C without the replay memory M or other supports, and the cumulative sum (cid:80) E
Revised: Intuitively, this case implies SGD on C without the replay memory M or other supports, and the cumulative sum (cid:80) E

Original: [Γ t ] monotonically increases over time.
Revised: [Γ t ] monotonically increases over time.

Original: Then, f ( x t ) diverges as t → ∞ .
Revised: Then, f ( x t ) diverges as t → ∞ .

Original: We emphasize that the reason of catastrophic forgetting can be explained remarkably by the nonconvex optimization perspective.
Revised: We emphasize that the reason of catastrophic forgetting can be explained remarkably by the nonconvex optimization perspective.

Original: The optimization problem of nonconvex continual learning.
Revised: The optimization problem of nonconvex continual learning.

Original: The local optimal point x t after learning on C might be different from x 0 , because x t moves towards x ∗ P ∪ C as described in Figure 1.
Revised: The local optimal point x t after learning on C might be different from x 0 , because x t moves towards x ∗ P ∪ C as described in Figure 1.

Original: The above observation motivates the following formulation of continual learning to induce (cid:80) E
Revised: The above observation motivates the following formulation of continual learning to induce (cid:80) E

Original: [Γ t ] to converge as t → ∞ and keep (cid:15) -accuracy of f during T iterations.
Revised: [Γ t ] to converge as t → ∞ and keep (cid:15) -accuracy of f during T iterations.

Original: More speciﬁcally, we solve the following problem over T :
Revised: More speciﬁcally, we solve the following problem over T :

Original: [Equation]
Revised: [Equation]

Original: Recall that we have the condition of two learning rates to prove Theorem 1 and Lemma 3.
Revised: Recall that we have the condition of two learning rates to prove Theorem 1 and Lemma 3.

Original: This condition needs to be constraints of the above optimization problem.
Revised: This condition needs to be constraints of the above optimization problem.

Original: Finally, we note that our result ﬁrst shows a theoretical convergence analysis of memory based nonconvex continual learning in the setting of Corollary 1.
Revised: Finally, we note that our result ﬁrst shows a theoretical convergence analysis of memory based nonconvex continual learning in the setting of Corollary 1.

Original: More precisely, we can now explain this method by Problem 14 with theoretical guarantee.
Revised: More precisely, we can now explain this method by Problem 14 with theoretical guarantee.

Original: 3. A DAPTIVE L EARNING R ATES
Revised: 3. A DAPTIVE L EARNING R ATES

Original: As discussed in Section 3.3, the minimization problem 14 leads to tight the upper bound of convergence rate in nonconvex setting.
Revised: As discussed in Section 3.3, the minimization problem 14 leads to tight the upper bound of convergence rate in nonconvex setting.

Original: The result of convergence analysis provides a simple continual learning framework that only adjusts two learning rates in Equation 7.
Revised: The result of convergence analysis provides a simple continual learning framework that only adjusts two learning rates in Equation 7.

Original: First, we review A-GEM and other methods with ﬁxed learning rate such as GSS (Aljundi et al., 2019) and ER-Reservoir (Chaudhry et al., 2019b) through our result.
Revised: First, we review A-GEM and other methods with ﬁxed learning rate such as GSS (Aljundi et al., 2019) and ER-Reservoir (Chaudhry et al., 2019b) through our result.

Original: A-GEM (Chaudhry et al., 2019a) propose a surrogate of ∇ g J t ( x t ) as the following equation to avoid violating the constraint when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) ≤ 0 : ∇ g J t ( x t ) − (cid:28) ∇ f I t ( x t ) (cid:107)∇ f I t ( x t ) (cid:107) , ∇ g J t ( x t ) (cid:29) ∇ f I t ( x t ) (cid:107)∇ f I t ( x t ) (cid:107) .
Revised: A-GEM (Chaudhry et al., 2019a) propose a surrogate of ∇ g J t ( x t ) as the following equation to avoid violating the constraint when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) ≤ 0 : ∇ g J t ( x t ) − (cid:28) ∇ f I t ( x t ) (cid:107)∇ f I t ( x t ) (cid:107) , ∇ g J t ( x t ) (cid:29) ∇ f I t ( x t ) (cid:107)∇ f I t ( x t ) (cid:107) .

Original: We can interpret the surrogate as boosting the learning rate α H t to cancel out the negative component of ∇ f I t ( x t ) on ∇ g J t ( x t ) .
Revised: We can interpret the surrogate as boosting the learning rate α H t to cancel out the negative component of ∇ f I t ( x t ) on ∇ g J t ( x t ) .

Original: After applying the surrogate, E [Γ t ] is reduced but still nonnegative, which we show in Appendix C. A-GEM also fails to fully use the advantage of the case (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: After applying the surrogate, E [Γ t ] is reduced but still nonnegative, which we show in Appendix C. A-GEM also fails to fully use the advantage of the case (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: Fixed learning rates imply that building a replay memory is the key to success in continual learning.
Revised: Fixed learning rates imply that building a replay memory is the key to success in continual learning.

Original: It means that this memory setting is an abundant pool to sample I t , which satisﬁes (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 , as much as possible.
Revised: It means that this memory setting is an abundant pool to sample I t , which satisﬁes (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 , as much as possible.

Original: This can reduce (cid:80) E
Revised: This can reduce (cid:80) E

Original: [Γ t ] .
Revised: [Γ t ] .

Original: However, two above methods cannot enjoy the case (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 enough.
Revised: However, two above methods cannot enjoy the case (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 enough.

Original: As discussed above,we note that Γ t is a quadratic polynomial of β H t where β H t > 0 .
Revised: As discussed above,we note that Γ t is a quadratic polynomial of β H t where β H t > 0 .

Original: We can solve the minimum of the polynomial on β H t when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: We can solve the minimum of the polynomial on β H t when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: By differentiating on β H t , we can easily ﬁnd the minimum Γ ∗ t and the optimal learning rate β ∗ H t
Revised: By differentiating on β H t , we can easily ﬁnd the minimum Γ ∗ t and the optimal learning rate β ∗ H t

Original: [Equation]
Revised: [Equation]

Original: We integrate the above result to propose a better nonconvex continual learning algorithm with the theoretical guarantee.
Revised: We integrate the above result to propose a better nonconvex continual learning algorithm with the theoretical guarantee.

Original: More speciﬁcally, we propose an adaptive learning rate method that can reduce (cid:80) E
Revised: More speciﬁcally, we propose an adaptive learning rate method that can reduce (cid:80) E

Original: [Γ t ] in both cases of (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) ≤ 0 and (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: [Γ t ] in both cases of (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) ≤ 0 and (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: Then, the proposed adaptive learning rate scheme is as follows.
Revised: Then, the proposed adaptive learning rate scheme is as follows.

Original: [Equation]
Revised: [Equation]

Original: We derive the details of the above result in Appendix C.
Revised: We derive the details of the above result in Appendix C.

Original: In addition, our result shows that A-GEM can be viewed as an adaptive learning rate scheme using samples in M t directly.
Revised: In addition, our result shows that A-GEM can be viewed as an adaptive learning rate scheme using samples in M t directly.

Original: This fact implies that there is no difference between A-GEM and experience replay methods.
Revised: This fact implies that there is no difference between A-GEM and experience replay methods.

Original: Figure 1 illustrates intuitively how scaling learning rates achieve the convergence to a mutual stationary point x ∗ P ∪ C as we proved the theoretical complexity in Corollary 1.
Revised: Figure 1 illustrates intuitively how scaling learning rates achieve the convergence to a mutual stationary point x ∗ P ∪ C as we proved the theoretical complexity in Corollary 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed NCCL and the state of the art online continual learning models on the following benchmarks.
Revised: We evaluate the proposed NCCL and the state of the art online continual learning models on the following benchmarks.

Original: We report more detailed experimental materials and results in Appendix.
Revised: We report more detailed experimental materials and results in Appendix.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: We consider a continual learning scenario as a sequence of different tasks from the same dataset.
Revised: We consider a continual learning scenario as a sequence of different tasks from the same dataset.

Original: Permuted-MNIST (Kirkpatrick et al., 2017) is a MNIST (LeCun et al., 1998) based dataset, where each task has a ﬁxed permutation of pixels and transform data points by the permutation to make each task distribution unrelated.
Revised: Permuted-MNIST (Kirkpatrick et al., 2017) is a MNIST (LeCun et al., 1998) based dataset, where each task has a ﬁxed permutation of pixels and transform data points by the permutation to make each task distribution unrelated.

Original: Split-MNIST (Zenke et al., 2017) splits MNIST dataset into ﬁve tasks.
Revised: Split-MNIST (Zenke et al., 2017) splits MNIST dataset into ﬁve tasks.

Original: Each task consists of two classes, for example (1, 7), (3, 4), and has approximately 12K images.
Revised: Each task consists of two classes, for example (1, 7), (3, 4), and has approximately 12K images.

Original: Split-CIFAR10 and 100 also split CIFAR-10 and 100 datasets (Krizhevsky et al., 2009) into ﬁve tasks and 20 tasks, respectively.
Revised: Split-CIFAR10 and 100 also split CIFAR-10 and 100 datasets (Krizhevsky et al., 2009) into ﬁve tasks and 20 tasks, respectively.

Original: Baselines.
Revised: Baselines.

Original: We report the experimental evaluation on the online continual setting which implies a model is trained with a single epoch.
Revised: We report the experimental evaluation on the online continual setting which implies a model is trained with a single epoch.

Original: We compare with the following continual learning baselines.
Revised: We compare with the following continual learning baselines.

Original: iid-ofﬂine and iid-online is a baseline model trained on an i.i.d.
Revised: iid-ofﬂine and iid-online is a baseline model trained on an i.i.d.

Original: stream of the dataset, unlike continual learning, for a single epoch.
Revised: stream of the dataset, unlike continual learning, for a single epoch.

Original: iid-ofﬂine shows the best performance of a given model trained ofﬂine for multiple epochs.
Revised: iid-ofﬂine shows the best performance of a given model trained ofﬂine for multiple epochs.

Original: Fine-tune is a simple method that a model trains observed data naively without any support, such as replay memory.
Revised: Fine-tune is a simple method that a model trains observed data naively without any support, such as replay memory.

Original: Elastic weight consolidation (EWC) is a regularization based method by Fisher Information (Kirkpatrick et al., 2017).
Revised: Elastic weight consolidation (EWC) is a regularization based method by Fisher Information (Kirkpatrick et al., 2017).

Original: ER-Reservoir chooses samples to store from a data stream with a probability proportional to the number of observed data points.
Revised: ER-Reservoir chooses samples to store from a data stream with a probability proportional to the number of observed data points.

Original: The replay memory returns a random subset of samples at each iteration for experience replay.
Revised: The replay memory returns a random subset of samples at each iteration for experience replay.

Original: ER-Reservoir (Chaudhry et al., 2019b) shows a powerful performance in continual learning scenario.
Revised: ER-Reservoir (Chaudhry et al., 2019b) shows a powerful performance in continual learning scenario.

Original: GEM and A-GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a) use gradient episodic memory to overcome catastrophic forgetting.
Revised: GEM and A-GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a) use gradient episodic memory to overcome catastrophic forgetting.

Original: The key idea of GEM is gradient projection with quadratic programming and A-GEM simpliﬁes this procedure.
Revised: The key idea of GEM is gradient projection with quadratic programming and A-GEM simpliﬁes this procedure.

Original: Gradient-based sample selection (GSS) (Aljundi et al., 2019) introduce a sampling method for replay memory, which maximize the diversity of gradients of data points in the memory.
Revised: Gradient-based sample selection (GSS) (Aljundi et al., 2019) introduce a sampling method for replay memory, which maximize the diversity of gradients of data points in the memory.

Original: Continual neural Dirichlet process mixture (CN-DPM)
Revised: Continual neural Dirichlet process mixture (CN-DPM)

Original: (Lee et al., 2020) is a expansion-based method that allocates a subset of the data into a expert network.
Revised: (Lee et al., 2020) is a expansion-based method that allocates a subset of the data into a expert network.

Original: CN-DPM expands the number of experts under the Bayesian nonparametric framework.
Revised: CN-DPM expands the number of experts under the Bayesian nonparametric framework.

Original: Other methods .
Revised: Other methods .

Original: We also compare with iCarl, MER, ORTHOG-SUBSPACE (Chaudhry et al., 2020), and stable-SGD (Mirzadeh et al., 2020).
Revised: We also compare with iCarl, MER, ORTHOG-SUBSPACE (Chaudhry et al., 2020), and stable-SGD (Mirzadeh et al., 2020).

Original: Architecture and Training detail.
Revised: Architecture and Training detail.

Original: For a fair comparison, we follow the commonly used model architecture and hyperparameters of Lee et al.
Revised: For a fair comparison, we follow the commonly used model architecture and hyperparameters of Lee et al.

Original: ; Chaudhry et al.
Revised: ; Chaudhry et al.

Original: For Permuted-MNIST and Split-MNIST, we use fully-connected neural networks with two hidden layers of [400 , 400] or [256 , 256] and ReLU activation.
Revised: For Permuted-MNIST and Split-MNIST, we use fully-connected neural networks with two hidden layers of [400 , 400] or [256 , 256] and ReLU activation.

Original: ResNet-18 with the number of ﬁlters n f = 64 , 20 (He et al., 2016) is applied for Split CIFAR-10 and 100.
Revised: ResNet-18 with the number of ﬁlters n f = 64 , 20 (He et al., 2016) is applied for Split CIFAR-10 and 100.

Original: All experiments conduct a single-pass over the data stream.
Revised: All experiments conduct a single-pass over the data stream.

Original: It is also called 1 epoch or 0.2 epoch (in the case of split tasks).
Revised: It is also called 1 epoch or 0.2 epoch (in the case of split tasks).

Original: We deal both cases with and without the task identiﬁers in the results of split-tasks to compare fairly with baselines.
Revised: We deal both cases with and without the task identiﬁers in the results of split-tasks to compare fairly with baselines.

Original: Batch sizes of data stream and memory are both 10.
Revised: Batch sizes of data stream and memory are both 10.

Original: All reported values are the average values of 5 runs with diffrent seeds, and we also provide standard deviation.
Revised: All reported values are the average values of 5 runs with diffrent seeds, and we also provide standard deviation.

Original: Other miscellaneous settings are the same as in Chaudhry et al.
Revised: Other miscellaneous settings are the same as in Chaudhry et al.

Original: We implement the baselines and the propose method on Tensorﬂow 1.
Revised: We implement the baselines and the propose method on Tensorﬂow 1.

Original: For evaluation, we use an NVIDIA 2080ti GPU along with 3.60 GHz Intel i9-9900K CPU and 64 GB RAM.
Revised: For evaluation, we use an NVIDIA 2080ti GPU along with 3.60 GHz Intel i9-9900K CPU and 64 GB RAM.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: The following tables show our main experimental result, which is averaged over 5 runs.
Revised: The following tables show our main experimental result, which is averaged over 5 runs.

Original: We denote the number of example per class per task at the top of each column.
Revised: We denote the number of example per class per task at the top of each column.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Overall, NCCL with ER-Reservoir outperforms vanilla ER-Reservoir and other baseline methods.
Revised: Overall, NCCL with ER-Reservoir outperforms vanilla ER-Reservoir and other baseline methods.

Original: Our goal is to demonstrate usefulness of the adaptive learning rate scheme, and verify the proposed theoretical convergence analysis.Table ??
Revised: Our goal is to demonstrate usefulness of the adaptive learning rate scheme, and verify the proposed theoretical convergence analysis.

Original: shows the test accuracy on three benchmark datasets.

Original: The batch size of memory and data-stream is 10 for ER-Reservoir, GSS, and our method.

Original: The memory size is 500.

Original: We remark that our adaptive learning rate successfully suppress the catastrophic forgetting when we compared to ER-Reservoir in particular.
Revised: We remark that our adaptive learning rate successfully suppress the catastrophic forgetting when we compared to ER-Reservoir in particular.

Original: ER-Reservoir is known as one of the powerful method even if this method does not consider the inner product condition.
Revised: ER-Reservoir is known as one of the powerful method even if this method does not consider the inner product condition.

Original: A-GEM does not apply any method on (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: A-GEM does not apply any method on (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: These are the points that the proposed method has the better performance.
Revised: These are the points that the proposed method has the better performance.

Original: We clipped β H t to increase the performance.
Revised: We clipped β H t to increase the performance.

Original: As we discussed earlier, we can boost the learning procedure for both previous and current task when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: As we discussed earlier, we can boost the learning procedure for both previous and current task when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: However, we observe that the magnitude of gradients for previous tasks suddenly increases because of catastrophic forgetting in the worst case.
Revised: However, we observe that the magnitude of gradients for previous tasks suddenly increases because of catastrophic forgetting in the worst case.

Original: This fact yields the very large learning rate β H t which forces the model into escaping from the approximate stationary point.
Revised: This fact yields the very large learning rate β H t which forces the model into escaping from the approximate stationary point.

Original: It worsen the overall performance of algorithm, so we clipped β H t to prevent this problem.
Revised: It worsen the overall performance of algorithm, so we clipped β H t to prevent this problem.

Original: Clipping the learning rate still has the effect of reducing the catastrophic forgetting term Γ t .
Revised: Clipping the learning rate still has the effect of reducing the catastrophic forgetting term Γ t .

Original: By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.
Revised: By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.

Original: We show that NCCL is a potentially powerful alternative for popular baseline continual learning algorithms.
Revised: We show that NCCL is a potentially powerful alternative for popular baseline continual learning algorithms.

Original: Even with tiny replay memory, NCCL stil peforms better than some baselines.
Revised: Even with tiny replay memory, NCCL stil peforms better than some baselines.

Original: We note that NCCL variants show the best performance on the forggeting metric.
Revised: We note that NCCL variants show the best performance on the forggeting metric.

Original: It implies that NCCL prevent catastrophic forgetting more efﬁciently than others by minimizing the catastrophic forgetting term in the proposed optimization problem.
Revised: It implies that NCCL prevent catastrophic forgetting more efﬁciently than others by minimizing the catastrophic forgetting term in the proposed optimization problem.

Original: However, the average accuracy of NCCL+ER-ring is slightly lower than ER-Ring.
Revised: However, the average accuracy of NCCL+ER-ring is slightly lower than ER-Ring.

Original: We think that the purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring and
Revised: We think that the purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring and

Original: ORTHOG-subspace.
Revised: ORTHOG-subspace.

Revised: We note that the forgetting metric for NCCL is much lower than ER-Ring.

Revised: This implies that our algorithm is focused on reducing the catastrophic forgetting.

Revised: Then, our method has learned the current task C slightly less than ER-Ring or ER-Reservoir.

Revised: In appendix, we add more results with larger sizes of memory, which shows that our algorithm outperforms baselines.

Revised: In this point of view, we can conclude that the transfer effect by the small size of memory is less effective with adaptive learning rates.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our problem setting is related to the theoretical convergence analysis of smooth nonconvex optimization.
Revised: Our problem setting is related to the theoretical convergence analysis of smooth nonconvex optimization.

Original: Smooth nonconvex ﬁnite-sum optimization problem has been widely employed to derive the theoretical complexity of computation for stochastic gradient methods (Ghadimi & Lan, 2013; 2016;
Revised: Smooth nonconvex ﬁnite-sum optimization problem has been widely employed to derive the theoretical complexity of computation for stochastic gradient methods (Ghadimi & Lan, 2013; 2016;

Original: Lei et al., 2017; Zaheer et al., 2018; Reddi et al., 2016a).
Revised: Lei et al., 2017; Zaheer et al., 2018; Reddi et al., 2016a).

Original: Unlike the convex optimization, the gradient based algorithms are not expected to converge to the global minimum but are evaluated by measuring the convergence rate to the stationary points in the nonconvex case.
Revised: Unlike the convex optimization, the gradient based algorithms are not expected to converge to the global minimum but are evaluated by measuring the convergence rate to the stationary points in the nonconvex case.

Original: Prior works have only focused on analyze the two component problems with additional regularization term (Yao & Kwok, 2016).
Revised: Prior works have only focused on analyze the two component problems with additional regularization term (Yao & Kwok, 2016).

Original: Here, we instead focuses on the problem with the sum of split objectives, which is originally from an entire data stream.
Revised: Here, we instead focuses on the problem with the sum of split objectives, which is originally from an entire data stream.

Original: This setting is interesting since we can observe the convergence of each task respectively.
Revised: This setting is interesting since we can observe the convergence of each task respectively.

Original: Otherwise, (Knoblauch et al., 2020) ﬁrst develops a theoretical approach on continual learning by set theory, and shows that optimal continual learning algorithms and building a perfect memory is equivalent.
Revised: Otherwise, (Knoblauch et al., 2020) ﬁrst develops a theoretical approach on continual learning by set theory, and shows that optimal continual learning algorithms and building a perfect memory is equivalent.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented the ﬁrst generic theoretical convergence analysis of continual learning.
Revised: In this paper, we have presented the ﬁrst generic theoretical convergence analysis of continual learning.

Original: Our proof shows that a training model can circumvent catastrophic forgetting by suppressing the disturbance term on the convergence of previously learned tasks.
Revised: Our proof shows that a training model can circumvent catastrophic forgetting by suppressing the disturbance term on the convergence of previously learned tasks.

Original: We also demonstrate theoretically and empirically that the performance of past tasks by nonconvex continual learning with replay memory is degraded by two separate reasons, catastrophic forgetting and overﬁtting to memory.
Revised: We also demonstrate theoretically and empirically that the performance of past tasks by nonconvex continual learning with replay memory is degraded by two separate reasons, catastrophic forgetting and overﬁtting to memory.

Original: To tackle these problems, nonconvex continual learning uses two methods, scaling learning rates adaptive to mini-batches and sampling mini-batches with small size from the replay memory.
Revised: To tackle these problems, nonconvex continual learning uses two methods, scaling learning rates adaptive to mini-batches and sampling mini-batches with small size from the replay memory.

Original: Finally, it is expected the proposed nonconvex framework is helpful to analyze the convergence rate of other continual learning algorithms.
Revised: Finally, it is expected the proposed nonconvex framework is helpful to analyze the convergence rate of other continual learning algorithms.

Original: A PPENDIX A A DDITIONAL E XPERIMENTAL RESULTS AND DETAILS A. H YPERPARAMETER GRIDS
Revised: A PPENDIX A A DDITIONAL E XPERIMENTAL RESULTS AND DETAILS A. H YPERPARAMETER GRIDS

Original: We report the hyper-paramters grid we used in our experiments below.
Revised: We report the hyper-paramters grid we used in our experiments below.

Original: Except for the proposed algorithm, we adopted the hyper-paramters that are reported in the original papers.
Revised: Except for the proposed algorithm, we adopted the hyper-paramters that are reported in the original papers.

Original: We used grid search to ﬁnd the optimal parameters for each model.
Revised: We used grid search to ﬁnd the optimal parameters for each model.

Original: • ﬁnetune - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • EWC - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] - regularization: [0.1, 1, 10 (MNIST,CIFAR), 100, 1000] • AGEM - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Reservoir - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ORTHOG-SUBSPACE - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST), 0.2, 0.4 (CIFAR), 1.0] • MER - learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR), 0.1, 0.3, 1.0] - within batch meta-learning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] - current batch learning rate multiplier:
Revised: • ﬁnetune - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • EWC - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] - regularization: [0.1, 1, 10 (MNIST,CIFAR), 100, 1000] • AGEM - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Reservoir - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ORTHOG-SUBSPACE - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST), 0.2, 0.4 (CIFAR), 1.0] • MER - learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR), 0.1, 0.3, 1.0] - within batch meta-learning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] - current batch learning rate multiplier:

Original: [1, 2, 5 (CIFAR), 10 (MNIST)] • iid-ofﬂine and iid-online - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-reservoir - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] • NCCL-Ring (default) - learning rate α : [0.003, 0.001(CIFAR), 0.01, 0.03, 0.1, 0.3, 1.0] • NCCL-Reservoir - learning rate α : [0.003(CIFAR), 0.001, 0.01, 0.03, 0.1, 0.3, 1.0]
Revised: [1, 2, 5 (CIFAR), 10 (MNIST)] • iid-ofﬂine and iid-online - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-reservoir - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] • NCCL-Ring (default) - learning rate α : [0.003, 0.001(CIFAR), 0.01, 0.03, 0.1, 0.3, 1.0] • NCCL-Reservoir - learning rate α : [0.003(CIFAR), 0.001, 0.01, 0.03, 0.1, 0.3, 1.0]

Original: A. A DDITIONAL RESULTS
Revised: A. A DDITIONAL RESULTS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: B T HEORETICAL A NALYSIS
Revised: B T HEORETICAL A NALYSIS

Original: In this section we provide the proofs of the results for nonconvex continual learning.
Revised: In this section we provide the proofs of the results for nonconvex continual learning.

Original: We ﬁrst start with the derivation of Equation 5 in Assumption 1.
Revised: We ﬁrst start with the derivation of Equation 5 in Assumption 1.

Original: B. A SSUMPTION AND A DDITIONAL L EMMA
Revised: B. A SSUMPTION AND A DDITIONAL L EMMA

Original: Derivation of Equation 5 .
Revised: Derivation of Equation 5 .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Note that f i is differentiable and nonconvex.
Revised: Note that f i is differentiable and nonconvex.

Original: We deﬁne a function g ( t ) =
Revised: We deﬁne a function g ( t ) =

Original: f i ( y + t ( x − y )) for t ∈ [0 , 1] and an objective function f i .
Revised: f i ( y + t ( x − y )) for t ∈ [0 , 1] and an objective function f i .

Original: By the fundamental theorem of calculus,
Revised: By the fundamental theorem of calculus,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the property, we have
Revised: By the property, we have

Original: [Equation]
Revised: [Equation]

Original: Using the Cauchy-Schwartz inequality,
Revised: Using the Cauchy-Schwartz inequality,

Original: [Equation]
Revised: [Equation]

Original: Since f i satisﬁes Equation
Revised: Since f i satisﬁes Equation

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Let p =
Revised: Let p =

Original: [ p 1 , · · · p D ] , q = [ q 1 , · · · , q D ] be two statistically independent random vectors with dimension D .
Revised: [ p 1 , · · · p D ] , q = [ q 1 , · · · , q D ] be two statistically independent random vectors with dimension D .

Original: Then the expectation of the inner product of two random vectors E [ (cid:104) p, q (cid:105) ] is (cid:80)
Revised: Then the expectation of the inner product of two random vectors E [ (cid:104) p, q (cid:105) ] is (cid:80)

Original: Dd =1 E [ p d ] E [ q d ] .
Revised: Dd =1 E [ p d ] E [ q d ] .

Original: Proof.
Revised: Proof.

Original: By the property of expectation,
Revised: By the property of expectation,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF M AIN R ESULTS
Revised: B. P ROOF OF M AIN R ESULTS

Original: We now show the main results of our work.
Revised: We now show the main results of our work.

Original: Proof of Lemma 1 .
Revised: Proof of Lemma 1 .

Original: To clarify the issue of E M t [ E I t [ e t | M t ]] = 0 , let us explain the details of constructing replay-memory as follows.
Revised: To clarify the issue of E M t [ E I t [ e t | M t ]] = 0 , let us explain the details of constructing replay-memory as follows.

Original: We have considered episodic memory and reservoir sampling in the paper.
Revised: We have considered episodic memory and reservoir sampling in the paper.

Original: We will ﬁrst show the case of episodic memory by describing the sampling method for replay memory.
Revised: We will ﬁrst show the case of episodic memory by describing the sampling method for replay memory.

Original: We can also derive the case of reservoir sampling by simply applying the result of episodic memory.
Revised: We can also derive the case of reservoir sampling by simply applying the result of episodic memory.

Original: Episodic memory (ring buffer).
Revised: Episodic memory (ring buffer).

Original: We divide the entire dataset of continual learning into the previous task P and the current task C on the time step t = 0 .
Revised: We divide the entire dataset of continual learning into the previous task P and the current task C on the time step t = 0 .

Original: For the previous task P , the data stream of
Revised: For the previous task P , the data stream of

Original: P is i.i.d., and its sequence is random on every trial (episode).
Revised: P is i.i.d., and its sequence is random on every trial (episode).

Original: The trial (episode) implies that a continual learning agent learns from an online data stream with two consecutive data sequences of P and C .
Revised: The trial (episode) implies that a continual learning agent learns from an online data stream with two consecutive data sequences of P and C .

Original: Episodic memory takes the last data points of the given memory size m by the First In First
Revised: Episodic memory takes the last data points of the given memory size m by the First In First

Original: Out (FIFO) rule, and holds the entire data points until learning on C is ﬁnished.
Revised: Out (FIFO) rule, and holds the entire data points until learning on C is ﬁnished.

Original: Then, we note that
Revised: Then, we note that

Original: M t =
Revised: M t =

Original: M 0 for all t ≥ 0 and M 0 is uniformly sampled from the i.i.d.
Revised: M 0 for all t ≥ 0 and M 0 is uniformly sampled from the i.i.d.

Original: sequence of P .
Revised: sequence of P .

Original: By the law of total expectation, we derive E M 0 ⊂ P
Revised: By the law of total expectation, we derive E M 0 ⊂ P

Original: [ E I t [ ∇ f I t ( x t ) | M 0 ]] for any x t , ∀ t ≥ 0 as follows.
Revised: [ E I t [ ∇ f I t ( x t ) | M 0 ]] for any x t , ∀ t ≥ 0 as follows.

Original: [Equation]
Revised: [Equation]

Original: We know that M 0 was uniformly sampled from P on each trial before training on the current task C .
Revised: We know that M 0 was uniformly sampled from P on each trial before training on the current task C .

Original: Then, we take expectation with respect to every trial that implies the expected value over the memory distribution M 0 .
Revised: Then, we take expectation with respect to every trial that implies the expected value over the memory distribution M 0 .

Original: We get
Revised: We get

Original: [Equation]
Revised: [Equation]

Original: for any x t , ∀ t .
Revised: for any x t , ∀ t .

Original: We can consider ∇ f M t ( x t ) as a sample mean of P on every trial for any x t , ∀ t ≥ 0 .
Revised: We can consider ∇ f M t ( x t ) as a sample mean of P on every trial for any x t , ∀ t ≥ 0 .

Original: Although x t is constructed iteratively, the expected value of the sample mean for any x t , E M 0 ⊂ P [ ∇ f M 0 ( x t )] is also derived as ∇ f ( x t ) .
Revised: Although x t is constructed iteratively, the expected value of the sample mean for any x t , E M 0 ⊂ P [ ∇ f M 0 ( x t )] is also derived as ∇ f ( x t ) .

Original: Reservoir sampling.
Revised: Reservoir sampling.

Original: To clarify the notation for reservoir sampling ﬁrst, we denote the expectation with respect to the history of replay memory M [0: t ] =
Revised: To clarify the notation for reservoir sampling ﬁrst, we denote the expectation with respect to the history of replay memory M [0: t ] =

Original: ( M 0 , · · · , M t ) as E M [0: t ] .
Revised: ( M 0 , · · · , M t ) as E M [0: t ] .

Original: This is the revised version of E
Revised: This is the revised version of E

Original: M t .
Revised: M t .

Original: Reservoir sampling is the more tricky case than episodic memory, but
Revised: Reservoir sampling is the more tricky case than episodic memory, but

Original: E M [0: t ] [ E I t [ e t | M t ]] = 0 still holds.
Revised: E M [0: t ] [ E I t [ e t | M t ]] = 0 still holds.

Original: Suppose that M 0 is full of the data points from P as the episodic memory is sampled and the mini-batch size from C is 1 for simplicity.
Revised: Suppose that M 0 is full of the data points from P as the episodic memory is sampled and the mini-batch size from C is 1 for simplicity.

Original: The reservoir sampling algorithm drops a data point in M t − 1 and replaces the dropped data point with a data point in the current mini-batch from C with probability p = m/n , where m is the memory size and n is the number of visited data points so far.
Revised: The reservoir sampling algorithm drops a data point in M t − 1 and replaces the dropped data point with a data point in the current mini-batch from C with probability p = m/n , where m is the memory size and n is the number of visited data points so far.

Original: The exact pseudo-code for reservoir sampling is described in [1].
Revised: The exact pseudo-code for reservoir sampling is described in [1].

Original: The replacement procedure uniformly chooses the data point which will be dropped.
Revised: The replacement procedure uniformly chooses the data point which will be dropped.

Original: We can also consider the replacement procedure as follows.
Revised: We can also consider the replacement procedure as follows.

Original: The memory M t for P is reduced in size 1 from
Revised: The memory M t for P is reduced in size 1 from

Original: M t − 1 , and the replaced data point d C from C contributes in terms of ∇ g d C ( x t ) if d C is sampled from the replay memory.
Revised: M t − 1 , and the replaced data point d C from C contributes in terms of ∇ g d C ( x t ) if d C is sampled from the replay memory.

Original: Let M t − 1 =
Revised: Let M t − 1 =

Original: [ d 1 , · · · , d | M t − 1 | ] where | · | denotes the cardinality of the memory.
Revised: [ d 1 , · · · , d | M t − 1 | ] where | · | denotes the cardinality of the memory.

Original: The sample mean of M t − 1 is given as
Revised: The sample mean of M t − 1 is given as

Original: [Equation]
Revised: [Equation]

Original: By the rule of reservoir sampling, we assume that the replacement procedure reduces the memory from M t − 1 to M t with size | M t − 1 | − 1 and the set of remained upcoming data points C t ∈ C from the current data stream for online continual learning is reformulated into C t − 1 ∪ [ d C ] .
Revised: By the rule of reservoir sampling, we assume that the replacement procedure reduces the memory from M t − 1 to M t with size | M t − 1 | − 1 and the set of remained upcoming data points C t ∈ C from the current data stream for online continual learning is reformulated into C t − 1 ∪ [ d C ] .

Original: Then, d C can be resampled from C t − 1 ∪ [ d C ] to be composed of the minibatch of reservoir sampling with the dfferent probability.
Revised: Then, d C can be resampled from C t − 1 ∪ [ d C ] to be composed of the minibatch of reservoir sampling with the dfferent probability.

Original: However, we ignore the probability issue now to focus on the effect of replay-memory on ∇ f .
Revised: However, we ignore the probability issue now to focus on the effect of replay-memory on ∇ f .

Original: Now, we sample M t from M t − 1 , then we get the random vector ∇ f M t ( x t ) as
Revised: Now, we sample M t from M t − 1 , then we get the random vector ∇ f M t ( x t ) as

Original: [Equation]
Revised: [Equation]

Original: where the index i is uniformly sampled from i ∼ [1 , · · · , | M t − 1 | ] , and W ij is the indicator function that W ij is 0 if i =
Revised: where the index i is uniformly sampled from i ∼ [1 , · · · , | M t − 1 | ] , and W ij is the indicator function that W ij is 0 if i =

Original: j else 1.
Revised: j else 1.

Original: The above description implies the dropping rule, and M t can be considered as an uniformly sampled set with size | M t | from M t − 1 .
Revised: The above description implies the dropping rule, and M t can be considered as an uniformly sampled set with size | M t | from M t − 1 .

Original: There could also be M t =
Revised: There could also be M t =

Original: M t − 1 with probability 1 − p = 1 − m/n .
Revised: M t − 1 with probability 1 − p = 1 − m/n .

Original: Then the expectation of ∇ f M t ( x t ) given M t − 1 is derived as
Revised: Then the expectation of ∇ f M t ( x t ) given M t − 1 is derived as

Original: E M
Revised: E M

Original: [Equation]
Revised: [Equation]

Original: [ ∇ f
Revised: [ ∇ f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When we consider the mini-batch sampling, we can formally reformulate the above equation as
Revised: When we consider the mini-batch sampling, we can formally reformulate the above equation as

Original: [Equation]
Revised: [Equation]

Original: Now, we apply the above equation recursively.
Revised: Now, we apply the above equation recursively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Similar to episodic memory, M 0 is uniformly sampled from P .
Revised: Similar to episodic memory, M 0 is uniformly sampled from P .

Original: Therefore, we conclude that
Revised: Therefore, we conclude that

Original: [Equation]
Revised: [Equation]

Original: by taking expectation over the history M [0: t ] =
Revised: by taking expectation over the history M [0: t ] =

Original: ( M 1 , M 2 , · · · , M t ) .
Revised: ( M 1 , M 2 , · · · , M t ) .

Original: Note that taking expectation iteratively with respect to the history M [ t ] is needed to compute the expected value of gradients for M t .
Revised: Note that taking expectation iteratively with respect to the history M [ t ] is needed to compute the expected value of gradients for M t .

Original: However, the result E M 0 , ··· ,M t [ E I t [ e t | M t ]]
Revised: However, the result E M 0 , ··· ,M t [ E I t [ e t | M t ]]

Original: = 0 still holds in terms of expectation.
Revised: = 0 still holds in terms of expectation.

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2 .
Revised: Proof of Lemma 2 .

Original: We analyze the convergence of nonconvex continual learning with replay memory here.
Revised: We analyze the convergence of nonconvex continual learning with replay memory here.

Original: Recall that the gradient update is the following
Revised: Recall that the gradient update is the following

Original: [Equation]
Revised: [Equation]

Original: for all t ∈ { 1 , 2 , · · · , T } .
Revised: for all t ∈ { 1 , 2 , · · · , T } .

Original: Let e t = ∇ f I t ( x t ) − ∇ f ( x t ) .
Revised: Let e t = ∇ f I t ( x t ) − ∇ f ( x t ) .

Original: Since we assume that f, g is L -smooth
Revised: Since we assume that f, g is L -smooth

Original: [Equation]
Revised: [Equation]

Original: we have the following inequality by applying Equation 5:
Revised: we have the following inequality by applying Equation 5:

Original: [Equation]
Revised: [Equation]

Original: To show the proposed theoretical convergence analysis of nonconvex continual learning
Revised: To show the proposed theoretical convergence analysis of nonconvex continual learning

Original: [Equation]
Revised: [Equation]

Original: we deﬁne the catastrophic forgetting term Γ t and the overﬁtting term B t as follows:
Revised: we deﬁne the catastrophic forgetting term Γ t and the overﬁtting term B t as follows:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we can rewrite Equation 24 as
Revised: we can rewrite Equation 24 as

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst note that B t is dependent of the error term e t with the batch I t .
Revised: We ﬁrst note that B t is dependent of the error term e t with the batch I t .

Original: In the continual learning step, an training agent cannot access ∇ f ( x t ) , then we cannot get the exact value of e t .
Revised: In the continual learning step, an training agent cannot access ∇ f ( x t ) , then we cannot get the exact value of e t .

Original: Furthermore, Γ t is dependent of the gradients ∇ f I t ( x t ) , ∇ g I t ( x t ) and the learning rates α H t , β H t .
Revised: Furthermore, Γ t is dependent of the gradients ∇ f I t ( x t ) , ∇ g I t ( x t ) and the learning rates α H t , β H t .

Original: Taking expectations with respect to I t on both sides given J t
Revised: Taking expectations with respect to I t on both sides given J t

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now, taking expectations over the whole stochasticity we obtain
Revised: Now, taking expectations over the whole stochasticity we obtain

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms and assume that 1 1 − Lα Ht / 2 > 0 , we have
Revised: Rearranging the terms and assume that 1 1 − Lα Ht / 2 > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 .
Revised: Proof of Theorem 1 .

Original: Suppose that the learning rate α H t is a constant α = c/ √ T , for c > 0 , 1 − L 2 α = 1 A > 0 .
Revised: Suppose that the learning rate α H t is a constant α = c/ √ T , for c > 0 , 1 − L 2 α = 1 A > 0 .

Original: Then, by summing Equation 10 from t = 0 to T − 1 , we have
Revised: Then, by summing Equation 10 from t = 0 to T − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: We note that a batch I t is sampled from a memory M t ⊂ M which is a random vector whose element is a datapoint d ∈ P ∪ C .
Revised: We note that a batch I t is sampled from a memory M t ⊂ M which is a random vector whose element is a datapoint d ∈ P ∪ C .

Original: Then, taking expectation over I t ⊂ M t ⊂ P ∪ C implies that E [ B t ] = 0 .
Revised: Then, taking expectation over I t ⊂ M t ⊂ P ∪ C implies that E [ B t ] = 0 .

Original: Therefore, we get the minimum of expected square of the norm of gradients as follows:
Revised: Therefore, we get the minimum of expected square of the norm of gradients as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3 .
Revised: Proof of Lemma 3 .

Original: By the assumption, it is equivalent to update on M ∪ C .
Revised: By the assumption, it is equivalent to update on M ∪ C .

Original: Then, the non-convex ﬁnite sum optimization is given as
Revised: Then, the non-convex ﬁnite sum optimization is given as

Original: [Equation]
Revised: [Equation]

Original: where g i is the same function as f i , and | M | is the number of elements in M .
Revised: where g i is the same function as f i , and | M | is the number of elements in M .

Original: This problem can be solved by a simple SGD algorithm (Reddi et al., 2016b).
Revised: This problem can be solved by a simple SGD algorithm (Reddi et al., 2016b).

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let an upper bound β > β H t > 0 .
Revised: Let an upper bound β > β H t > 0 .

Original: For the worst case, the expectation of summing the catastrophic forgetting term over iterations T is
Revised: For the worst case, the expectation of summing the catastrophic forgetting term over iterations T is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we derive the rough upper bound of E [Γ t ] :
Revised: First, we derive the rough upper bound of E [Γ t ] :

Original: [Equation]
Revised: [Equation]

Original: In addition, the supremum of the variance of the mini-batch gradient ∇ g J t ( x t ) is derived as
Revised: In addition, the supremum of the variance of the mini-batch gradient ∇ g J t ( x t ) is derived as

Original: [Equation]
Revised: [Equation]

Original: where n g and b g denotes the size of C and minibatch J t , respectively.
Revised: where n g and b g denotes the size of C and minibatch J t , respectively.

Original: The detailed derivation is shown in [1].
Revised: The detailed derivation is shown in [1].

Original: By the triangular inequality
Revised: By the triangular inequality

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: By plugging the above equation into E [Γ t ]
Revised: By plugging the above equation into E [Γ t ]

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: The sum of catastrophic forgetting term (cid:80) Γ t is corrected as (cid:80) E [Γ t ] .
Revised: The sum of catastrophic forgetting term (cid:80) Γ t is corrected as (cid:80) E [Γ t ] .

Original: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as
Revised: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above equation
Revised: Rearranging the above equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we can write (cid:80) T − 1 t =

Revised: Therefore, we can write (cid:80) T − 1 t =

Original: E [Γ t ] =
Revised: E [Γ t ] =

Original: O ( T ) .
Revised: O ( T ) .

Original: We note that the rough upper bound of (cid:80) Γ t increases monotonically with training step as in the previous result in the paper.
Revised: We note that the rough upper bound of (cid:80) Γ t increases monotonically with training step as in the previous result in the paper.

Original: Now we provide the rigorous derivation of the convergence rate by the result of Corollary 1 as follows:
Revised: Now we provide the rigorous derivation of the convergence rate by the result of Corollary 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: This result is obtained by dividing (cid:80) E
Revised: This result is obtained by dividing (cid:80) E

Original: [Γ t ] by √ T as in the proof of Thm. 1.
Revised: [Γ t ] by √ T as in the proof of Thm. 1.

Original: (cid:4)
Revised: (cid:4)

Original: On the other hand, Γ t can be negative when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .
Revised: On the other hand, Γ t can be negative when (cid:104)∇ f I t ( x t ) , ∇ g J t ( x t ) (cid:105) > 0 .

Original: It implies that the cumulative sum of Γ t does not increase monotonically.
Revised: It implies that the cumulative sum of Γ t does not increase monotonically.

Original: Therefore, for some large number N , we can denote the cumulative sum of Γ t over the ﬁnite steps T as follows:
Revised: Therefore, for some large number N , we can denote the cumulative sum of Γ t over the ﬁnite steps T as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 1 .
Revised: Proof of Corollary 1 .

Original: To formulate the IFO calls, Recall that T ( (cid:15) )
Revised: To formulate the IFO calls, Recall that T ( (cid:15) )

Original: [Equation]
Revised: [Equation]

Original: Additionally, a single IFO call is invested in calculating each step.
Revised: Additionally, a single IFO call is invested in calculating each step.

Original: As seen in Theorem 1, NCCL has a convergence rate of
Revised: As seen in Theorem 1, NCCL has a convergence rate of

Original: [Equation]
Revised: [Equation]

Original: We note that the convergence rate for the worst case is
Revised: We note that the convergence rate for the worst case is

Original: [Equation]
Revised: [Equation]

Original: where the given model diverges on the convergence of f ( x )
Revised: where the given model diverges on the convergence of f ( x )

Original: Then, IFO calls are denoted as ∞ .
Revised: Then, IFO calls are denoted as ∞ .

Original: For the case of Equation 44, we obtain the convergence rate O (1 / √ T ) .
Revised: For the case of Equation 44, we obtain the convergence rate O (1 / √ T ) .

Original: Thus we get O (1 /(cid:15) 2 ) in this case.
Revised: Thus we get O (1 /(cid:15) 2 ) in this case.

Original: [Equation]
Revised: [Equation]

Original: C D ERIVATION OF E QUATIONS IN S ECTION 3. Derivation for A-GEM
Revised: C D ERIVATION OF E QUATIONS IN S ECTION 3. Derivation for A-GEM

Original: Let the surrogate ∇ ˜
Revised: Let the surrogate ∇ ˜

Original: g J t ( x t ) as
Revised: g J t ( x t ) as

Original: [Equation]
Revised: [Equation]

Original: where α H t = α (1 − (cid:104)∇ f It ( x t ) , ∇ g Jt ( x t ) (cid:105) (cid:107)∇ f It ( x t ) (cid:107)) and β H t = α for Equation 7.
Revised: where α H t = α (1 − (cid:104)∇ f It ( x t ) , ∇ g Jt ( x t ) (cid:105) (cid:107)∇ f It ( x t ) (cid:107)) and β H t = α for Equation 7.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Note that this result is smaller than the original E [Γ t ] .
Revised: Note that this result is smaller than the original E [Γ t ] .

Original: Derivation of optimal Γ ∗
Revised: Derivation of optimal Γ ∗

Original: t and β ∗ H t
Revised: t and β ∗ H t

Original: For a ﬁxed learning rate α , we have
Revised: For a ﬁxed learning rate α , we have

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: D O VERFITTING TO REPLAY M EMORY
Revised: D O VERFITTING TO REPLAY M EMORY

Original: In the main text
Revised: In the main text

Original: [Equation]
Revised: [Equation]

Original: we discussed a theoretical convergence analysis of continual learning for a smooth nonconvex ﬁnite-sum optimization problems.
Revised: we discussed a theoretical convergence analysis of continual learning for a smooth nonconvex ﬁnite-sum optimization problems.

Original: The practical continual learning tasks have the restriction on full access to the entire data points of previously learned tasks.
Revised: The practical continual learning tasks have the restriction on full access to the entire data points of previously learned tasks.

Original: Unlike taking expectation over I t ∼ M and M ∼ P ∪ C , we have to compute on the given memory in the practical scenario.
Revised: Unlike taking expectation over I t ∼ M and M ∼ P ∪ C , we have to compute on the given memory in the practical scenario.

Original: Then, we note that E [ B t | M ] (cid:54) = 0 .
Revised: Then, we note that E [ B t | M ] (cid:54) = 0 .

Original: Now we rewirte Equation 26 for the worst case as follows.
Revised: Now we rewirte Equation 26 for the worst case as follows.

Original: [Equation]
Revised: [Equation]

Original: We note that (cid:80) B t is a random variable, which is unpredictible, and choosing ∇ f M ( t ) = ∇ f ( x t ) over entire period is impossible.
Revised: We note that (cid:80) B t is a random variable, which is unpredictible, and choosing ∇ f M ( t ) = ∇ f ( x t ) over entire period is impossible.

Original: Then, the cumulative sum of B t is increasing over T .
Revised: Then, the cumulative sum of B t is increasing over T .

Original: Therefore, we conclude that for the overﬁtting to memory degrades the convergence rate of NCCL empirically.
Revised: Therefore, we conclude that for the overﬁtting to memory degrades the convergence rate of NCCL empirically.