Original: Adaptive Methods for Nonconvex Continual Learning
Revised: Adaptive Methods for Nonconvex Continual Learning

Revised: SEUNGYUBHAN @

Original: author names withheld
Revised: SNU . AC . KR MEARK 234@ SNU .

Revised: AC .

Revised: KR

Original: Under Review for OPT
Revised: TALIUM @ SNU . AC . KR

Revised: JUNGLEE @

Revised: SNU .

Revised: AC .

Revised: KR

Original: Abstract
Revised: Abstract

Original: One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially, and the existing solutions have been driven by the conceptualization of the plasticity-stability dilemma.
Revised: One of the objectives of continual learning is to prevent catastrophic forgetting in learning multiple tasks sequentially, and the existing solutions have been driven by the conceptualization of the plasticity-stability dilemma.

Original: However, the convergence of continual learning for each sequential task is less studied so far.
Revised: However, the convergence of continual learning for each sequential task is less studied so far.

Original: In this paper, we provide a convergence analysis of memory-based continual learning with stochastic gradient descent and empirical evidence that training current tasks causes the cumulative degradation of previous tasks.
Revised: In this paper, we provide a convergence analysis of memory-based continual learning with stochastic gradient descent and empirical evidence that training current tasks causes the cumulative degradation of previous tasks.

Original: We propose an adaptive method for nonconvex continual learning (NCCL), which adjusts step sizes of both previous and current tasks with the gradients.
Revised: We propose an adaptive method for nonconvex continual learning (NCCL), which adjusts step sizes of both previous and current tasks with the gradients.

Original: The proposed method can achieve the same convergence rate as the SGD method when the catastrophic forgetting term which we define in the paper is suppressed at each iteration.
Revised: The proposed method can achieve the same convergence rate as the SGD method when the catastrophic forgetting term which we define in the paper is suppressed at each iteration.

Original: Further, we demonstrate that the proposed algorithm improves the performance of continual learning over existing methods for several image classification tasks.
Revised: Further, we demonstrate that the proposed algorithm improves the performance of continual learning over existing methods for several image classification tasks.

Original: 1. Introduction
Revised: 1. Introduction

Original: Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans.
Revised: Learning new tasks without forgetting previously learned tasks is a key aspect of artificial intelligence to be as versatile as humans.

Original: Unlike the conventional deep learning that observes tasks from an i.i d. distribution, continual learning train sequentially a model on a non-stationary stream of data [28, 31].
Revised: Unlike the conventional deep learning that observes tasks from an i.i.d. distribution, continual learning train sequentially a model on a non-stationary stream of data [28, 31].

Original: The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted [7].
Revised: The continual learning AI systems struggle with catastrophic forgetting when the data access of previously learned tasks is restricted [7].

Original: Although novel continual learning methods successfully learn the non-stationary stream sequentially, studies on the theoretical convergence analysis of both previous tasks and a current task have not yet been addressed.
Revised: Although novel continual learning methods successfully learn the non-stationary stream sequentially, studies on the theoretical convergence analysis of both previous tasks and a current task have not yet been addressed.

Original: In this line of research, nonconvex stochastic optimization problems have been well studied on a single task to train deep neural networks and prove theoretical guarantees of good convergence.
Revised: In this line of research, nonconvex stochastic optimization problems have been well studied on a single task to train deep neural networks and prove theoretical guarantees of good convergence.

Original: Previous continual learning algorithms have introduced novel methods such as a replay memory to store and replay the previously learned examples [1, 2, 19], regularization methods that penalize neural networks [11, 36], Bayesian methods that utilize the uncertainty of parameters or data points [6, 22] and other recent approaches [15, 34].
Revised: Previous continual learning algorithms have introduced novel methods such as a replay memory to store and replay the previously learned examples [1, 2, 19], regularization methods that penalize neural networks [11, 36], Bayesian methods that utilize the uncertainty of parameters or data points [6, 22], and other recent approaches [15, 34].

Original: The study of continual learning in Bayesian frameworks formulate a trained model for previous tasks parameter into an approximate posterior to learn a probabilistic model which have empirically good performance on entire tasks.
Revised: The study of continual learning in Bayesian frameworks formulate a trained model for previous tasks parameter into an approximate posterior to learn a probabilistic model which have empirically good performance on entire tasks.

Original: However,
Revised: However,

Original: Bayesian approaches can fail in practice and it can be hard to analyze the rigorous convergence due to the approximation.
Revised: Bayesian approaches can fail in practice and it can be hard to analyze the rigorous convergence due to the approximation.

Original: The memory-based methods are more straightforward approaches, where the learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep a model staying in a feasible region without losing the performance on the previous tasks.
Revised: The memory-based methods are more straightforward approaches, where the learner stores a small subset of the data for previous tasks into a memory and utilizes the memory by replaying samples to keep a model staying in a feasible region without losing the performance on the previous tasks.

Original: Gradient episodic memory (GEM) [19] first formulated the replay based continual learning as a constrained optimization problem.
Revised: Gradient episodic memory (GEM) [19] first formulated the replay based con- tinual learning as a constrained optimization problem.

Original: This formulation allows us to rephrase the© . constraints on objectives for previous tasks as inequalities based on the inner product of loss gradient vectors for previous tasks and a current task.
Revised: This formulation allows us to rephrase the constraints on objectives for previous tasks as inequalities based on the inner product of loss gradient vectors for previous tasks and a current task.

Original: However, the gradient update by GEM variants cannot guarantee both theoretical and empirical convergence of its constrained optimization problem.
Revised: However, the gradient update by GEM variants cannot guarantee both theoretical and empirical convergence of its constrained optimization problem.

Original: The modified gradient updates do not always satisfy the loss constraint theoretically, and we can also observe the forgetting phenomenon occurs empirically.
Revised: The modified gradient updates do not always satisfy the loss constraint theoretically, and we can also observe the forgetting phenomenon occurs empirically.

Original: It also implies that this intuitive reformulation violates the constrained optimization problem and cannot provide theoretical guarantee to prevent catastrophic forgetting without a rigorous convergence analysis.
Revised: It also implies that this intuitive reformulation violates the constrained optimization problem and cannot provide theoretical guarantee to prevent catastrophic forgetting without a rigorous convergence analysis.

Original: In this work, we explain the cause of catastrophic forgetting by describing continual learning with a smooth nonconvex finite-sum optimization problem.
Revised: In this work, we explain the cause of catastrophic forgetting by describing continual learning with a smooth nonconvex finite-sum optimization problem.

Original: In the standard single task case, SGD [8], ADAM [26], YOGI [35], SVRG [24], and SCSG [17] are the algorithms for solving nonconvex problems that arise in deep learning.
Revised: In the standard single task case, SGD [8], ADAM [26], YOGI [35], SVRG [24], and SCSG [17] are the algorithms for solving nonconvex problems that arise in deep learning.

Original: To analyze the convergence of those algorithms, previous works study the following nonconvex finite-sum problem
Revised: To analyze the convergence of those algorithms, previous works study the following nonconvex finite-sum problem

Original: [Equation]
Revised: [Equation]

Original: where we assume that each objective f i ( x ) with a model x and a data point index i ∈ [ n ] for a dataset with size n (by the convention for notations in nonconvex optimization literature [24]) is nonconvex with L -smoothness assumption.
Revised: where we assume that each objective f i ( x ) with a model x and a data point index i ∈ [ n ] for a dataset with size n (by the convention for notations in nonconvex optimization literature [24]) is nonconvex with L -smoothness assumption.

Original: In general, we denote f i ( x ) as f ( x ; d i ) where d i is a datapoint tuple ( INPUT , OUTPUT ) with index i .
Revised: In general, we denote f i ( x ) as f ( x ; d i ) where d i is a datapoint tuple ( INPUT , OUTPUT ) with index i .

Original: We expect that a stochastic gradient descent based algorithm reaches a stationary point instead of the global minimum in nonconvex optimization.
Revised: We expect that a stochastic gradient descent based algorithm reaches a stationary point instead of the global minimum in nonconvex optimization.

Original: Unlike the convex case, the convergence is generally measured by the expectation of the squared norm of a gradient E ∥∇ f ( x ) ∥ 2 .
Revised: Unlike the convex case, the convergence is generally measured by the expectation of the squared norm of a gradient E ∥∇ f ( x ) ∥ 2 .

Original: The theoretical computational complexity is derived from the ϵ -accurate solution, which is also known as a stationary point with E ∥∇ f
Revised: The theoretical computational complexity is derived from the ϵ -accurate solution, which is also known as a stationary point with E ∥∇ f

Original: ( x ) ∥ 2 ≤ ϵ .
Revised: ( x ) ∥ 2 ≤ ϵ .

Original: The general nonconvex finite-sum problems assume that all data points can be sampled during training iterations.
Revised: The general nonconvex finite-sum problems assume that all data points can be sampled during training iterations.

Original: This fact is an obstacle to directly apply (1) for continual learning problem.
Revised: This fact is an obstacle to directly apply (1) for continual learning problem.

Original: We provide a solution of the above issue by leveraging memory-based methods, which allow models to access a partial access to the dataset of previous tasks.
Revised: We provide a solution of the above issue by leveraging memory-based methods, which allow models to access a partial access to the dataset of previous tasks.

Original: In this setting, we can analyze nonconvex stochastic optimization problems on the convergence of previous tasks with limited access.
Revised: In this setting, we can analyze nonconvex stochastic optimization problems on the convergence of previous tasks with limited access.

Original: Similar with adaptive methods for noncovex optimization, we apply adaptive step sizes during optimization to minimize forgetting with theoretical guarantee.
Revised: Similar with adaptive methods for noncovex optimization, we apply adaptive step sizes during optimization to minimize forgetting with theoretical guarantee.

Original: 2. Preliminaries
Revised: 2. Preliminaries

Original: Suppose that we observe the learning procedure on a data stream of continual learning at some arbitrary observation point.
Revised: Suppose that we observe the learning procedure on a data stream of continual learning at some arbitrary observation point.

Original: Let us consider time step t = 0 as given observation point.
Revised: Let us consider time step t = 0 as given observation point.

Original: We define the previous task P for t < 0 as all visited data points and the current task C for t ≥ 0 as all data points which will face in the future.
Revised: We define the previous task P for t < 0 as all visited data points and the current task C for t ≥ 0 as all data points which will face in the future.

Original: Then, P and C can be defined as the sets of data points in P and
Revised: Then, P and C can be defined as the sets of data points in P and

Original: C at time step t = 0 , respectively.
Revised: C at time step t = 0 , respectively.

Original: Note that the above task description is based on not a sequence of multiple tasks, but two separate sets to analyze the convergence of each of P and C when starting to update the given batch at the current task C at some arbitrary observation point.
Revised: Note that the above task description is based on not a sequence of multiple tasks, but two separate sets to analyze the convergence of each of P and C when starting to update the given batch at the current task C at some arbitrary observation point.

Original: For clarity, we use f ( x ) =
Revised: For clarity, we use f ( x ) =

Original: h ( x ) | P and g ( x ) = h ( x )
Revised: h ( x ) | P and g ( x ) = h ( x )

Original: | C for the restriction of h to each dataset P and C , respectively.
Revised: | C for the restriction of h to each dataset P and C , respectively.

Original: f i ( x ) and g j ( x ) also denotes the objective terms induced from data where each index is i ∈ P and j ∈ C , respectively.
Revised: f i ( x ) and g j ( x ) also denotes the objective terms induced from data where each index is i ∈ P and j ∈ C , respectively.

Original: We consider a continual learning problem as a smooth nonconvex finite-sum
Revised: We consider a continual learning problem as a smooth nonconvex finite-sum

Original: ∇
Revised: ∇

Original: − β
Revised: − β

Original: ∇ end optimization problem with two decomposed objectives
Revised: ∇ end optimization problem with two decomposed objectives

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where n f and n g are the numbers of elements for P and C .
Revised: where n f and n g are the numbers of elements for P and C .

Original: Suppose that the replay memories M t for time step ∈ [0 , T ] are random variables which are the subsets of P ∪ C to cover prior memory-based approaches [2, 3].
Revised: Suppose that the replay memories M t for time step ∈ [0 , T ] are random variables which are the subsets of P ∪ C to cover prior memory-based approaches [2, 3].

Original: To formulate an algorithm for memory-based approaches, we define mini-batches I t which are sampled from a memory M t at step t .
Revised: To formulate an algorithm for memory-based approaches, we define mini-batches I t which are sampled from a memory M t at step t .

Original: We now define the stochastic update of memory-based method
Revised: We now define the stochastic update of memory-based method

Original: [Equation]
Revised: [Equation]

Original: (3) where I t ⊂ M t and J t ⊂ C denote the mini-batches from the replay memory and the current data stream, respectively.
Revised: (3) where I t ⊂ M t and J t ⊂ C denote the mini-batches from the replay memory and the current data stream, respectively.

Original: Here, H t is the union of I t and J t .
Revised: Here, H t is the union of I t and J t .

Original: In addition, for a given set S , ∇ f S ( x t ) , ∇ g S ( x t ) denote the loss gradient of a model x t with the mini-batch S at time step t
Revised: In addition, for a given set S , ∇ f S ( x t ) , ∇ g S ( x t ) denote the loss gradient of a model x t with the mini-batch S at time step t

Original: The adaptive step sizes (learning rates) of ∇ f I t ( x t ) and ∇ g J t ( x t ) are denoted by α H t and β H t which are the functions of H t .
Revised: The adaptive step sizes (learning rates) of ∇ f I t ( x t ) and ∇ g J t ( x t ) are denoted by α H t and β H t which are the functions of H t .

Original: It should be noted the mini-batch I t from M t might contain a datapoint j ∈ C for some cases, such as ER-Reservoir.
Revised: It should be noted the mini-batch I t from M t might contain a datapoint j ∈ C for some cases, such as ER-Reservoir.

Original: In appendix, we provide assumptions used in the proof.
Revised: In appendix, we provide assumptions used in the proof.

Original: 3. Continual Learning as Nonconvex Optimization
Revised: 3. Continual Learning as Nonconvex Optimization

Original: We present a theoretical convergence analysis of memory-based continual learning in nonconvex setting.
Revised: We present a theoretical convergence analysis of memory-based continual learning in nonconvex setting.

Original: We aim to understand why catastrophic forgetting occurs in terms of the convergence rate, and reformulate the optimization problem of continual learning into a nonconvex setting with theoretical guarantee.
Revised: We aim to understand why catastrophic forgetting occurs in terms of the convergence rate, and reformulate the optimization problem of continual learning into a nonconvex setting with theoretical guarantee.

Original: For completeness we present all proofs in Appendix D.
Revised: For completeness we present all proofs in Appendix D.

Original: The amount of effect on convergence by a single update can be measured by using Assumption 4.
Revised: The amount of effect on convergence by a single update can be measured by using Assumption 4.

Original: By Equation 7, we have
Revised: By Equation 7, we have

Original: [Equation]
Revised: [Equation]

Original: by letting x ← x t +1 and y ← x t .
Revised: by letting x ← x t +1 and y ← x t .

Original: We now propose two terms of interest in a gradient update of nonconvex continual learning (NCCL).
Revised: We now propose two terms of interest in a gradient update of nonconvex continual learning (NCCL).

Original: We define the overfitting term B t and the catastrophic forgetting term Γ t as follows:
Revised: We define the overfitting term B t and the catastrophic forgetting term Γ t as follows:

Original: [Equation]
Revised: [Equation]

Original: These two terms essentially measure the performance degradation in NCCL with respect to time.
Revised: These two terms essentially measure the performance degradation in NCCL with respect to time.

Original: It should be noted that Γ t has ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ , which is a key factor to determine interference and transfer [27].
Revised: It should be noted that Γ t has ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ , which is a key factor to determine interference and transfer [27].

Original: On the other hand, B t includes e t , which is an error gradient between the batch from M t and the entire dataset P .
Revised: On the other hand, B t includes e t , which is an error gradient between the batch from M t and the entire dataset P .

Original: We now describe a convergence analysis of Algorithm 1.
Revised: We now describe a convergence analysis of Algorithm 1.

Original: We telescope over training iterations for the current task, which leads to obtain the following theorem.
Revised: We telescope over training iterations for the current task, which leads to obtain the following theorem.

Original: Theorem 1 Let α H t
Revised: Theorem 1 Let α H t

Original: = α = c √ T for some 0 < c ≤√
Revised: = α = c √ T for some 0 < c ≤√

Original: TL and t ∈ { 0 , · · · , T − 1 } .
Revised: TL and t ∈ { 0 , · · · , T − 1 } .

Original: By Lemma 7, the iterates of NCCL satisfy min t
Revised: By Lemma 7, the iterates of NCCL satisfy min t

Original: [Equation]
Revised: [Equation]

Original: where A = 1 / (1 − Lα/ 2) .
Revised: where A = 1 / (1 − Lα/ 2) .

Original: One key observation is that E [Γ t ] are cumulatively added on the upper bound of E ∥∇ f ( x ) ∥ 2 .
Revised: One key observation is that E [Γ t ] are cumulatively added on the upper bound of E ∥∇ f ( x ) ∥ 2 .

Original: The loss gap ∆ f and the variance of gradients σ f are fixed values.
Revised: The loss gap ∆ f and the variance of gradients σ f are fixed values.

Original: In practice, tightening (cid:80) t E [Γ t ] appears to be critical for the performance of NCCL.
Revised: In practice, tightening (cid:80) t E [Γ t ] appears to be critical for the performance of NCCL.

Original: However, (cid:80) T − 1 t =

Revised: However, (cid:80) T − 1 t =

Original: E [Γ t ] / √ T is not guaranteed to converge to 0.
Revised: E [Γ t ] / √ T is not guaranteed to converge to 0.

Original: This fact gives rise to forgetting in terms of a nondecreasing upper bound.
Revised: This fact gives rise to forgetting in terms of a nondecreasing upper bound.

Original: Lemma 2 Let an upper bound β > β H t > 0 .
Revised: Lemma 2 Let an upper bound β > β H t > 0 .

Original: Consider two cases, β < α and β ≥ α for α in
Revised: Consider two cases, β < α and β ≥ α for α in

Original: Theorem 1.
Revised: Theorem 1.

Original: We have the following bound
Revised: We have the following bound

Original: [Equation]
Revised: [Equation]

Original: Corollary 3 For β H t <
Revised: Corollary 3 For β H t <

Original: α = c √ T for all t , the IFO complexity of Algorithm 1 for f ( x ) to obtain an ϵ -accurate solution is as follows:
Revised: α = c √ T for all t , the IFO complexity of Algorithm 1 for f ( x ) to obtain an ϵ -accurate solution is as follows:

Original: [Equation]
Revised: [Equation]

Original: To prevent forgetting, β H t should be lower than the step size of f ( x ) , α H t .
Revised: To prevent forgetting, β H t should be lower than the step size of f ( x ) , α H t .

Original: We provide more details in Appendix.
Revised: We provide more details in Appendix.

Original: It should also be noted that E M [1: t ]
Revised: It should also be noted that E M [1: t ]

Original: [ B t | M 0 ] is not always 0 for any M 0 .
Revised: [ B t | M 0 ] is not always 0 for any M 0 .

Original: This implies that, from time step 0, each trial with different given M 0 also has the non-zero cumulative sum (cid:80)
Revised: This implies that, from time step 0, each trial with different given M 0 also has the non-zero cumulative sum (cid:80)

Original: E M [1: T ] [ B t | M 0 ] , which occurs overestimating bias.
Revised: E M [1: T ] [ B t | M 0 ] , which occurs overestimating bias.

Original: We reformulate the constraint optimization problem for CL [19] into the problem on E [Γ t ] in Appendix D.3.
Revised: We reformulate the constraint optimization problem for CL [19] into the problem on E [Γ t ] in Appendix D.3.

Original: We note that E [Γ t ] is a quadratic
Revised: We note that E [Γ t ] is a quadratic

Original: [Figure]
Revised: [Figure]

Original: polynomial of β H t where β H t > 0 .
Revised: polynomial of β H t where β H t > 0 .

Original: The analytic minimum on β H t can be obtained when Λ H t >by differentiating on β H t .
Revised: The analytic minimum on β H t can be obtained when Λ H t >by differentiating on β H t .

Original: Then the minimum E [Γ ∗ t ] and the optimal β ∗ H t can be obtained as
Revised: Then the minimum E [Γ ∗ t ] and the optimal β ∗ H t can be obtained as

Original: [Equation]
Revised: [Equation]

Original: To satisfy the constraints of (9) in Appendix, we should update ∇ f I t ( x t ) with non-zero step size and β H t < α H t for all t .
Revised: To satisfy the constraints of (9) in Appendix, we should update ∇ f I t ( x t ) with non-zero step size and β H t < α H t for all t .

Original: Then the proposed adaptive method is given by α
Revised: Then the proposed adaptive method is given by α

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: where α = c/ √ T and δ is some constant 0 < δ ≪ 1 .
Revised: where α = c/ √ T and δ is some constant 0 < δ ≪ 1 .

Original: For Λ H t ≤ 0 , see details in Appendix D.4.
Revised: For Λ H t ≤ 0 , see details in Appendix D.4.

Original: 4. Experiments
Revised: 4. Experiments

Original: We use two following metrics to evaluate algorithms.
Revised: We use two following metrics to evaluate algorithms.

Original: (1) Average accuracy is defined as 1 T (cid:80) Tj =1 a T,j , where a i,j denotes the test accuracy on task j after training on task i .
Revised: (1) Average accuracy is defined as 1 T (cid:80) Tj =1 a T,j , where a i,j denotes the test accuracy on task j after training on task i .

Original: (2) Forgetting is the average maximum forgetting is defined as
Revised: (2) Forgetting is the average maximum forgetting is defined as

Original: [Equation]
Revised: [Equation]

Original: Due to limited space, we report the training details and missing results with additional datasets in Appendix.
Revised: Due to limited space, we report the training details and missing results with additional datasets in Appendix.

Original: The following table show our main experimental results, which is averaged over 5 runs.
Revised: The following table show our main experimental results, which is averaged over 5 runs.

Original: We denote the number of example per class per task at the top of each column.
Revised: We denote the number of example per class per task at the top of each column.

Original: Overall, NCCL + memory schemes outperform baseline methods especially in the forgetting metric.
Revised: Overall, NCCL + memory schemes outperform baseline methods especially in the forgetting metric.

Original: Our goal is to demonstrate the usefulness of the adaptive methods to reduce the catastrophic forgetting, and to show empirical evidence for our convergence analysis.
Revised: Our goal is to demonstrate the usefulness of the adaptive methods to reduce the catastrophic forgetting, and to show empirical evidence for our convergence analysis.

Original: We remark that NCCL successfully suppress forgetting by a large margin compared to baselines.
Revised: We remark that NCCL successfully suppress forgetting by a large margin compared to baselines.

Original: It is noted that NCCL also outperforms A-GEM, which does not maximize transfer when Λ H t > 0 and violates the proposed constraints in (9)
Revised: It is noted that NCCL also outperforms A-GEM, which does not maximize transfer when Λ H t > 0 and violates the proposed constraints in (9)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We now investigate the proposed terms with regard to memory-based continual learning, (cid:80) E [Γ t ] and B t .
Revised: We now investigate the proposed terms with regard to memory-based continual learning, (cid:80) E [Γ t ] and B t .

Original: To verify our theoretical analysis, in Figure 1 we show the cumulative catastrophic forgetting term (cid:80) t
Revised: To verify our theoretical analysis, in Figure 1 we show the cumulative catastrophic forgetting term (cid:80) t

Original: E [Γ t ] is the key factor of the convergence of the first task in split-CIFAR100.
Revised: E [Γ t ] is the key factor of the convergence of the first task in split-CIFAR100.

Original: During continual learning, (cid:80) t E [Γ t ] increases in all methods of Figure 1b.
Revised: During continual learning, (cid:80) t E [Γ t ] increases in all methods of Figure 1b.

Original: Figrure 1a, 1d, 1e show that the larger (cid:80) t E [Γ t ] causes the larger forgetting and ∥∇ f ( x ) ∥ for the first task.
Revised: Figrure 1a, 1d, 1e show that the larger (cid:80) t E [Γ t ] causes the larger forgetting and ∥∇ f ( x ) ∥ for the first task.

Original: We can observe that ∥∇ f ( x ) ∥ gets larger than 4, which is for the red line, when (cid:80) t E [Γ t ] becomes larger than 2.
Revised: We can observe that ∥∇ f ( x ) ∥ gets larger than 4, which is for the red line, when (cid:80) t E [Γ t ] becomes larger than 2.

Original: We also verify that the theoretical result E t [ B t ] = 0 is valid in Figure 1f.
Revised: We also verify that the theoretical result E t [ B t ] = 0 is valid in Figure 1f.

Original: It implies that the empirical results of Lemma 5, which show the effect of B t on Equation 14.
Revised: It implies that the empirical results of Lemma 5, which show the effect of B t on Equation 14.

Original: Furthermore, the memory bias helps to tighten the convergence rate of P by having negative values in practice.
Revised: Furthermore, the memory bias helps to tighten the convergence rate of P by having negative values in practice.

Original: Even with tiny memory, the estimated B t has much smaller value than E [Γ t ] as we can observe in Figure 1.
Revised: Even with tiny memory, the estimated B t has much smaller value than E [Γ t ] as we can observe in Figure 1.

Original: For experience replay, we need not to worry about the degradation by memory bias and would like to emphasize that tiny memory can slightly help to keep the convergence on P empirically.
Revised: For experience replay, we need not to worry about the degradation by memory bias and would like to emphasize that tiny memory can slightly help to keep the convergence on P empirically.

Original: We conclude that the overfitting bias term might not be a major factor in degrading the performance of continual learning agent when it is compared to the catastrophic forgetting term Γ t .
Revised: We conclude that the overfitting bias term might not be a major factor in degrading the performance of continual learning agent when it is compared to the catastrophic forgetting term Γ t .

Original: 5. Conclusion
Revised: 5. Conclusion

Original: In this paper, we have presented a theoretical convergence analysis of continual learning.
Revised: In this paper, we have presented a theoretical convergence analysis of continual learning.

Original: Our proof shows that a training model can circumvent catastrophic forgetting by suppressing catastrophic forgetting term in terms of the convergence on previous task.
Revised: Our proof shows that a training model can circumvent catastrophic forgetting by suppressing catastrophic forgetting term in terms of the convergence on previous task.

Original: We demonstrate theoretically and empirically that adaptive methods with memory schemes show the better performance in terms of forgetting.
Revised: We demonstrate theoretically and empirically that adaptive methods with memory schemes show the better performance in terms of forgetting.

Original: It is also noted that there exist two factors on the convergence of previous task: catastrophic forgetting and overfitting to memory.
Revised: It is also noted that there exist two factors on the convergence of previous task: catastrophic forgetting and overfitting to memory.

Original: Finally, it is expected the proposed nonconvex framework is helpful to analyze the convergence rate of other continual learning algorithms.
Revised: Finally, it is expected the proposed nonconvex framework is helpful to analyze the convergence rate of other continual learning algorithms.

Original: Appendix A. Review of terminology
Revised: Appendix A. Review of terminology

Original: (Restriction of f )
Revised: (Restriction of f )

Original: If f : A → B and if A 0 is a subset of A , then the restriction of f to A 0 is the function
Revised: If f : A → B and if A 0 is a subset of A , then the restriction of f to A 0 is the function

Original: [Equation]
Revised: [Equation]

Original: given by f | A 0 ( x ) =
Revised: given by f | A 0 ( x ) =

Original: f
Revised: f

Original: ( x ) for x ∈ A 0 .
Revised: ( x ) for x ∈ A 0 .

Original: A.1. Summary of notations
Revised: A.1. Summary of notations

Original: Notations
Revised: Notations

Original: Definitions
Revised: Definitions

Original: Notations
Revised: Notations

Original: Definitions x model parameter
Revised: Definitions x model parameter

Original: H t the union of I t and J t
Revised: H t the union of I t and J t

Original: P previous task n f the number of data points in P
Revised: P previous task n f the number of data points in P

Original: C current task n g the number of data points in C
Revised: C current task n g the number of data points in C

Original: P dataset of P ⟨· , ·⟩ inner product
Revised: P dataset of P ⟨· , ·⟩ inner product

Original: C dataset of C
Revised: C dataset of C

Original: L
Revised: L

Original: L -smoothness constant h ( x ) mean loss of x on entire datasets α H t adaptive step size for f with H t f ( x ) mean loss of x on P β H t adaptive step size for g with H t g ( x ) mean loss of x on C
Revised: L -smoothness constant h ( x ) mean loss of x on entire datasets α H t adaptive step size for f with H t f ( x ) mean loss of x on P β H t adaptive step size for g with H t g ( x ) mean loss of x on C

Original: M t memory at time t f i ( x ) loss of x on a data point
Revised: M t memory at time t f i ( x ) loss of x on a data point

Original: i ∈ P e t error of estimate f at time t g j ( x ) loss of x on a data point j ∈ C e M t error of estimate f with M t
Revised: i ∈ P e t error of estimate f at time t g j ( x ) loss of x on a data point j ∈ C e M t error of estimate f with M t

Original: f I t ( x ) mini-batch loss of x on a batch
Revised: f I t ( x ) mini-batch loss of x on a batch

Original: I t f M t mean loss of x with M t g J t ( x ) mini-batch loss of x on a batch J t
Revised: I t f M t mean loss of x with M t g J t ( x ) mini-batch loss of x on a batch J t

Original: M [ t 1: t 2] the history of memory from t 1 to t
Revised: M [ t 1: t 2] the history of memory from t 1 to t

Original: I t minibatch sampled from P
Revised: I t minibatch sampled from P

Original: B t memory bias term at t
Revised: B t memory bias term at t

Original: J t minibatch sampled from C Γ t forgetting term at t
Revised: J t minibatch sampled from C Γ t forgetting term at t

Original: E t total expectation from 0 to time t Λ H t inner product between ∇ f I t and ∇ g J t
Revised: E t total expectation from 0 to time t Λ H t inner product between ∇ f I t and ∇ g J t

Original: Appendix B. Related Work
Revised: Appendix B. Related Work

Original: Memory-based methods.
Revised: Memory-based methods.

Original: Early memory-based methods utilize memory by the distillation [18, 23] or the optimization constraint [2, 19].
Revised: Early memory-based methods utilize memory by the distillation [18, 23] or the optimization constraint [2, 19].

Original: Especially, A-GEM [2] simplifies the approach for constraint violated update steps as the projected gradient on a reference gradient which ensures that the average memory loss over previous tasks does not increase.
Revised: Especially, A-GEM [2] simplifies the approach for constraint violated update steps as the projected gradient on a reference gradient which ensures that the average memory loss over previous tasks does not increase.

Original: Recent works [3, 4, 27] have shown that updating the gradients on memory directly, which is called experience replay, is a light and prominent approach.
Revised: Recent works [3, 4, 27] have shown that updating the gradients on memory directly, which is called experience replay, is a light and prominent approach.

Original: We focus on convergence of continual learning, but the above methods focus on increasing the empirical performance without theoretical guarantee.
Revised: We focus on convergence of continual learning, but the above methods focus on increasing the empirical performance without theoretical guarantee.

Original: Our analysis provides a legitimate theoretical convergence analysis under the standard smooth nonconvex finite-sum optimization problem setting.
Revised: Our analysis provides a legitimate theoretical convergence analysis under the standard smooth nonconvex finite-sum optimization problem setting.

Original: Further, [12] shows the perfect memory for optimal continual learning is NP-hard by using set-theory, but the quantitative analysis of performance degradation is less studied.
Revised: Further, [12] shows the perfect memory for optimal continual learning is NP-hard by using set-theory, but the quantitative analysis of performance degradation is less studied.

Original: Adaptive step sizes in nonconvex setting.
Revised: Adaptive step sizes in nonconvex setting.

Original: Adaptive step sizes under smooth nonconvex finitesum optimization problem have been studied on general single task cases [26, 35, 37] recently.
Revised: Adaptive step sizes under smooth nonconvex finitesum optimization problem have been studied on general single task cases [26, 35, 37] recently.

Original: [29, 30, 37] have revealed that there exists a heavy-tailed noise in some optimization problems for neural networks, such as attention models, and [37] shows that adaptive methods are helpful to achieve the faster convergence under the heavy-tailed distribution where stochastic gradients are poorly concentrated around the mean.
Revised: [29, 30, 37] have revealed that there exists a heavy-tailed noise in some optimization problems for neural networks, such as attention models, and [37] shows that adaptive methods are helpful to achieve the faster convergence under the heavy-tailed distribution where stochastic gradients are poorly concentrated around the mean.

Original: In this work, we treat the continual learning problem where stochastic gradients of previous tasks are considered as the out-of-distribution samples in regard to a current task, and develop adaptive methods which are well-performed in continual learning.
Revised: In this work, we treat the continual learning problem where stochastic gradients of previous tasks are considered as the out-of-distribution samples in regard to a current task, and develop adaptive methods which are well-performed in continual learning.

Original: Regularization based methods.
Revised: Regularization based methods.

Original: EWC has an additional penalization loss that prevent the update of parameters from losing the information of previous tasks.
Revised: EWC has an additional penalization loss that prevent the update of parameters from losing the information of previous tasks.

Original: When we update a model with
Revised: When we update a model with

Original: EWC, we have two gradient components from the current task and the penalization loss.
Revised: EWC, we have two gradient components from the current task and the penalization loss.

Original: task-specific model components.
Revised: task-specific model components.

Original: SupSup learns a separate subnetwork for each task to predict a given data by superimposing all supermasks.
Revised: SupSup learns a separate subnetwork for each task to predict a given data by superimposing all supermasks.

Original: It is a novel method to solve catastrophic forgetting with taking advantage of neural networks.
Revised: It is a novel method to solve catastrophic forgetting with taking advantage of neural networks.

Original: SGD methods without expereince replay.
Revised: SGD methods without expereince replay.

Original: stable SGD [20] and MC-SGD [10] show overall higher performance in terms of average accuracy than the proposed algorithm.
Revised: stable SGD [20] and MC-SGD [10] show overall higher performance in terms of average accuracy than the proposed algorithm.

Original: For average forgetting, our method has the lowest value, which means that NCCL prevents catastrophic forgetting successfully with achieving the reasonable performance on the current task.
Revised: For average forgetting, our method has the lowest value, which means that NCCL prevents catastrophic forgetting successfully with achieving the reasonable performance on the current task.

Original: We think that our method is focused on reducing catastrophic forgetting as we defined in the reformulated continual learning problem (12), so our method shows the better performance on average forgetting.
Revised: We think that our method is focused on reducing catastrophic forgetting as we defined in the reformulated continual learning problem (12), so our method shows the better performance on average forgetting.

Original: Otherwise, MC-SGD finds a low-loss paths with mode-connectivity by updating with the proposed regularization loss.
Revised: Otherwise, MC-SGD finds a low-loss paths with mode-connectivity by updating with the proposed regularization loss.

Original: This procedure implies that a continual learning model might find a better local minimum point for the new (current) task than NCCL.
Revised: This procedure implies that a continual learning model might find a better local minimum point for the new (current) task than NCCL.

Original: For non-memory based methods, the theoretical measure to observe forgetting and convergence during training does not exist.
Revised: For non-memory based methods, the theoretical measure to observe forgetting and convergence during training does not exist.

Original: Our theoretical results are the first attempt to analyze the convergence of previous tasks during continual learning procedure.
Revised: Our theoretical results are the first attempt to analyze the convergence of previous tasks during continual learning procedure.

Original: In future work, we can approximate the value of with fisher information for EWC and introduce Bayesian deep learning to analyze the convergence of each subnetworks for each task in the case of SupSup [33].
Revised: In future work, we can approximate the value of with fisher information for EWC and introduce Bayesian deep learning to analyze the convergence of each subnetworks for each task in the case of SupSup [33].

Original: Appendix C. Additional Experimental Results and Implementation Details
Revised: Appendix C. Additional Experimental Results and Implementation Details

Original: We implement the baselines and the proposed method on Tensorflow 1.
Revised: We implement the baselines and the proposed method on Tensorflow 1.

Original: For evaluation, we use an
Revised: For evaluation, we use an

Original: NVIDIA 2080ti GPU along with 3.60 GHz Intel i9-9900K CPU and 64 GB RAM.
Revised: NVIDIA 2080ti GPU along with 3.60 GHz Intel i9-9900K CPU and 64 GB RAM.

Original: C.1. Experimental setup
Revised: C.1. Experimental setup

Original: Datasets.
Revised: Datasets.

Original: We demonstrate the experimental results on standard continual learning benckmarks:
Revised: We demonstrate the experimental results on standard continual learning benckmarks:

Original: Permuted-MNIST [11] is a MNIST [14] based dataset, where each task has a fixed permutation of pixels and transform data points by the permutation to make each task distribution unrelated.
Revised: Permuted-MNIST [11] is a MNIST [14] based dataset, where each task has a fixed permutation of pixels and transform data points by the permutation to make each task distribution unrelated.

Original: SplitMNIST [36] splits MNIST dataset into five tasks.
Revised: SplitMNIST [36] splits MNIST dataset into five tasks.

Original: Each task consists of two classes, for example (1, 7), (3, 4), and has approximately 12K images.
Revised: Each task consists of two classes, for example (1, 7), (3, 4), and has approximately 12K images.

Original: Split-CIFAR10, 100, and MiniImagenet also split versions of CIFAR-10, 100 [13], and MiniImagenet [32] into five tasks and 20 tasks.
Revised: Split-CIFAR10, 100, and MiniImagenet also split versions of CIFAR-10, 100 [13], and MiniImagenet [32] into five tasks and 20 tasks.

Original: Baselines.
Revised: Baselines.

Original: We report the experimental evaluation on the online continual setting which implies a model is trained with a single epoch.
Revised: We report the experimental evaluation on the online continual setting which implies a model is trained with a single epoch.

Original: We compare with the following continual learning baselines.
Revised: We compare with the following continual learning baselines.

Original: Fine-tune is a simple method that a model trains observed data naively without any support, such as replay memory.
Revised: Fine-tune is a simple method that a model trains observed data naively without any support, such as replay memory.

Original: Elastic weight consolidation (EWC) is a regularization based method by Fisher
Revised: Elastic weight consolidation (EWC) is a regularization based method by Fisher

Original: Information [11].
Revised: Information [11].

Original: ER-Reservoir chooses samples to store from a data stream with a probability proportional to the number of observed data points.
Revised: ER-Reservoir chooses samples to store from a data stream with a probability proportional to the number of observed data points.

Original: The replay memory returns a random subset of samples at each iteration for experience replay.
Revised: The replay memory returns a random subset of samples at each iteration for experience replay.

Original: ER-Reservoir [3] shows a powerful performance in continual learning scenario.
Revised: ER-Reservoir [3] shows a powerful performance in continual learning scenario.

Original: GEM and A-GEM [2, 19] use gradient episodic memory to overcome forgetting.
Revised: GEM and A-GEM [2, 19] use gradient episodic memory to overcome forgetting.

Original: The key idea of GEM is gradient projection with quadratic programming and A-GEM simplifies this procedure.
Revised: The key idea of GEM is gradient projection with quadratic programming and A-GEM simplifies this procedure.

Original: We also compare with iCarl, MER, ORTHOG-SUBSPACE [5], stable
Revised: We also compare with iCarl, MER, ORTHOG-SUBSPACE [5], stable

Original: SGD [20], and MC-SGD [21].
Revised: SGD [20], and MC-SGD [21].

Original: C.2. Architecture and Training detail
Revised: C.2. Architecture and Training detail

Original: For fair comparison, we follow the commonly used model architecture and hyperparameters of [5, 16].
Revised: For fair comparison, we follow the commonly used model architecture and hyperparameters of [5, 16].

Original: For Permuted-MNIST and Split-MNIST, we use fully-connected neural networks with two hidden layers of [400 , 400] or [256 , 256] and ReLU activation.
Revised: For Permuted-MNIST and Split-MNIST, we use fully-connected neural networks with two hidden layers of [400 , 400] or [256 , 256] and ReLU activation.

Original: ResNet-18 with the number of filters n f = 64 , 20 [9] is applied for Split CIFAR-10 and 100.
Revised: ResNet-18 with the number of filters n f = 64 , 20 [9] is applied for Split CIFAR-10 and 100.

Original: All experiments conduct a single-pass over the data stream.
Revised: All experiments conduct a single-pass over the data stream.

Original: It is also called 1 epoch or 0.2 epoch (in the case of split tasks).
Revised: It is also called 1 epoch or 0.2 epoch (in the case of split tasks).

Original: We deal both cases with and without the task identifiers in the results of split-tasks to compare fairly with baselines.
Revised: We deal both cases with and without the task identifiers in the results of split-tasks to compare fairly with baselines.

Original: Batch sizes of data stream and memory are both 10.
Revised: Batch sizes of data stream and memory are both 10.

Original: All reported values are the average values ofruns with diffrent seeds, and we also provide standard deviation.
Revised: All reported values are the average values ofruns with diffrent seeds, and we also provide standard deviation.

Original: Other miscellaneous settings are the same as in [5].
Revised: Other miscellaneous settings are the same as in [5].

Original: C.3. Hyperparameter grids
Revised: C.3. Hyperparameter grids

Original: We report the hyper-paramters grid we used in our experiments below.
Revised: We report the hyper-paramters grid we used in our experiments below.

Original: Except for the proposed algorithm, we adopted the hyper-paramters that are reported in the original papers.
Revised: Except for the proposed algorithm, we adopted the hyper-paramters that are reported in the original papers.

Original: We used grid search to find the optimal parameters for each model.
Revised: We used grid search to find the optimal parameters for each model.

Original: • finetune - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • EWC - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] - regularization: [0.1, 1, 10 (MNIST,CIFAR), 100, 1000] • A-GEM - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Ring - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ORTHOG-SUBSPACE - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST), 0.2, 0.4 (CIFAR), 1.0] • MER - learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR), 0.1, 0.3, 1.0] - within batch metalearning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] - current batch learning rate multiplier:
Revised: • finetune - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • EWC - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] - regularization: [0.1, 1, 10 (MNIST,CIFAR), 100, 1000] • A-GEM - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Ring - learning rate: [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ORTHOG-SUBSPACE - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST), 0.2, 0.4 (CIFAR), 1.0] • MER - learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR), 0.1, 0.3, 1.0] - within batch metalearning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] - current batch learning rate multiplier:

Original: [1, 2, 5 (CIFAR), 10 (MNIST)] • iid-offline and iid-online - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Reservoir - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] • NCCL-Ring (default) - learning rate α : [0.003, 0.001(CIFAR), 0.01, 0.03, 0.1, 0.3, 1.0] • NCCL-Reservoir - learning rate α : [0.003(CIFAR), 0.001, 0.01, 0.03, 0.1, 0.3, 1.0]
Revised: [1, 2, 5 (CIFAR), 10 (MNIST)] • iid-offline and iid-online - learning rate [0.003, 0.01, 0.03 (CIFAR), 0.1 (MNIST), 0.3, 1.0] • ER-Reservoir - learning rate: [0.003, 0.01, 0.03, 0.1 (MNIST, CIFAR), 0.3, 1.0] • NCCL-Ring (default) - learning rate α : [0.003, 0.001(CIFAR), 0.01, 0.03, 0.1, 0.3, 1.0] • NCCL-Reservoir - learning rate α : [0.003(CIFAR), 0.001, 0.01, 0.03, 0.1, 0.3, 1.0]

Original: C.4. Hyperparameter Search on β max and Training Time
Revised: C.4. Hyperparameter Search on β max and Training Time

Original: We modify the clipping bound of β H t in Section 3 to resolve the lower performance in terms of average accuracy.
Revised: We modify the clipping bound of β H t in Section 3 to resolve the lower performance in terms of average accuracy.

Original: In Table 4, 1, NCCL+Ring does not have the best average accuracy score, even though it has the lowest value of (cid:80) E [Γ t ] .
Revised: In Table 4, 1, NCCL+Ring does not have the best average accuracy score, even though it has the lowest value of (cid:80) E [Γ t ] .

Original: As we discussed earlier
Revised: As we discussed earlier

Original: [Table]
Revised: [Table]

Original: it is because the convergence rate of C is slower than vanilla ER-Ring with the fixed step sizes.
Revised: it is because the convergence rate of C is slower than vanilla ER-Ring with the fixed step sizes.

Original: Now, we remove the restriction of β H t , min (cid:16) α (1 − δ ) , (1 − αL )Λ Ht L ∥∇ g Jt ( x t ) ∥ 2 (cid:17) for Λ H t > 0 , and instead apply the maximum clipping bound β max to maximize the transfer effect, which occurs if Λ H t > 0 , by getting E [Γ ∗ t ] .
Revised: Now, we remove the restriction of β H t , min (cid:16) α (1 − δ ) , (1 − αL )Λ Ht L ∥∇ g Jt ( x t ) ∥ 2 (cid:17) for Λ H t > 0 , and instead apply the maximum clipping bound β max to maximize the transfer effect, which occurs if Λ H t > 0 , by getting E [Γ ∗ t ] .

Original: In the original version, we force β H t < α to reduce theoretical catastrophic forgetting term completely.
Revised: In the original version, we force β H t < α to reduce theoretical catastrophic forgetting term completely.

Original: However, replacing with β max is helpful in terms of average accuracy as shown in Table 2.
Revised: However, replacing with β max is helpful in terms of average accuracy as shown in Table 2.

Original: It means that β max is a hyperparameter to increase the average accuracy by balancing between forgetting on P and learning on C .
Revised: It means that β max is a hyperparameter to increase the average accuracy by balancing between forgetting on P and learning on C .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C.5. Additional Experiment Results
Revised: C.5. Additional Experiment Results

Original: We add more results with larger sizes of memory, which shows that NCCL outperforms in terms of average accuracy.
Revised: We add more results with larger sizes of memory, which shows that NCCL outperforms in terms of average accuracy.

Original: It means that estimating transfer and interference in terms of Λ H t to alleviate forgetting by the small memory for NCCL is less effective
Revised: It means that estimating transfer and interference in terms of Λ H t to alleviate forgetting by the small memory for NCCL is less effective

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Appendix D. Main Theoretical Results D.1. Assumptions
Revised: Appendix D. Main Theoretical Results D.1. Assumptions

Original: Throughout the paper, we assume L -smoothness and the following statements.
Revised: Throughout the paper, we assume L -smoothness and the following statements.

Original: Assumption 4 f i is L -smooth that there exists a constant L > 0 such that for any x, y ∈ R d ,
Revised: Assumption 4 f i is L -smooth that there exists a constant L > 0 such that for any x, y ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ denotes the Euclidean norm.
Revised: where ∥·∥ denotes the Euclidean norm.

Original: Then the following inequality directly holds that
Revised: Then the following inequality directly holds that

Original: [Equation]
Revised: [Equation]

Original: We derive Equation 7 in Appendix D. Assumption 4 is a well-known and useful statement in nonconvex finite-sum optimization problem [24, 26, 35, 37], and also helps us to describe the convergence of continual learning.
Revised: We derive Equation 7 in Appendix D. Assumption 4 is a well-known and useful statement in nonconvex finite-sum optimization problem [24, 26, 35, 37], and also helps us to describe the convergence of continual learning.

Original: We also assume the supremum of loss gap between an initial point x 0 and a global optimum x ∗ as ∆ f , and the upper bound on the variance of the stochastic gradients as σ f in the following.
Revised: We also assume the supremum of loss gap between an initial point x 0 and a global optimum x ∗ as ∆ f , and the upper bound on the variance of the stochastic gradients as σ f in the following.

Original: [Equation]
Revised: [Equation]

Original: It should be noted that g j ( x ) , ∇ g j ( x ) , which denote the loss and the gradient for a current task, also satisfy all three above assumptions and the following statement.
Revised: It should be noted that g j ( x ) , ∇ g j ( x ) , which denote the loss and the gradient for a current task, also satisfy all three above assumptions and the following statement.

Original: To measure the efficiency of a stochastic gradient algorithm, we define the Incremental Firstorder Oracle (IFO) framework [8].
Revised: To measure the efficiency of a stochastic gradient algorithm, we define the Incremental Firstorder Oracle (IFO) framework [8].

Original: IFO call is defined as a unit of computational cost by taking an index i which gets the pair ( ∇ f i ( x ) , f i ( x )) , and IFO complexity of an algorithm is defined as the summation of IFO calls during optimization.
Revised: IFO call is defined as a unit of computational cost by taking an index i which gets the pair ( ∇ f i ( x ) , f i ( x )) , and IFO complexity of an algorithm is defined as the summation of IFO calls during optimization.

Original: For example, a vanilla stochastic gradient descent (SGD) algorithm requires computational cost as much as the batch size b t at each step, and the
Revised: For example, a vanilla stochastic gradient descent (SGD) algorithm requires computational cost as much as the batch size b t at each step, and the

Original: IFO complexity is the sum of batch sizes (cid:80)
Revised: IFO complexity is the sum of batch sizes (cid:80)

Original: Tt =1 b t .
Revised: Tt =1 b t .

Original: Let T ( ϵ ) be the minimum number of iterations to guarantee ϵ -accurate solutions.
Revised: Let T ( ϵ ) be the minimum number of iterations to guarantee ϵ -accurate solutions.

Original: The average bound of IFO complexity is less than or equal to (cid:80) T ( ϵ ) t =1 b t =
Revised: The average bound of IFO complexity is less than or equal to (cid:80) T ( ϵ ) t =1 b t =

Original: O (1 /ϵ 2 ) [24].
Revised: O (1 /ϵ 2 ) [24].

Original: D.2. Memory-based Nonconvex Continual Learning
Revised: D.2. Memory-based Nonconvex Continual Learning

Original: Unlike conventional smooth nonconvex finite-sum optimization problems where each mini-batch is i.i.d-sampled from the whole dataset P ∪ C , the replay memory based continual learning encounters a non-i.i.d stream of data C with access to a small sized memory M t .
Revised: Unlike conventional smooth nonconvex finite-sum optimization problems where each mini-batch is i.i.d-sampled from the whole dataset P ∪ C , the replay memory based continual learning encounters a non-i.i.d stream of data C with access to a small sized memory M t .

Original: Algorithm 1 provides the pseudocode for memory-based approach with the iterative update rule 3.
Revised: Algorithm 1 provides the pseudocode for memory-based approach with the iterative update rule 3.

Original: Now, we can analyze the convergence on P and C during a learning procedure on an arbitrary data stream from two consecutive sets P and C for continual learning [2, 3, 5].
Revised: Now, we can analyze the convergence on P and C during a learning procedure on an arbitrary data stream from two consecutive sets P and C for continual learning [2, 3, 5].

Original: By limited access to P , the expectation of gradient update E I t ⊂ M t [ ∇ f I t ( x t )] in Equation 3 for f ( x ) is a biased estimate of the gradient ∇ f ( x t ) .
Revised: By limited access to P , the expectation of gradient update E I t ⊂ M t [ ∇ f I t ( x t )] in Equation 3 for f ( x ) is a biased estimate of the gradient ∇ f ( x t ) .

Original: At the timestep t , we have
Revised: At the timestep t , we have

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where e t and e M t denote the error terms, ∇ f I t ( x t ) − ∇ f ( x t ) and the expectation over I t given
Revised: where e t and e M t denote the error terms, ∇ f I t ( x t ) − ∇ f ( x t ) and the expectation over I t given

Original: M t , respectively.
Revised: M t , respectively.

Original: It should be noted that a given replay memory M t with small size at timestep t introduces an inevitable overfitting bias.
Revised: It should be noted that a given replay memory M t with small size at timestep t introduces an inevitable overfitting bias.

Original: For example, there exist two popular memory schemes, episodic memory and ER-reservoir.
Revised: For example, there exist two popular memory schemes, episodic memory and ER-reservoir.

Original: The episodic memory M t = M 0 for all t is uniformly sampled once from a random sequence of P , and
Revised: The episodic memory M t = M 0 for all t is uniformly sampled once from a random sequence of P , and

Original: ER-reservoir iteratively samples the replay memory M t by the selection rule M t ⊂ M t − 1 ∪ J t .
Revised: ER-reservoir iteratively samples the replay memory M t by the selection rule M t ⊂ M t − 1 ∪ J t .

Original: Here, we denote the history of M t as M [0: t ] =
Revised: Here, we denote the history of M t as M [0: t ] =

Original: ( M 0 , · · · , M t ) .
Revised: ( M 0 , · · · , M t ) .

Original: To compute the expectation over all stochasticities of NCCL, we need to derive the expectation of ∇ f M t ( x t ) over the randomness of
Revised: To compute the expectation over all stochasticities of NCCL, we need to derive the expectation of ∇ f M t ( x t ) over the randomness of

Original: M t .
Revised: M t .

Original: We formalize the expectation over all learning trials with the selection randomness as follows.
Revised: We formalize the expectation over all learning trials with the selection randomness as follows.

Original: Lemma 

Revised: Lemma

Original: If M 0 is uniformly sampled from P , then both episodic memory and ER-reservoir satisfies
Revised: If M 0 is uniformly sampled from P , then both episodic memory and ER-reservoir satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Note that taking expectation iteratively with respect to the history M [0: t ] is needed to compute the expected value of gradients for M t .
Revised: Note that taking expectation iteratively with respect to the history M [0: t ] is needed to compute the expected value of gradients for M t .

Original: Surprisingly, taking the expectation of overfitting error over memory selection gets zero.
Revised: Surprisingly, taking the expectation of overfitting error over memory selection gets zero.

Original: However, it does not imply e t = 0 for each learning trial with some
Revised: However, it does not imply e t = 0 for each learning trial with some

Original: M [0: t ] .
Revised: M [0: t ] .

Original: D.3. Reformulated Problem of Continual Learning.
Revised: D.3. Reformulated Problem of Continual Learning.

Original: The previous section showed that the essential terms in continual learning to observe the theoretical convergence rate.
Revised: The previous section showed that the essential terms in continual learning to observe the theoretical convergence rate.

Original: We now reformulate the continual learning problem 2 as follows.
Revised: We now reformulate the continual learning problem 2 as follows.

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: It is noted that the above reformulation presents a theoretically guaranteed continual learning framework for memory-based approaches in nonconvex setting.
Revised: It is noted that the above reformulation presents a theoretically guaranteed continual learning framework for memory-based approaches in nonconvex setting.

Original: D.4. Proposed Adaptive Learning Rates
Revised: D.4. Proposed Adaptive Learning Rates

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we derive the proposed adaptive learning rates for both cases Λ H t ≤ 0 and Λ H t > 0 .
Revised: we derive the proposed adaptive learning rates for both cases Λ H t ≤ 0 and Λ H t > 0 .

Original: D.4.1. Λ H t >
Revised: D.4.1. Λ H t >

Original: Unlike the exisitnig algorithms
Revised: Unlike the exisitnig algorithms

Original: [Equation]
Revised: [Equation]

Original: we can leverage the transfer case Λ H t > 0 with theoretical guarantee.
Revised: we can leverage the transfer case Λ H t > 0 with theoretical guarantee.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we propose the optimal learning rates which tighten the upper bound of stationary of f ( x ) as follows.
Revised: we propose the optimal learning rates which tighten the upper bound of stationary of f ( x ) as follows.

Original: Derivation of optimal Γ ∗ t and β ∗ H t
Revised: Derivation of optimal Γ ∗ t and β ∗ H t

Original: For a fixed learning rate α , we have
Revised: For a fixed learning rate α , we have

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: D.4.2. Λ H t ≤
Revised: D.4.2. Λ H t ≤

Original: We adopt the algorithm of A-GEM partially when Λ H t ≤ 0 .
Revised: We adopt the algorithm of A-GEM partially when Λ H t ≤ 0 .

Original: In this section, we explain how AGEM take advantages in terms of our theoretical analysis.
Revised: In this section, we explain how AGEM take advantages in terms of our theoretical analysis.

Original: A-GEM propose a surrogate of ∇ g J t ( x t ) as the following equation to avoid violating the constraint when Λ H t ≤ 0 [2]:
Revised: A-GEM propose a surrogate of ∇ g J t ( x t ) as the following equation to avoid violating the constraint when Λ H t ≤ 0 [2]:

Original: [Equation]
Revised: [Equation]

Original: Let β be the step size for g ( x ) when the constraint is not violated.
Revised: Let β be the step size for g ( x ) when the constraint is not violated.

Original: Then we can interpret the surrogate as adaptive learning rate α H t , which is either α (1 − ⟨∇ f
Revised: Then we can interpret the surrogate as adaptive learning rate α H t , which is either α (1 − ⟨∇ f

Original: It ( x t ) , ∇ g Jt ( x t ) ⟩ ∥∇ f
Revised: It ( x t ) , ∇ g Jt ( x t ) ⟩ ∥∇ f

Original: It ( x t ) ∥) to cancel out the negative component of ∇ f I t ( x t ) on ∇ g J t ( x t ) or α H t = 0 for Λ H t > 0 .
Revised: It ( x t ) ∥) to cancel out the negative component of ∇ f I t ( x t ) on ∇ g J t ( x t ) or α H t = 0 for Λ H t > 0 .

Original: After applying the surrogate, E [Γ t ] is reduced as shown in the below.
Revised: After applying the surrogate, E [Γ t ] is reduced as shown in the below.

Original: It is noted that A-GEM theoretically violates the constraints of (9) to prevent catastrophic forgetting by letting α H t = 0 .
Revised: It is noted that A-GEM theoretically violates the constraints of (9) to prevent catastrophic forgetting by letting α H t = 0 .

Original: That is to say, A-GEM is an adaptive method without theoretical guarantee.
Revised: That is to say, A-GEM is an adaptive method without theoretical guarantee.

Original: Derivation for A-GEM
Revised: Derivation for A-GEM

Original: Let the surrogate ∇ ˜
Revised: Let the surrogate ∇ ˜

Original: g J t ( x t ) as
Revised: g J t ( x t ) as

Original: [Equation]
Revised: [Equation]

Original: where α H t = α (1 − ⟨∇ f
Revised: where α H t = α (1 − ⟨∇ f

Original: It ( x t ) , ∇ g Jt ( x t ) ⟩ ∥∇ f
Revised: It ( x t ) , ∇ g Jt ( x t ) ⟩ ∥∇ f

Original: It ( x t ) ∥) and β H t = α for Equation 3.
Revised: It ( x t ) ∥) and β H t = α for Equation 3.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we compare the catastrophic forgetting term between the original value with ∇ g J t ( x t ) and the above surrogate.
Revised: we compare the catastrophic forgetting term between the original value with ∇ g J t ( x t ) and the above surrogate.

Original: [Equation]
Revised: [Equation]

Original: Then, we can conclude that E [Γ t ] with the surrogate of A-GEM is smaller than the original E [Γ t ] .
Revised: Then, we can conclude that E [Γ t ] with the surrogate of A-GEM is smaller than the original E [Γ t ] .

Original: D.5. Technical Lemma
Revised: D.5. Technical Lemma

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we provide the proofs of the results for nonconvex continual learning.
Revised: we provide the proofs of the results for nonconvex continual learning.

Original: We first start with the derivation of Equation 7 in Assumption 4.
Revised: We first start with the derivation of Equation 7 in Assumption 4.

Original: Proof [ Derivation of Equation 7 ] Recall that
Revised: Proof [ Derivation of Equation 7 ] Recall that

Original: [Equation]
Revised: [Equation]

Original: Note that f i is differentiable and nonconvex.
Revised: Note that f i is differentiable and nonconvex.

Original: We define a function g ( t ) =
Revised: We define a function g ( t ) =

Original: f i ( y + t ( x − y )) for t ∈ [0 , 1] and an objective function f i .
Revised: f i ( y + t ( x − y )) for t ∈ [0 , 1] and an objective function f i .

Original: By the fundamental theorem of calculus,
Revised: By the fundamental theorem of calculus,

Original: [Equation]
Revised: [Equation]

Original: By the property
Revised: By the property

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using the Cauchy-Schwartz inequality,
Revised: Using the Cauchy-Schwartz inequality,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Since f i satisfies Equation 6, then we have
Revised: Since f i satisfies Equation 6, then we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 Let p =
Revised: Lemma 6 Let p =

Original: [ p 1 , · · · p D ] , q = [ q 1 , · · · , q D ] be two statistically independent random vectors with dimension D .
Revised: [ p 1 , · · · p D ] , q = [ q 1 , · · · , q D ] be two statistically independent random vectors with dimension D .

Original: Then the expectation of the inner product of two random vectors E [ ⟨ p, q ⟩ ] is (cid:80)
Revised: Then the expectation of the inner product of two random vectors E [ ⟨ p, q ⟩ ] is (cid:80)

Original: Dd =1 E [ p d ] E [ q d ] .
Revised: Dd =1 E [ p d ] E [ q d ] .

Original: Proof By the property of expectation,
Revised: Proof By the property of expectation,

Original: [Equation]
Revised: [Equation]

Original: D.6. Proof of Main Results
Revised: D.6. Proof of Main Results

Original: We now show the main results of our work.
Revised: We now show the main results of our work.

Original: We first state the stepwise change of upper bound.
Revised: We first state the stepwise change of upper bound.

Original: Since taking the expectation over all stochasticities of NCCL implies the total expectation, we define the operator of total expectation with respect to 0 ≤ t < T for ease of exposition as follows:
Revised: Since taking the expectation over all stochasticities of NCCL implies the total expectation, we define the operator of total expectation with respect to 0 ≤ t < T for ease of exposition as follows:

Original: [Equation]
Revised: [Equation]

Original: In addition, we denote E T − 1 = E .
Revised: In addition, we denote E T − 1 = E .

Original: Lemma 7 Suppose that Assumption 4 holds and 0 < α H t ≤ 2 L .
Revised: Lemma 7 Suppose that Assumption 4 holds and 0 < α H t ≤ 2 L .

Original: For x t updated by Algorithm 1, we have
Revised: For x t updated by Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof [ Proof of Lemma 5 ] To clarify the issue of E M t [ E I t [ e t | M t ]] = 0 , let us explain the details of constructing replay-memory as follows.
Revised: Proof [ Proof of Lemma 5 ] To clarify the issue of E M t [ E I t [ e t | M t ]] = 0 , let us explain the details of constructing replay-memory as follows.

Original: We have considered episodic memory and reservoir sampling in the paper.
Revised: We have considered episodic memory and reservoir sampling in the paper.

Original: We will first show the case of episodic memory by describing the sampling method for replay memory.
Revised: We will first show the case of episodic memory by describing the sampling method for replay memory.

Original: We can also derive the case of reservoir sampling by simply applying the result of episodic memory.
Revised: We can also derive the case of reservoir sampling by simply applying the result of episodic memory.

Original: Episodic memory (ring buffer).
Revised: Episodic memory (ring buffer).

Original: We divide the entire dataset of continual learning into the previous task P and the current task C on the time step t = 0 .
Revised: We divide the entire dataset of continual learning into the previous task P and the current task C on the time step t = 0 .

Original: For the previous task P , the data stream of P is i.i.d., and its sequence is random on every trial (episode).
Revised: For the previous task P , the data stream of P is i.i.d., and its sequence is random on every trial (episode).

Original: The trial (episode) implies that a continual learning agent learns from an online data stream with two consecutive data sequences of P and C .
Revised: The trial (episode) implies that a continual learning agent learns from an online data stream with two consecutive data sequences of P and C .

Original: Episodic memory takes the last data points of the given memory size m by the First In First Out (FIFO) rule, and holds the entire data points until learning on C is finished.
Revised: Episodic memory takes the last data points of the given memory size m by the First In First Out (FIFO) rule, and holds the entire data points until learning on C is finished.

Original: Then, we note that M t =
Revised: Then, we note that M t =

Original: M 0 for all t ≥ 0 and M 0 is uniformly sampled from the i.i.d.
Revised: M 0 for all t ≥ 0 and M 0 is uniformly sampled from the i.i.d.

Original: sequence of
Revised: sequence of

Original: P .
Revised: P .

Original: By the law of total expectation, we derive E M 0 ⊂ P (cid:2)
Revised: By the law of total expectation, we derive E M 0 ⊂ P (cid:2)

Original: E I t (cid:2) ∇ f I t ( x t )
Revised: E I t (cid:2) ∇ f I t ( x t )

Original: | M 0 (cid:3)(cid:3) for any x t , ∀ t ≥ 0 .
Revised: | M 0 (cid:3)(cid:3) for any x t , ∀ t ≥ 0 .

Original: [Equation]
Revised: [Equation]

Original: It is known that M 0 was uniformly sampled from P on each trial before training on the current task
Revised: It is known that M 0 was uniformly sampled from P on each trial before training on the current task

Original: C .
Revised: C .

Original: Then, we take expectation with respect to every trial that implies the expected value over the memory distribution M 0 .
Revised: Then, we take expectation with respect to every trial that implies the expected value over the memory distribution M 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: for any x t , ∀ t .
Revised: for any x t , ∀ t .

Original: We can consider ∇ f M t ( x t ) as a sample mean of P on every trial for any x t , ∀ t ≥ 0 .
Revised: We can consider ∇ f M t ( x t ) as a sample mean of P on every trial for any x t , ∀ t ≥ 0 .

Original: Although x t is constructed iteratively, the expected value of the sample mean for any x t ,
Revised: Although x t is constructed iteratively, the expected value of the sample mean for any x t ,

Original: E M 0 ⊂ P (cid:2) ∇ f M 0 ( x t ) (cid:3) is also derived as ∇ f ( x t ) .
Revised: E M 0 ⊂ P (cid:2) ∇ f M 0 ( x t ) (cid:3) is also derived as ∇ f ( x t ) .

Original: Reservoir sampling.
Revised: Reservoir sampling.

Original: To clarify the notation for reservoir sampling first, we denote the expectation with respect to the history of replay memory M [0: t ] =
Revised: To clarify the notation for reservoir sampling first, we denote the expectation with respect to the history of replay memory M [0: t ] =

Original: ( M 0 , · · · , M t ) as E M [0: t ] .
Revised: ( M 0 , · · · , M t ) as E M [0: t ] .

Original: This is the revised version of E M t .
Revised: This is the revised version of E M t .

Original: Reservoir sampling is a trickier case than episodic memory, but
Revised: Reservoir sampling is a trickier case than episodic memory, but

Original: E M [0: t ] [ E I t [ e t | M t ]] = 0 still holds.
Revised: E M [0: t ] [ E I t [ e t | M t ]] = 0 still holds.

Original: Suppose that M 0 is full of the data points from P as the episodic memory is sampled and the mini-batch size from C is 1 for simplicity.
Revised: Suppose that M 0 is full of the data points from P as the episodic memory is sampled and the mini-batch size from C is 1 for simplicity.

Original: The reservoir sampling algorithm drops a data point in M t − 1 and replaces the dropped data point with a data point in the current mini-batch from C with probability p = m/n , where m is the memory size and n is the number of visited data points so far.
Revised: The reservoir sampling algorithm drops a data point in M t − 1 and replaces the dropped data point with a data point in the current mini-batch from C with probability p = m/n , where m is the memory size and n is the number of visited data points so far.

Original: The exact pseudo-code for reservoir sampling is described in [1].
Revised: The exact pseudo-code for reservoir sampling is described in [1].

Original: The replacement procedure uniformly chooses the data point which will be dropped.
Revised: The replacement procedure uniformly chooses the data point which will be dropped.

Original: We can also consider the replacement procedure as follows.
Revised: We can also consider the replacement procedure as follows.

Original: The memory M t for P is reduced in size 1 from
Revised: The memory M t for P is reduced in size 1 from

Original: M t − 1 , and the replaced data point d C from C contributes in terms of ∇ g d C ( x t ) if d C is sampled from the replay memory.
Revised: M t − 1 , and the replaced data point d C from C contributes in terms of ∇ g d C ( x t ) if d C is sampled from the replay memory.

Original: Let M t − 1 =
Revised: Let M t − 1 =

Original: [ d 1 , · · · , d | M t − 1 | ] where | · | denotes the cardinality of the memory.
Revised: [ d 1 , · · · , d | M t − 1 | ] where | · | denotes the cardinality of the memory.

Original: The sample mean of M t − 1 is given as
Revised: The sample mean of M t − 1 is given as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the rule of reservoir sampling, we assume that the replacement procedure reduces the memory from M t − 1 to M t with size | M t − 1 | − 1 and the set of remained upcoming data points C t ∈ C from the current data stream for online continual learning is reformulated into C t − 1 ∪ [ d C ] .
Revised: By the rule of reservoir sampling, we assume that the replacement procedure reduces the memory from M t − 1 to M t with size | M t − 1 | − 1 and the set of remained upcoming data points C t ∈ C from the current data stream for online continual learning is reformulated into C t − 1 ∪ [ d C ] .

Original: Then, d C can be resampled from C t − 1 ∪ [ d C ] to be composed of the minibatch of reservoir sampling with the dfferent probability.
Revised: Then, d C can be resampled from C t − 1 ∪ [ d C ] to be composed of the minibatch of reservoir sampling with the dfferent probability.

Original: However, we ignore the probability issue now to focus on the effect of replay-memory on ∇ f .
Revised: However, we ignore the probability issue now to focus on the effect of replay-memory on ∇ f .

Original: Now, we sample M t from M t − 1 , then we get the random vector ∇ f M t ( x t ) as
Revised: Now, we sample M t from M t − 1 , then we get the random vector ∇ f M t ( x t ) as

Original: [Equation]
Revised: [Equation]

Original: where the index i is uniformly sampled from i ∼ [1 , · · · , | M t − 1 | ]
Revised: where the index i is uniformly sampled from i ∼ [1 , · · · , | M t − 1 | ]

Original: [Equation]
Revised: [Equation]

Original: and W ij is the indicator function that W ij is 

Revised: and W ij is the indicator function that W ij is

Original: if i =
Revised: if i =

Original: j else 1.
Revised: j else 1.

Original: The above description implies the dropping rule, and M t can be considered as an uniformly sampled set with size | M t | from M t − 1 .
Revised: The above description implies the dropping rule, and M t can be considered as an uniformly sampled set with size | M t | from M t − 1 .

Original: There could also be M t =
Revised: There could also be M t =

Original: M t − 1 with probability 1 − p = 1 − m/n .
Revised: M t − 1 with probability 1 − p = 1 − m/n .

Original: Then the expectation of ∇ f M t ( x t ) given M t − 1 is derived as
Revised: Then the expectation of ∇ f M t ( x t ) given M t − 1 is derived as

Original: [Equation]
Revised: [Equation]

Original: When we consider the mini-batch sampling
Revised: When we consider the mini-batch sampling

Original: [Equation]
Revised: [Equation]

Original: we can formally reformulate the above equation as
Revised: we can formally reformulate the above equation as

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we apply the above equation recursively.
Revised: we apply the above equation recursively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Similar to episodic memory, M 0 is uniformly sampled from P .
Revised: Similar to episodic memory, M 0 is uniformly sampled from P .

Original: Therefore, we conclude that
Revised: Therefore, we conclude that

Original: [Equation]
Revised: [Equation]

Original: by taking expectation over the history M [0: t ] =
Revised: by taking expectation over the history M [0: t ] =

Original: ( M 1 , M 2 , · · · , M t ) .
Revised: ( M 1 , M 2 , · · · , M t ) .

Original: Note that taking expectation iteratively with respect to the history M [ t ] is needed to compute the expected value of gradients for M t .
Revised: Note that taking expectation iteratively with respect to the history M [ t ] is needed to compute the expected value of gradients for M t .

Original: However, the result E M 0 , ··· ,M t [ E I t [ e t | M t ]]
Revised: However, the result E M 0 , ··· ,M t [ E I t [ e t | M t ]]

Original: = 0 still holds in terms of expectation.
Revised: = 0 still holds in terms of expectation.

Original: Furthermore, we also discuss that the effect of reservoir sampling on the convergence of C .
Revised: Furthermore, we also discuss that the effect of reservoir sampling on the convergence of C .

Original: Unlike we simply update g ( x ) by the stochastic gradient descent on C , the datapoints d ∈ M ∩ C have a little larger sampling probability than other datapoints d C − M ∈ C − M .
Revised: Unlike we simply update g ( x ) by the stochastic gradient descent on C , the datapoints d ∈ M ∩ C have a little larger sampling probability than other datapoints d C − M ∈ C − M .

Original: The expectation of gradient norm on the averaged loss E ∥∇ g ( x t ) ∥ 2 is based on the uniform and equiprobable sampling over C , but the nature of reservoir sampling distort this measure slightly.
Revised: The expectation of gradient norm on the averaged loss E ∥∇ g ( x t ) ∥ 2 is based on the uniform and equiprobable sampling over C , but the nature of reservoir sampling distort this measure slightly.

Original: In this paper, we focus on the convergence of the previous task C while training on the current task C with several existing memory-based methods.
Revised: In this paper, we focus on the convergence of the previous task C while training on the current task C with several existing memory-based methods.

Original: Therefore, analyzing the convergence of reservoir sampling method will be a future work.
Revised: Therefore, analyzing the convergence of reservoir sampling method will be a future work.

Original: Proof [ Proof of Lemma 7 ] We analyze the convergence of nonconvex continual learning with replay memory here.
Revised: Proof [ Proof of Lemma 7 ] We analyze the convergence of nonconvex continual learning with replay memory here.

Original: Recall that the gradient update is the following
Revised: Recall that the gradient update is the following

Original: [Equation]
Revised: [Equation]

Original: for all t ∈ { 1 , 2 , · · · , T } .
Revised: for all t ∈ { 1 , 2 , · · · , T } .

Original: Let e t = ∇ f I t ( x t ) − ∇ f ( x t ) .
Revised: Let e t = ∇ f I t ( x t ) − ∇ f ( x t ) .

Original: Since we assume that f, g is L -smooth, we have the following inequality by applying Equation 7:
Revised: Since we assume that f, g is L -smooth, we have the following inequality by applying Equation 7:

Original: [Equation]
Revised: [Equation]

Original: To show the proposed theoretical convergence analysis of nonconvex continual learning
Revised: To show the proposed theoretical convergence analysis of nonconvex continual learning

Original: [Equation]
Revised: [Equation]

Original: we define the catastrophic forgetting term Γ t and the overfitting term B t as follows:
Revised: we define the catastrophic forgetting term Γ t and the overfitting term B t as follows:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we can rewrite Equation 20 as
Revised: we can rewrite Equation 20 as

Original: [Equation]
Revised: [Equation]

Original: We first note that B t is dependent of the error term e t with the batch I t .
Revised: We first note that B t is dependent of the error term e t with the batch I t .

Original: In the continual learning step, an training agent cannot access ∇ f ( x t ) , then we cannot get the exact value of e t .
Revised: In the continual learning step, an training agent cannot access ∇ f ( x t ) , then we cannot get the exact value of e t .

Original: Furthermore, Γ t is dependent of the gradients ∇ f I t ( x t ) , ∇ g I t ( x t ) and the learning rates α H t , β H t .
Revised: Furthermore, Γ t is dependent of the gradients ∇ f I t ( x t ) , ∇ g I t ( x t ) and the learning rates α H t , β H t .

Original: Taking expectations with respect to I t on both sides given J t , we have
Revised: Taking expectations with respect to I t on both sides given J t , we have

Original: [Equation]
Revised: [Equation]

Original: Now, taking expectations over the whole stochasticity we obtain
Revised: Now, taking expectations over the whole stochasticity we obtain

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms and assume that
Revised: Rearranging the terms and assume that

Original: [Equation]
Revised: [Equation]

Original: 1 − Lα Ht / 2 > 0 , we have
Revised: 1 − Lα Ht / 2 > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof [ Proof of Theorem 1 ] Suppose that the learning rate α H t is a constant α = c/ √ T , for c > 0 , 1 − L 2 α = 1 A > 0 .
Revised: Proof [ Proof of Theorem 1 ] Suppose that the learning rate α H t is a constant α = c/ √ T , for c > 0 , 1 − L 2 α = 1 A > 0 .

Original: Then, by summing Equation 14 from t = 0 to T −
Revised: Then, by summing Equation 14 from t = 0 to T −

Original: [Equation]
Revised: [Equation]

Original: we have min t
Revised: we have min t

Original: [Equation]
Revised: [Equation]

Original: We note that a batch I t is sampled from a memory M t ⊂ M which is a random vector whose element is a datapoint d ∈ P ∪ C .
Revised: We note that a batch I t is sampled from a memory M t ⊂ M which is a random vector whose element is a datapoint d ∈ P ∪ C .

Original: Then, taking expectation over I t ⊂ M t ⊂ P ∪ C implies that
Revised: Then, taking expectation over I t ⊂ M t ⊂ P ∪ C implies that

Original: E [ B t ] = 0 .
Revised: E [ B t ] = 0 .

Original: Therefore, we get the minimum of expected square of the norm of gradients
Revised: Therefore, we get the minimum of expected square of the norm of gradients

Original: [Equation]
Revised: [Equation]

Original: Lemma 8 Suppose that I t ∩ J t = ∅ , Taking expectation over I t ⊂ M t and J t ⊂ C , we have
Revised: Lemma 8 Suppose that I t ∩ J t = ∅ , Taking expectation over I t ⊂ M t and J t ⊂ C , we have

Original: [Equation]
Revised: [Equation]

Original: where ∆ h | M ∪ C and σ h | M ∪ C is the version of loss gap and the variance for h on M ∪ C , respectively.
Revised: where ∆ h | M ∪ C and σ h | M ∪ C is the version of loss gap and the variance for h on M ∪ C , respectively.

Original: Proof [ Proof of Lemma 8 ] To simplify the proof, we assume that learning rates α H t , β H t are a same fixed value β = c ′ / √ T .
Revised: Proof [ Proof of Lemma 8 ] To simplify the proof, we assume that learning rates α H t , β H t are a same fixed value β = c ′ / √ T .

Original: The assumption is reasonable, because it is observed that the RHS of Equation 14 is not perturbed drastically by small learning rates in 0 <
Revised: The assumption is reasonable, because it is observed that the RHS of Equation 14 is not perturbed drastically by small learning rates in 0 <

Original: α H t , β H t ≤ 2 /L ≪ 1 .
Revised: α H t , β H t ≤ 2 /L ≪ 1 .

Original: Let us denote the union of M t over time 0 ≤ t ≤ T − 1 as M =
Revised: Let us denote the union of M t over time 0 ≤ t ≤ T − 1 as M =

Original: (cid:83) t M t .
Revised: (cid:83) t M t .

Original: By the assumption
Revised: By the assumption

Original: [Equation]
Revised: [Equation]

Original: it is equivalent to update on M ∪ C .
Revised: it is equivalent to update on M ∪ C .

Original: Then, the non-convex finite sum optimization is given as
Revised: Then, the non-convex finite sum optimization is given as

Original: [Equation]
Revised: [Equation]

Original: where | M | is the number of elements in M .
Revised: where | M | is the number of elements in M .

Original: This problem can be solved by a simple SGD algorithm [25].
Revised: This problem can be solved by a simple SGD algorithm [25].

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 9 For any C ⊂ D ⊂ M ∪ C , define ω 2 h | D as
Revised: Lemma 9 For any C ⊂ D ⊂ M ∪ C , define ω 2 h | D as

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Proof [ Proof of Lemma 9 ] We arrive at the following result by Jensen’s inequality
Revised: Proof [ Proof of Lemma 9 ] We arrive at the following result by Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: By the triangular inequality
Revised: By the triangular inequality

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: For continual learning, the model x 0 reaches to an ϵ -stationary point of f ( x ) when we have finished to learn P and start to learn C .
Revised: For continual learning, the model x 0 reaches to an ϵ -stationary point of f ( x ) when we have finished to learn P and start to learn C .

Original: Now, we discuss the frequency of transfer and interference during continual learning before showing Lemma 2.
Revised: Now, we discuss the frequency of transfer and interference during continual learning before showing Lemma 2.

Original: It is well known that the frequencies between interference and transfer have similar values (the frequency of constraint violation is approximately 0.5 for AGEM) as shown in Appendix D of [2].
Revised: It is well known that the frequencies between interference and transfer have similar values (the frequency of constraint violation is approximately 0.5 for AGEM) as shown in Appendix D of [2].

Original: Even if memory-based continual learning has a small memory buffer which contains a subset of P , random sampling from the buffer allows to have similar frequencies between interference and transfer.
Revised: Even if memory-based continual learning has a small memory buffer which contains a subset of P , random sampling from the buffer allows to have similar frequencies between interference and transfer.

Original: In this paper, we consider two cases for the upper bound of E [Γ t ] , the moderate case and the worst case.
Revised: In this paper, we consider two cases for the upper bound of E [Γ t ] , the moderate case and the worst case.

Original: For the moderate case , which covers most continual learning scenarios, we assume that the inner product term ⟨∇
Revised: For the moderate case , which covers most continual learning scenarios, we assume that the inner product term ⟨∇

Original: f I t ( x t ) , ∇ g J t ( x t ) ⟩ has the same probabilities of being positive (transfer) and negative (interference).
Revised: f I t ( x t ) , ∇ g J t ( x t ) ⟩ has the same probabilities of being positive (transfer) and negative (interference).

Original: Then, we can approximate E [ ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ ] ≈ 0 over all randomness.
Revised: Then, we can approximate E [ ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ ] ≈ 0 over all randomness.

Original: For the worst case , we assume that all ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ has negative values.
Revised: For the worst case , we assume that all ⟨∇ f I t ( x t ) , ∇ g J t ( x t ) ⟩ has negative values.

Original: Proof [ Proof of Lemma 2 ] For the moderate case, we derive the rough upper bound of E [Γ t ] :
Revised: Proof [ Proof of Lemma 2 ] For the moderate case, we derive the rough upper bound of E [Γ t ] :

Original: [Equation]
Revised: [Equation]

Original: By plugging Lemma 9 into E [Γ t ] , we obtain that
Revised: By plugging Lemma 9 into E [Γ t ] , we obtain that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as
Revised: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we consider the randomness of memory choice.
Revised: we consider the randomness of memory choice.

Original: Let D ∗ be as follows:
Revised: Let D ∗ be as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, we obtain the following inequality,
Revised: Then, we obtain the following inequality,

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above equation, we get
Revised: Rearranging the above equation, we get

Original: [Equation]
Revised: [Equation]

Original: For the moderate case , we provide the derivations of the convergence rate for two cases of β as follows.
Revised: For the moderate case , we provide the derivations of the convergence rate for two cases of β as follows.

Original: When β < α = c/ √ T , the upper bound always satisfies
Revised: When β < α = c/ √ T , the upper bound always satisfies

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: For β ≥ α = c/ √ T , we cannot derive a tighter bound, so we still have
Revised: For β ≥ α = c/ √ T , we cannot derive a tighter bound, so we still have

Original: T − 1 (cid:88) t =
Revised: T − 1 (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: For the worst case , we assume that there exists a constant c f,g which satisfies c f,g ∥∇ g J t ( x t ) ∥ ≥ ∥∇ f I t ( x t ) ∥ .
Revised: For the worst case , we assume that there exists a constant c f,g which satisfies c f,g ∥∇ g J t ( x t ) ∥ ≥ ∥∇ f I t ( x t ) ∥ .

Original: [Equation]
Revised: [Equation]

Original: By plugging Lemma 9 into E [Γ t ] , we obtain that
Revised: By plugging Lemma 9 into E [Γ t ] , we obtain that

Original: [Equation]
Revised: [Equation]

Original: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as
Revised: We use the technique for summing up in the proof of Theorem 1, then the cumulative sum of catastrophic forgetting term is derived as

Original: [Equation]
Revised: [Equation]

Original: For the worst case
Revised: For the worst case

Original: [Equation]
Revised: [Equation]

Original: we provide the derivations of the convergence rate for two cases of β as follows.
Revised: we provide the derivations of the convergence rate for two cases of β as follows.

Original: When β < α = c/ √ T , the upper bound always satisfies
Revised: When β < α = c/ √ T , the upper bound always satisfies

Original: [Equation]
Revised: [Equation]

Original: For β ≥ α = c/ √ T , we cannot derive a tighter bound, so we still have
Revised: For β ≥ α = c/ √ T , we cannot derive a tighter bound, so we still have

Original: T − 1 (cid:88) t =
Revised: T − 1 (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: Even if we consider the worst case, we still have O (1) for the cumulative forgetting E [Γ t ] when β < α .
Revised: Even if we consider the worst case, we still have O (1) for the cumulative forgetting E [Γ t ] when β < α .

Original: This implies that we have the theoretical condition for control the forgetting on f ( x ) while evolving on C .
Revised: This implies that we have the theoretical condition for control the forgetting on f ( x ) while evolving on C .

Original: In the main text, we only discuss the moderate case to emphasize f ( x ) can be converged by the effect of transfer during continual learning, but we have also considered the worst case can be well treated by our theoretical condition by keeping the convergence of f ( x ) over time as follows.
Revised: In the main text, we only discuss the moderate case to emphasize f ( x ) can be converged by the effect of transfer during continual learning, but we have also considered the worst case can be well treated by our theoretical condition by keeping the convergence of f ( x ) over time as follows.

Original: Corollary 10 Let β H t < α = c √ T for all t .
Revised: Corollary 10 Let β H t < α = c √ T for all t .

Original: Then we have the convergence rate
Revised: Then we have the convergence rate

Original: [Equation]
Revised: [Equation]

Original: Otherwise, f ( x ) is not guaranteed to converge when β ≥ α and might diverge at the rate O ( √ T ) .
Revised: Otherwise, f ( x ) is not guaranteed to converge when β ≥ α and might diverge at the rate O ( √ T ) .

Original: Proof [ Proof of Corollary 10 ]
Revised: Proof [ Proof of Corollary 10 ]

Original: By Lemma 2, we have
Revised: By Lemma 2, we have

Original: [Equation]
Revised: [Equation]

Original: for β < α for the moderate case .
Revised: for β < α for the moderate case .

Original: Then, we can apply the result into RHS of Equation 4 in Theorem 1 as follows.
Revised: Then, we can apply the result into RHS of Equation 4 in Theorem 1 as follows.

Original: [Equation]
Revised: [Equation]

Original: In addition, we have the convergence rate of f ( x ) for the worst case as follows:
Revised: In addition, we have the convergence rate of f ( x ) for the worst case as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies that f ( x ) can keep the convergence while evolving on C .
Revised: which implies that f ( x ) can keep the convergence while evolving on C .

Original: Proof [ Proof of Corollary 3 ] To formulate the IFO calls, Recall that T ( ϵ )
Revised: Proof [ Proof of Corollary 3 ] To formulate the IFO calls, Recall that T ( ϵ )

Original: [Equation]
Revised: [Equation]

Original: A single IFO call is invested in calculating each step, and we now compute IFO calls to reach an ϵ -accurate solution.
Revised: A single IFO call is invested in calculating each step, and we now compute IFO calls to reach an ϵ -accurate solution.

Original: [Equation]
Revised: [Equation]

Original: When β < α , we get
Revised: When β < α , we get

Original: IFO calls =
Revised: IFO calls =

Original: O (cid:18)
Revised: O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: Otherwise, when β ≥ α , we cannot guarantee the upper bound of stationary decreases over time.
Revised: Otherwise, when β ≥ α , we cannot guarantee the upper bound of stationary decreases over time.

Original: Then, we cannot compute IFO calls for this case.
Revised: Then, we cannot compute IFO calls for this case.

Original: D.7. Remarks
Revised: D.7. Remarks

Original: Surprisingly, we observe E t [ B t ] = 0 by Lemma 5.
Revised: Surprisingly, we observe E t [ B t ] = 0 by Lemma 5.

Original: It should be also noted that the individual trial with a randomly given M 0 cannot cancel the effect of B t .
Revised: It should be also noted that the individual trial with a randomly given M 0 cannot cancel the effect of B t .

Original: We discuss more details of overfitting to memory in Appendix E.
Revised: We discuss more details of overfitting to memory in Appendix E.

Original: The convergence rate with respect to the marginalization on M 0 in Corollary 10 exactly match the usual nonconvex SGD rates.
Revised: The convergence rate with respect to the marginalization on M 0 in Corollary 10 exactly match the usual nonconvex SGD rates.

Original: As empirically shown in stable A-GEM and stable ER-Reservoir [20], the condition of β H t < α theoretically implies that decaying step size is a key solution to continual learning considering we can pick any arbitrary observation points.
Revised: As empirically shown in stable A-GEM and stable ER-Reservoir [20], the condition of β H t < α theoretically implies that decaying step size is a key solution to continual learning considering we can pick any arbitrary observation points.

Original: In addition, it should be also noted that the selection rules for M 0 with various memory schemes are important to perturb the convergence rate based on the mean value in Equation 56 for each trial.
Revised: In addition, it should be also noted that the selection rules for M 0 with various memory schemes are important to perturb the convergence rate based on the mean value in Equation 56 for each trial.

Original: This is why memory schemes matters in continual learning in terms of variance.
Revised: This is why memory schemes matters in continual learning in terms of variance.

Original: Please see more details in Appendix E.
Revised: Please see more details in Appendix E.

Original: Appendix E. Overfitting to replay Memory
Revised: Appendix E. Overfitting to replay Memory

Original: In Lemma 7, we show the expectation of stepwise change of upper bound.
Revised: In Lemma 7, we show the expectation of stepwise change of upper bound.

Original: Now, we discuss the distribution of the upper bound by analyzing the random variable B t .
Revised: Now, we discuss the distribution of the upper bound by analyzing the random variable B t .

Original: As B t is computed by getting
Revised: As B t is computed by getting

Original: [Equation]
Revised: [Equation]

Original: The purpose of our convergence analysis is to compute the upper bound of Equation 14, then we compute the upper bound of B t .
Revised: The purpose of our convergence analysis is to compute the upper bound of Equation 14, then we compute the upper bound of B t .

Original: [Equation]
Revised: [Equation]

Original: It is noted that the upper bound is related to the distribution of the norm of e t .
Revised: It is noted that the upper bound is related to the distribution of the norm of e t .

Original: We have already know that E [ e t ] = 0 , so we consider its variance, Var ( ∥ e t ∥ ) in this section.
Revised: We have already know that E [ e t ] = 0 , so we consider its variance, Var ( ∥ e t ∥ ) in this section.

Original: Let us denote the number of data points of P in a memory M 0 as m P .
Revised: Let us denote the number of data points of P in a memory M 0 as m P .

Original: We assume that M 0 is uniformly sampled from P .
Revised: We assume that M 0 is uniformly sampled from P .

Original: Then the sample variance, Var ( ∥ e t ∥ ) is computed as
Revised: Then the sample variance, Var ( ∥ e t ∥ ) is computed as

Original: [Equation]
Revised: [Equation]

Original: by the similar derivation with Equation 27.
Revised: by the similar derivation with Equation 27.

Original: The above result directly can be applied to the variance of B t .
Revised: The above result directly can be applied to the variance of B t .

Original: This implies m t is a key feature which has an effect on the convergence rate.
Revised: This implies m t is a key feature which has an effect on the convergence rate.

Original: It is noted that the larger m P has the smaller variance by applying schemes, such as larger memory.
Revised: It is noted that the larger m P has the smaller variance by applying schemes, such as larger memory.

Original: In addition, the distributions of e t and ∇ f I t ( x t ) are different with various memory schemes.
Revised: In addition, the distributions of e t and ∇ f I t ( x t ) are different with various memory schemes.

Original: Therefore, we can observe that memory schemes differ the performance even if we apply same step sizes.
Revised: Therefore, we can observe that memory schemes differ the performance even if we apply same step sizes.