Original: I MPROVING
Revised: I MPROVING

Original: I NFORMATION
Revised: I NFORMATION

Original: R ETENTION
Revised: R ETENTION

Original: IN
Revised: IN

Original: L ARGE
Revised: L ARGE

Original: S CALE O NLINE C ONTINUAL 
Revised: S CALE O NLINE C ONTINUAL L

Original: L EARNING
Revised: EARNING

Original: Anonymous authors
Revised: Zhipeng Cai Intel Labs

Original: Paper under double-blind review
Revised: Vladlen Koltun Apple Ozan Sener Apple

Original: A BSTRACT
Revised: A BSTRACT

Original: Given a stream of data sampled from non-stationary distributions, online continual learning (OCL) aims to adapt efficiently to new data while retaining existing knowledge.
Revised: Given a stream of data sampled from non-stationary distributions, online continual learning (OCL) aims to adapt efficiently to new data while retaining existing knowledge.

Original: The typical approach to address information retention (the ability to retain previous knowledge) is keeping a replay buffer of a fixed size and computing gradients using a mixture of new data and the replay buffer.
Revised: The typical approach to address information retention (the ability to retain previous knowledge) is keeping a replay buffer of a fixed size and computing gradients using a mixture of new data and the replay buffer.

Original: Surprisingly, the recent work (Cai et al., 2021) suggests that information retention remains a problem in large scale OCL even when the replay buffer is unlimited, i.e. , the gradients are computed using all past data.
Revised: Surprisingly, the recent work (Cai et al., 2021) suggests that information retention remains a problem in large scale OCL even when the replay buffer is unlimited, i.e. , the gradients are computed using all past data.

Original: This paper focuses on this peculiarity to understand and address information retention.
Revised: This paper focuses on this peculiarity to understand and address information retention.

Original: To pinpoint the source of this problem, we theoretically show that, given limited computation budgets at each time step, even without strict storage limit, naively applying SGD with constant or constantly decreasing learning rates fails to optimize information retention in the long term.
Revised: To pinpoint the source of this problem, we theoretically show that, given limited computation budgets at each time step, even without strict storage limit, naively applying SGD with constant or constantly decreasing learning rates fails to optimize information retention in the long term.

Original: We propose using a moving average family of methods to improve optimization for non-stationary objectives.
Revised: We propose using a moving average family of methods to improve optimization for non-stationary objectives.

Original: Specifically, we design an adaptive moving average (AMA) optimizer and a moving-average-based learning rate schedule (MALR).
Revised: Specifically, we design an adaptive moving average (AMA) optimizer and a moving-average-based learning rate schedule (MALR).

Original: We demonstrate the effectiveness of AMA+MALR on large-scale benchmarks, including Continual Localization (CLOC), Google Landmarks, and ImageNet.
Revised: We demonstrate the effectiveness of AMA+MALR on large-scale benchmarks, including Continual Localization (CLOC), Google Landmarks, and ImageNet.

Original: Code will be released upon publication.
Revised: Code will be released upon publication.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Supervised learning commonly assumes that the data is independent and identically distributed (iid.).
Revised: Supervised learning commonly assumes that the data is independent and identically distributed (iid.).

Original: This assumption is violated in practice when the data comes from a non-stationary distribution that evolves over time.
Revised: This assumption is violated in practice when the data comes from a non-stationary distribution that evolves over time.

Original: Continual learning aims to solve this problem by designing algorithms that efficiently learn and retain knowledge over time from a data stream.
Revised: Continual learning aims to solve this problem by designing algorithms that efficiently learn and retain knowledge over time from a data stream.

Original: Continual learning can be classified into online and offline.
Revised: Continual learning can be classified into online and offline.

Original: The offline setting (Li & Hoiem, 2017) mainly limits the storage: only a fixed amount of training data can be stored at each time step.
Revised: The offline setting (Li & Hoiem, 2017) mainly limits the storage: only a fixed amount of training data can be stored at each time step.

Original: The computation is not limited in offline continual learning: the model can be trained from scratch until convergence at each step.
Revised: The computation is not limited in offline continual learning: the model can be trained from scratch until convergence at each step.

Original: In contrast, the online setting only allows a limited amount of storage and computation at each time step.
Revised: In contrast, the online setting only allows a limited amount of storage and computation at each time step.

Original: A number of metrics can be used to evaluate a continual learner.
Revised: A number of metrics can be used to evaluate a continual learner.

Original: If the model is directly evaluated on the incoming data, the objective is learning efficacy ; this evaluates the ability to efficiently adapt to new data.
Revised: If the model is directly evaluated on the incoming data, the objective is learning efficacy ; this evaluates the ability to efficiently adapt to new data.

Original: If the model is evaluated on historical data, the objective is information retention ; this evaluates the ability to retain existing knowledge.
Revised: If the model is evaluated on historical data, the objective is information retention ; this evaluates the ability to retain existing knowledge.

Original: These two objectives are in conflict, and their trade-off is known as the plasticity-stability dilemma (McCloskey & Cohen, 1989).
Revised: These two objectives are in conflict, and their trade-off is known as the plasticity-stability dilemma (McCloskey & Cohen, 1989).

Original: Following recent counterintuitive results, we single out information retention in this work.
Revised: Following recent counterintuitive results, we single out information retention in this work.

Original: A common assumption in the continual learning literature is that information retention is only a problem due to the storage constraint.
Revised: A common assumption in the continual learning literature is that information retention is only a problem due to the storage constraint.

Original: Take replay-buffer-based methods as an exemplar.
Revised: Take replay-buffer-based methods as an exemplar.

Original: It is tacitly understood that since they cannot store the entire history, they forget past knowledge which is not stored.
Revised: It is tacitly understood that since they cannot store the entire history, they forget past knowledge which is not stored.

Original: However, this intuition is challenged by recent empirical results.
Revised: However, this intuition is challenged by recent empirical results.

Original: For example, Cai et al.
Revised: For example, Cai et al.

Original: (2021) show that the information retention problem persists even when past data is stored in its entirety.
Revised: (2021) show that the information retention problem persists even when past data is stored in its entirety.

Original: We argue that, at least in part, the culprit for information loss is optimization.
Revised: We argue that, at least in part, the culprit for information loss is optimization.

Original: Direct application of SGD to a continual data stream is problematic.
Revised: Direct application of SGD to a continual data stream is problematic.

Original: Informally, consider the learning rate ( i.e. , step size).
Revised: Informally, consider the learning rate ( i.e. , step size).

Original: It needs to decrease to zero over time to guarantee convergence (Ghadimi &
Revised: It needs to decrease to zero over time to guarantee convergence (Ghadimi &

Original: Lan, 2013).
Revised: Lan, 2013).

Original: However, this cannot be applied to a continual and non-iid.
Revised: However, this cannot be applied to a continual and non-iid.

Original: stream since infinitesimal learning rates would simply ignore the new information and fail to adapt, resulting in underfitting.
Revised: stream since infinitesimal learning rates would simply ignore the new information and fail to adapt, resulting in underfitting.

Original: This underfitting would worsen over time as the distribution continues to shift.
Revised: This underfitting would worsen over time as the distribution continues to shift.

Original: We formalize this problem and further show that there is no straightforward method to control this trade-off, and this issue holds even when common adaptive learning rate heuristics are applied.
Revised: We formalize this problem and further show that there is no straightforward method to control this trade-off, and this issue holds even when common adaptive learning rate heuristics are applied.

Original: Orthogonal to continual learning, one recently proposed remedy to guarantee SGD convergence with high learning rates is using the moving average of SGD iterates (Mandt et al., 2016; Tarvainen &
Revised: Orthogonal to continual learning, one recently proposed remedy to guarantee SGD convergence with high learning rates is using the moving average of SGD iterates (Mandt et al., 2016; Tarvainen &

Original: Valpola, 2017).
Revised: Valpola, 2017).

Original: Informally, SGD with large learning rates bounces around the optimum.
Revised: Informally, SGD with large learning rates bounces around the optimum.

Original: Averaging its trajectory dampens the bouncing and tracks the optimum better (Mandt et al., 2016).
Revised: Averaging its trajectory dampens the bouncing and tracks the optimum better (Mandt et al., 2016).

Original: We apply these ideas to OCL for the first time to improve information retention.
Revised: We apply these ideas to OCL for the first time to improve information retention.

Original: To summarize, we theoretically analyze the behavior of SGD for OCL.
Revised: To summarize, we theoretically analyze the behavior of SGD for OCL.

Original: Following this analysis, we propose a moving average strategy to optimize information retention.
Revised: Following this analysis, we propose a moving average strategy to optimize information retention.

Original: Our method uses SGD with large learning rates to adapt to non-stationarity, and utilizes the average of SGD iterates for better convergence.
Revised: Our method uses SGD with large learning rates to adapt to non-stationarity, and utilizes the average of SGD iterates for better convergence.

Original: We propose an adaptive moving average (AMA) algorithm to control the moving average weight over time.
Revised: We propose an adaptive moving average (AMA) algorithm to control the moving average weight over time.

Original: Based on the statistics of the SGD and AMA models, we further propose a moving-average-based learning rate schedule (MALR) to better control the learning rate.
Revised: Based on the statistics of the SGD and AMA models, we further propose a moving-average-based learning rate schedule (MALR) to better control the learning rate.

Original: Experiments on Continual Localization (CLOC) (Cai et al., 2021), Google Landmarks (Weyand et al., 2020), and
Revised: Experiments on Continual Localization (CLOC) (Cai et al., 2021), Google Landmarks (Weyand et al., 2020), and

Original: ImageNet (Deng et al., 2009) demonstrate superior information retention and long-term transfer for large-scale OCL.
Revised: ImageNet (Deng et al., 2009) demonstrate superior information retention and long-term transfer for large-scale OCL.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Optimization in OCL.
Revised: Optimization in OCL.

Original: OCL methods typically focus on improving learning efficacy.
Revised: OCL methods typically focus on improving learning efficacy.

Original: Cai et al.
Revised: Cai et al.

Original: (2021) proposed several strategies, including adaptive learning rates, adaptive replay buffer sizes, and small batch sizes.
Revised: (2021) proposed several strategies, including adaptive learning rates, adaptive replay buffer sizes, and small batch sizes.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) proposed a new optimizer, ConGrad, which, at each time step, adaptively controls the number of online gradient descent steps (Hazan, 2019) to balance generalization and training loss reduction.
Revised: (2020) proposed a new optimizer, ConGrad, which, at each time step, adaptively controls the number of online gradient descent steps (Hazan, 2019) to balance generalization and training loss reduction.

Original: Our work instead focuses on information retention and proposes a new optimizer and learning rate schedule that trade off learning efficacy to improve long-term transfer.
Revised: Our work instead focuses on information retention and proposes a new optimizer and learning rate schedule that trade off learning efficacy to improve long-term transfer.

Original: In terms of replay buffer strategies, mixed replay (Chaudhry et al., 2019), originating from offline continual learning, forms a minibatch by sampling half of the data from the online stream and the other half from the history.
Revised: In terms of replay buffer strategies, mixed replay (Chaudhry et al., 2019), originating from offline continual learning, forms a minibatch by sampling half of the data from the online stream and the other half from the history.

Original: It has been applied in OCL to optimize learning efficacy (Cai et al., 2021).
Revised: It has been applied in OCL to optimize learning efficacy (Cai et al., 2021).

Original: Our work uses pure replay instead to optimize information retention, where a minibatch is formed by sampling uniformly from all history.
Revised: Our work uses pure replay instead to optimize information retention, where a minibatch is formed by sampling uniformly from all history.

Original: Continual learning algorithms.
Revised: Continual learning algorithms.

Original: We focus on the optimization aspect of OCL.
Revised: We focus on the optimization aspect of OCL.

Original: Other aspects, such as the data integration (Aljundi et al., 2019b) and the sampling procedure of the replay buffer (Aljundi et al., 2019a; Chrysakis & Moens, 2020), are complementary and orthogonal to our study.
Revised: Other aspects, such as the data integration (Aljundi et al., 2019b) and the sampling procedure of the replay buffer (Aljundi et al., 2019a; Chrysakis & Moens, 2020), are complementary and orthogonal to our study.

Original: These aspects are critical for a successful OCL strategy and can potentially be used in conjuction with our optimizers.
Revised: These aspects are critical for a successful OCL strategy and can potentially be used in conjuction with our optimizers.

Original: Offline continual learning (Li & Hoiem, 2017; Kirkpatrick et al., 2017) aims to improve information retention with limited storage.
Revised: Offline continual learning (Li & Hoiem, 2017; Kirkpatrick et al., 2017) aims to improve information retention with limited storage.

Original: Unlike the online setting, SGD works in this case since the model can be retrained until convergence at each time step.
Revised: Unlike the online setting, SGD works in this case since the model can be retrained until convergence at each time step.

Original: We refer the readers to Delange et al.
Revised: We refer the readers to Delange et al.

Original: (2021) for a detailed survey of offline continual learning algorithms.
Revised: (2021) for a detailed survey of offline continual learning algorithms.

Original: Moving average in optimization.
Revised: Moving average in optimization.

Original: We propose a new moving-average-based optimizer for OCL.
Revised: We propose a new moving-average-based optimizer for OCL.

Original: Although we are the first to apply this idea to OCL, moving average optimizers have been widely utilized for convex (Ruppert, 1988; Polyak & Juditsky, 1992) and non-convex optimization (Izmailov et al., 2018;
Revised: Although we are the first to apply this idea to OCL, moving average optimizers have been widely utilized for convex (Ruppert, 1988; Polyak & Juditsky, 1992) and non-convex optimization (Izmailov et al., 2018;

Original: Maddox et al., 2019; He et al., 2020).
Revised: Maddox et al., 2019; He et al., 2020).

Original: Beyond supervised learning (Izmailov et al., 2018;
Revised: Beyond supervised learning (Izmailov et al., 2018;

Original: Maddox et al., 2019), the moving average model has also been used as a teacher of the SGD model in semi-supervised (Tarvainen & Valpola, 2017) and self-supervised learning (He et al., 2020).
Revised: Maddox et al., 2019), the moving average model has also been used as a teacher of the SGD model in semi-supervised (Tarvainen & Valpola, 2017) and self-supervised learning (He et al., 2020).

Original: The moving average of stochastic gradients (rather than model weights) has also been widely used in
Revised: The moving average of stochastic gradients (rather than model weights) has also been widely used in

Original: ADAM-based optimizers (Kingma & Ba, 2014).
Revised: ADAM-based optimizers (Kingma & Ba, 2014).

Original: Continual learning benchmarks.
Revised: Continual learning benchmarks.

Original: We need a large-scale and realistic benchmark to evaluate OCL.
Revised: We need a large-scale and realistic benchmark to evaluate OCL.

Original: For language modeling, Hu et al.
Revised: For language modeling, Hu et al.

Original: (2020) created the Firehose benchmark using a large stream of
Revised: (2020) created the Firehose benchmark using a large stream of

Original: Twitter posts.
Revised: Twitter posts.

Original: The task of Firehose is continual per-user tweet prediction, which is self-supervised and multi-task.
Revised: The task of Firehose is continual per-user tweet prediction, which is self-supervised and multi-task.

Original: For visual recognition, Lin et al.
Revised: For visual recognition, Lin et al.

Original: (2021) created the CLEAR benchmark by manually labeling images on a subset of YFCC100M (Thomee et al., 2016).
Revised: (2021) created the CLEAR benchmark by manually labeling images on a subset of YFCC100M (Thomee et al., 2016).

Original: Though the images are ordered in time, the number of labeled images is small (33K).
Revised: Though the images are ordered in time, the number of labeled images is small (33K).

Original: Cai et al.
Revised: Cai et al.

Original: (2021) proposed the continual localization (CLOC) benchmark using a subset of YFCC100M with time stamps and geographic locations.
Revised: (2021) proposed the continual localization (CLOC) benchmark using a subset of YFCC100M with time stamps and geographic locations.

Original: The task of CLOC is geolocalization, which is formulated as image classification.
Revised: The task of CLOC is geolocalization, which is formulated as image classification.

Original: CLOC has significant scale (39 million images taken over more than 8 years) compared to other benchmarks.
Revised: CLOC has significant scale (39 million images taken over more than 8 years) compared to other benchmarks.

Original: Due to the large scale and natural distribution shifts, we use CLOC as the main dataset to study OCL.
Revised: Due to the large scale and natural distribution shifts, we use CLOC as the main dataset to study OCL.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we formalize the online continual learning problem and introduce our notation.
Revised: In this section, we formalize the online continual learning problem and introduce our notation.

Original: We then discuss the dataset and the metrics we use.
Revised: We then discuss the dataset and the metrics we use.

Original: Problem definition.
Revised: Problem definition.

Original: Given the input domain X and the label space Y , online continual learning learns a parametric function f ( ·| θ ) :
Revised: Given the input domain X and the label space Y , online continual learning learns a parametric function f ( ·| θ ) :

Original: X → Y that maps an input x ∈ X to the corresponding label y ∈ Y .
Revised: X → Y that maps an input x ∈ X to the corresponding label y ∈ Y .

Original: At each time step t ∈ { 1 , 2 , ..., ∞} the learner interacts with the environment as follows:
Revised: At each time step t ∈ { 1 , 2 , ..., ∞} the learner interacts with the environment as follows:

Original: The environment samples the data { X t , Y t } ∼ π t and reveals the inputs X t to the learner.
Revised: The environment samples the data { X t , Y t } ∼ π t and reveals the inputs X t to the learner.

Original: The learner predicts the label for each datum x i t t ∈ X t via ˆ y i t t = f
Revised: The learner predicts the label for each datum x i t t ∈ X t via ˆ y i t t = f

Original: ( x i t t | θ t − 1 ) .
Revised: ( x i t t | θ t − 1 ) .

Original: The environment reveals the true labels Y t and the learner integrates { X t , Y t } into the data pool S t via S t = update ( S t − 1 , { X t , Y t } ) .
Revised: The environment reveals the true labels Y t and the learner integrates { X t , Y t } into the data pool S t via S t = update ( S t − 1 , { X t , Y t } ) .

Original: The learner updates the model θ t using S t .
Revised: The learner updates the model θ t using S t .

Original: Dataset.
Revised: Dataset.

Original: Due to the large scale, the smooth data stream, and natural distribution shifts, we mainly use the Continual Localization (CLOC) benchmark (Cai et al., 2021) to study OCL.
Revised: Due to the large scale, the smooth data stream, and natural distribution shifts, we mainly use the Continual Localization (CLOC) benchmark (Cai et al., 2021) to study OCL.

Original: CLOC formulates image geolocalization as a classification problem.
Revised: CLOC formulates image geolocalization as a classification problem.

Original: It contains 39 million labeled images for training,thousand images for performance evaluation, and another 2 million images for initial hyperparameter tuning.
Revised: It contains 39 million labeled images for training,thousand images for performance evaluation, and another 2 million images for initial hyperparameter tuning.

Original: The images are ordered according to their time stamps, so that images taken earlier are seen by the model first.
Revised: The images are ordered according to their time stamps, so that images taken earlier are seen by the model first.

Original: Images for training and performance evaluation cover the same time span and are sampled uniformly over time.
Revised: Images for training and performance evaluation cover the same time span and are sampled uniformly over time.

Original: Images for hyperparameter tuning are from a different time span that does not overlap with the training and evaluation data.
Revised: Images for hyperparameter tuning are from a different time span that does not overlap with the training and evaluation data.

Original: We refer to the 39 million training images as the training data , the 39 thousand evaluation images as the evaluation data , and the 2 million images for initial hyperparameter tuning as the preprocessing data .
Revised: We refer to the 39 million training images as the training data , the 39 thousand evaluation images as the evaluation data , and the 2 million images for initial hyperparameter tuning as the preprocessing data .

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: We evaluate OCL algorithms using the following metrics.
Revised: We evaluate OCL algorithms using the following metrics.

Original: Learning Efficacy: For online learning applications, the model θ t is deployed for inference on data from the next time step, i.e., X t +1 .
Revised: Learning Efficacy: For online learning applications, the model θ t is deployed for inference on data from the next time step, i.e., X t +1 .

Original: In this case, we measure the ability to adapt to new data as
Revised: In this case, we measure the ability to adapt to new data as

Original: [Equation]
Revised: [Equation]

Original: where acc ( S , θ ) is the average accuracy of the model θ on a set of data S .
Revised: where acc ( S , θ ) is the average accuracy of the model θ on a set of data S .

Original: Information Retention: When the data for inference comes from the history, we measure the ability to retain previous knowledge as
Revised: Information Retention: When the data for inference comes from the history, we measure the ability to retain previous knowledge as

Original: [Equation]
Revised: [Equation]

Original: where { X Ej , Y Ej } is the evaluation data from time step j , which has the same distribution as { X j , Y j } but unseen by the model.
Revised: where { X Ej , Y Ej } is the evaluation data from time step j , which has the same distribution as { X j , Y j } but unseen by the model.

Original: P IR ( t ) measures the generalization of θ t to historical data.
Revised: P IR ( t ) measures the generalization of θ t to historical data.

Original: Forward Transfer: When long-term predictions are required, we measure ability to generalize from current time ( t ) to future time steps ( t + k 1 to t + k 2 , k 2 > k 1 ≥ 1 ) as
Revised: Forward Transfer: When long-term predictions are required, we measure ability to generalize from current time ( t ) to future time steps ( t + k 1 to t + k 2 , k 2 > k 1 ≥ 1 ) as

Original: [Equation]
Revised: [Equation]

Original:  M ETHOD
Revised:  M ETHOD

Original: We consider replay-based methods, where the common choice for continual learning is mixed replay (Chaudhry et al., 2019).
Revised: We consider replay-based methods, where the common choice for continual learning is mixed replay (Chaudhry et al., 2019).

Original: The objective of mixed replay at time step t isminimize
Revised: The objective of mixed replay at time step t is

Original: [Equation]
Revised: [Equation]

Original: where { X t , Y t } is the data received at time t , S t − 1 is the historical data accumulated in the replay buffer, and l ( · , θ ) is the loss function.
Revised: where { X t , Y t } is the data received at time t , S t − 1 is the historical data accumulated in the replay buffer, and l ( · , θ ) is the loss function.

Original: Mixed replay constructs a minibatch of training data by sampling half from the current time step t and another half from the history (from t − B t to t −where B t decides the time range).
Revised: Mixed replay constructs a minibatch of training data by sampling half from the current time step t and another half from the history (from t − B t to t −where B t decides the time range).

Original: In offline continual learning (Chaudhry et al., 2019), B t is set to t − 1 for full coverage.
Revised: In offline continual learning (Chaudhry et al., 2019), B t is set to t − 1 for full coverage.

Original: Cai et al.
Revised: Cai et al.

Original: (2021) adaptively choose B t to optimize learning efficacy in OCL.
Revised: (2021) adaptively choose B t to optimize learning efficacy in OCL.

Original: In this paper, we focus on information retention, more specifically its optimization.
Revised: In this paper, we focus on information retention, more specifically its optimization.

Original: A reasonable objective for pure information retention is pure replay , which is defined as follows at each time step t :
Revised: A reasonable objective for pure information retention is pure replay , which is defined as follows at each time step t :

Original: [Equation]
Revised: [Equation]

Original: Pure replay constructs a minibatch of training data by sampling uniformly from all historical time steps, i.e., from 1 to t , which encourages remembering all historical knowledge.
Revised: Pure replay constructs a minibatch of training data by sampling uniformly from all historical time steps, i.e., from 1 to t , which encourages remembering all historical knowledge.

Original: As shown in
Revised: As shown in

Original: Appx.
Revised: Appx.

Original: C.6, optimizing (5) (as opposed to (4)) improves both information retention and forward transfer by sacrificing learning efficacy.
Revised: C.6, optimizing (5) (as opposed to (4)) improves both information retention and forward transfer by sacrificing learning efficacy.

Original: 4. O PTIMIZING I NFORMATION R ETENTION
Revised: 4. O PTIMIZING I NFORMATION R ETENTION

Original: When the distribution of S t remains unchanged over time (i.e., the loss function is stationary), the standard optimizer for problem (5) is Stochastic Gradient Descent (SGD).
Revised: When the distribution of S t remains unchanged over time (i.e., the loss function is stationary), the standard optimizer for problem (5) is Stochastic Gradient Descent (SGD).

Original: This choice is typically carried over to continual learning.
Revised: This choice is typically carried over to continual learning.

Original: At each update iteration k ( k ̸ = t if multiple updates are allowed at each time step of continual learning), SGD updates the model via
Revised: At each update iteration k ( k ̸ = t if multiple updates are allowed at each time step of continual learning), SGD updates the model via

Original: [Equation]
Revised: [Equation]

Original: 6) where α k is the learning rate and g k ( θ k − 1 ) is the stochastic gradient of the objective at iteration k .
Revised: (6) where α k is the learning rate and g k ( θ k − 1 ) is the stochastic gradient of the objective at iteration k .

Original: Although the convergence behavior of SGD is well understood for the case of stationary losses, its implications for OCL are not clear.
Revised: Although the convergence behavior of SGD is well understood for the case of stationary losses, its implications for OCL are not clear.

Original: Hence, we extend the analysis of SGD (Ghadimi & Lan, 2013) to the continual learning case.
Revised: Hence, we extend the analysis of SGD (Ghadimi & Lan, 2013) to the continual learning case.

Original: Denote the loss function at iteration k by l k ( θ ) and assume:
Revised: Denote the loss function at iteration k by l k ( θ ) and assume:

Original: A1 ∥
Revised: [Equation]

Original:  ▽ l k ( θ ) − ▽ l k ( θ ′ )
Revised: ∥ ▽ l k ( θ ) − ▽ l k ( θ ′ )

Original: ∥ ≤ L ∥
Revised: ∥ ≤ L ∥

Original: θ − θ ′ ∥ ,
Revised: θ − θ ′ ∥ ,

Original: ∀ θ , θ ′ ∈ R d and k (Lipschitz smoothness).
Revised: ∀ θ , θ ′ ∈ R d and k (Lipschitz smoothness).

Original: A2 E [ g k ( θ )] =
Revised: A2 E [ g k ( θ )] =

Original: ▽ l k ( θ ) , ∀ θ ∈ R d and k (Unbiased gradient estimates).
Revised: ▽ l k ( θ ) , ∀ θ ∈ R d and k (Unbiased gradient estimates).

Original: A3 ∥
Revised: A3 ∥

Original: ▽ l k ( θ ) − g k ( θ ) ∥ 2 ≤ ρ 2 , ∀ θ
Revised: ▽ l k ( θ ) − g k ( θ ) ∥ 2 ≤ ρ 2 , ∀ θ

Original: ∈ R
Revised: ∈ R

Original: d (Bounded gradient noise).
Revised: d (Bounded gradient noise).

Original: A4 | l k +1 ( θ ) − l k ( θ ) | ≤ χ k , ∀ θ and k (Bounded non-stationarity).
Revised: A4 | l k +1 ( θ ) − l k ( θ ) | ≤ χ k , ∀ θ and k (Bounded non-stationarity).

Original: A1 to A3 extend the standard assumptions for SGD analysis (Ghadimi & Lan, 2013) from the stationary case to the non-stationary case.
Revised: A1 to A3 extend the standard assumptions for SGD analysis (Ghadimi & Lan, 2013) from the stationary case to the non-stationary case.

Original: And A4 bounds the degree of non-stationarity between consecutive iterations.
Revised: And A4 bounds the degree of non-stationarity between consecutive iterations.

Original: The expectations are taken over the randomness of the stochastic gradient noise.
Revised: The expectations are taken over the randomness of the stochastic gradient noise.

Original: With these assumptions, we state the following theorem and defer its proof to Appx.
Revised: With these assumptions, we state the following theorem and defer its proof to Appx.

Original: A.2 (also see Appx.
Revised: A.2 (also see Appx.

Original: A.2 for details about how to interpret this result under limited storage):
Revised: A.2 for details about how to interpret this result under limited storage):

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume A1 to A4 , and let α k <
Revised: Assume A1 to A4 , and let α k <

Original: L 2 ,
Revised: L 2 ,

Original: [Equation]
Revised: [Equation]

Original: where T 1 = 2( l 1 ( θ 
Revised: where T 1 = 2( l 1 ( θ 0 ) − E [ l k +2 ( θ k +1 )]) (cid:80) kj

Original: [Equation]
Revised: [Equation]

Original: − E [
Revised: and T

Original: [Equation]
Revised: [Equation]

Original: Therorem 1 bounds the minimum gradient norm achievable by SGD during OCL.
Revised: Therorem 1 bounds the minimum gradient norm achievable by SGD during OCL.

Original: A similar bound in the stationary case (Theorem 2 in Appx.
Revised: A similar bound in the stationary case (Theorem 2 in Appx.

Original: A.1) has only the terms T 1 and T 2 .
Revised: A.1) has only the terms T 1 and T 2 .

Original: Hence, T 3 is the cost of the non-stationarity.
Revised: Hence, T 3 is the cost of the non-stationarity.

Original: In the stationary case (when T 3 = 0 ), we can simply find the optimal learning rate by trading off between T 1 and T 2 .
Revised: In the stationary case (when T 3 = 0 ), we can simply find the optimal learning rate by trading off between T 1 and T 2 .

Original: Specifically, for a constant learning rate, T 1 converges toat a linear rate, but T 2 is constant.
Revised: Specifically, for a constant learning rate, T 1 converges toat a linear rate, but T 2 is constant.

Original: Hence, the strategy for convergence is reducing learning rates to control T 2 simultaneously with T 1 .
Revised: Hence, the strategy for convergence is reducing learning rates to control T 2 simultaneously with T 1 .

Original: This result supports the effectiveness of the standard “ reducewhen-plateau ” (RWP) learning rate heuristic in the stationary case.
Revised: This result supports the effectiveness of the standard “ reducewhen-plateau ” (RWP) learning rate heuristic in the stationary case.

Original: Since the access to the true gradient norm is not practical, RWP often uses the validation accuracy/loss as a surrogate to control the learning rate α k , where we reduce α k by a certain factor β once the validation accuracy/loss plateaus (Goodfellow et al., 2016;
Revised: Since the access to the true gradient norm is not practical, RWP often uses the validation accuracy/loss as a surrogate to control the learning rate α k , where we reduce α k by a certain factor β once the validation accuracy/loss plateaus (Goodfellow et al., 2016;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: In the continual learning setting (when T 3 ̸ = 0 ), RWP can be less effective.
Revised: In the continual learning setting (when T 3 ̸ = 0 ), RWP can be less effective.

Original: For example, if χ k ≥ χ > 0 for a constant χ , i.e., the loss function l k ( · ) changes at least at a linear rate, then Tcan grow linearly when the learning rate is reduced to 0.
Revised: For example, if χ k ≥ χ > 0 for a constant χ , i.e., the loss function l k ( · ) changes at least at a linear rate, then Tcan grow linearly when the learning rate is reduced to 0.

Original: Hence in OCL, there can be cases where simultaneous convergence of T 1 , T 2 , and T 3 are not possible.
Revised: Hence in OCL, there can be cases where simultaneous convergence of T 1 , T 2 , and T 3 are not possible.

Original: Controlling T 3 requires a reasonably large learning rate, whereas using a large learning rate increases T 2 .
Revised: Controlling T 3 requires a reasonably large learning rate, whereas using a large learning rate increases T 2 .

Original: Finally, the χ k term can cause the learning rate annealing signal in RWP, i.e., the gradient norm or the validation loss/accuracy, to be unstable in OCL.
Revised: Finally, the χ k term can cause the learning rate annealing signal in RWP, i.e., the gradient norm or the validation loss/accuracy, to be unstable in OCL.

Original: We argue that these are the core obstacles in application of SGD to OCL.
Revised: We argue that these are the core obstacles in application of SGD to OCL.

Original: In summary, the major issues hindering the performance of OCL when optimized with SGD are:
Revised: In summary, the major issues hindering the performance of OCL when optimized with SGD are:

Original: P1 OCL needs large learning rates 
Revised: P1 OCL needs large learning rates due

Original: due to non-stationarity.
Revised: to non-stationarity.

Original: However, convergence requires smaller learning rates.
Revised: However, convergence requires smaller learning rates

Original: P2 RWP allows the learning rate to be infinitesimal, which prevents the model from adapting to new data.
Revised: P2 RWP allows the learning rate to be infinitesimal, which prevents the model from adapting to new data.

Original: P3 The validation performance is not always indicative of learning.
Revised: P3 The validation performance is not always indicative of learning.

Original: E.g., due to the distribution shift of OCL, the validation accuracy can decrease, even when the model is still improving.
Revised: E.g., due to the distribution shift of OCL, the validation accuracy can decrease, even when the model is still improving.

Original: To address P1 , we propose an Adaptive Moving Average (AMA) optimizer in Sec. 4.2.
Revised: To address P1 , we propose an Adaptive Moving Average (AMA) optimizer in Sec. 4.2.

Original: Then, to address P2 and P3 , we introduce a moving-average-based learning rate (MALR) schedule in Sec. 4.3.
Revised: Then, to address P2 and P3 , we introduce a moving-average-based learning rate (MALR) schedule in Sec. 4.3.

Original: 4. A DAPTIVE M OVING A VERAGE
Revised: 4. A DAPTIVE M OVING A VERAGE

Original: Our Adaptive Moving Average (AMA) optimizer extends Exponential Moving Average (EMA) (Tarvainen & Valpola, 2017).
Revised: Our Adaptive Moving Average (AMA) optimizer extends Exponential Moving Average (EMA) (Tarvainen & Valpola, 2017).

Original: Moving average optimizers maintain two models: the SGD model θ k and the MA model θ MAk .
Revised: Moving average optimizers maintain two models: the SGD model θ k and the MA model θ MAk .

Original: At each iteration k , θ k is first updated with (6).
Revised: At each iteration k , θ k is first updated with (6).

Original: Then, θ MAk is updated via
Revised: Then, θ MAk is updated via

Original: [Equation]
Revised: [Equation]

Original: (8) where the MA weight γ k ∈ [0 , 1] is a hyperparameter.
Revised: (8) where the MA weight γ k ∈ [0 , 1] is a hyperparameter.

Original: In EMA, γ k is a constant.
Revised: In EMA, γ k is a constant.

Original: In AMA, we adapt γ k over time using population-based search (Jaderberg et al., 2017) to improve the performance.
Revised: In AMA, we adapt γ k over time using population-based search (Jaderberg et al., 2017) to improve the performance.

Original: Why does AMA help information retention?
Revised: Why does AMA help information retention?

Original: Given stationary losses, SGD can converge to local optima by reducing the learning rate.
Revised: Given stationary losses, SGD can converge to local optima by reducing the learning rate.

Original: In this case, the final solution found by AMA will not be very different from SGD as we validate in Appx.
Revised: In this case, the final solution found by AMA will not be very different from SGD as we validate in Appx.

Original: C.10.
Revised: C.10.

Original: In the non-stationary case, SGD needs reasonably large learning rates to track the shifting optimum over time.
Revised: In the non-stationary case, SGD needs reasonably large learning rates to track the shifting optimum over time.

Original: However, the variance ( T 2 in (7)) of
Revised: However, the variance ( T 2 in (7)) of

Original: SGD increases with large learning rates, hurting convergence.
Revised: SGD increases with large learning rates, hurting convergence.

Original: The advantage of AMA is to reduce the variance of SGD.
Revised: The advantage of AMA is to reduce the variance of SGD.

Original: Given stationary losses, Mandt et al.
Revised: Given stationary losses, Mandt et al.

Original: (2016) have shown that SGD with a constant learning rate will eventually bounce around a region centered at the optimum.
Revised: (2016) have shown that SGD with a constant learning rate will eventually bounce around a region centered at the optimum.

Original: Averaging the SGD trajectory with uniform weights estimates the region center, i.e., the optimal solution.
Revised: Averaging the SGD trajectory with uniform weights estimates the region center, i.e., the optimal solution.

Original: This observation inspires the design of AMA.
Revised: This observation inspires the design of AMA.

Original: Informally, SGD with large learning rates would follow and bounce around the shifting optimum in OCL.
Revised: Informally, SGD with large learning rates would follow and bounce around the shifting optimum in OCL.

Original: The moving average of these high-variance iterates tracks the shifting optimum better.
Revised: The moving average of these high-variance iterates tracks the shifting optimum better.

Original: As a further informal justification, consider the unfolded AMA update rule (8):
Revised: As a further informal justification, consider the unfolded AMA update rule (8):

Original: [Equation]
Revised: [Equation]

Original: where θ 0 is the initial solution of SGD and AMA.
Revised: where θ 0 is the initial solution of SGD and AMA.

Original: (9) shows that the AMA model θ MAk is a linear combination of the SGD trajectory { θ 0 , ..., θ k } .
Revised: (9) shows that the AMA model θ MAk is a linear combination of the SGD trajectory { θ 0 , ..., θ k } .

Original: Since γ k ∈ [0 , 1] , θ MAk lies inside the convex hull of { θ 0 , ..., θ k } .
Revised: Since γ k ∈ [0 , 1] , θ MAk lies inside the convex hull of { θ 0 , ..., θ k } .

Original: The population-based search aims to find the best point within the convex hull by adapting γ k .
Revised: The population-based search aims to find the best point within the convex hull by adapting γ k .

Original: We summarize our method in Alg. 1.
Revised: We summarize our method in Alg. 1.

Original: To adapt γ k over time, we maintain two MA models, θ MAand θ MA 2 , and update them using different weights, γ MA 1 and γ MA 2 (Line 4).
Revised: To adapt γ k over time, we maintain two MA models, θ MAand θ MA 2 , and update them using different weights, γ MA 1 and γ MA 2 (Line 4).

Original: To decide the best model and weight, we use the validation accuracy Acc 1 and Acc 2 (Line 5), which we define next.
Revised: To decide the best model and weight, we use the validation accuracy Acc 1 and Acc 2 (Line 5), which we define next.

Original: Once every K W iterations, we copy the parameter of the best MA model to the other one, and adapt the MA weights (Line 7 to 10).
Revised: Once every K W iterations, we copy the parameter of the best MA model to the other one, and adapt the MA weights (Line 7 to 10).

Original: In all our experiments, we set the initial MA weight γ 0 to 0.99, the
Revised: In all our experiments, we set the initial MA weight γ 0 to 0.99, the

Original: MA weight adjustment coefficient δ to 5, and the interval K W to adjust weights to 10000 iterations.
Revised: MA weight adjustment coefficient δ to 5, and the interval K W to adjust weights to 10000 iterations.

Original: We continuously track a validation metric for adaptation.
Revised: We continuously track a validation metric for adaptation.

Original: At each time step t , a holdout information retention validation set S Vt is updated, which has the same distribution as S t .
Revised: At each time step t , a holdout information retention validation set S Vt is updated, which has the same distribution as S t .

Original: To maintain S Vt online, whenever new data { X t , Y t } is received, we put a random subset ( 5% in all experiments) into S Vt and put the rest into S t .
Revised: To maintain S Vt online, whenever new data { X t , Y t } is received, we put a random subset ( 5% in all experiments) into S Vt and put the rest into S t .

Original: We compute the information retention validation accuracy/loss on S Vt in an online fashion (described in Appx. B).
Revised: We compute the information retention validation accuracy/loss on S Vt in an online fashion (described in Appx. B).

Original: In this way, we can parallelize validation with model updates,
Revised: In this way, we can parallelize validation with model updates,

Original: Require: Initial model θ 0 , initial moving average weight γ 0 , MA weight adjustment coefficient δ( > 0) , MA weight update interval K W .
Revised: Require: Initial model θ 0 , initial moving average weight γ 0 , MA weight adjustment coefficient δ

Original: Update Update validation accuracy Acc 1 and Acc 2 online 6: If Acc 1 >Acc 2 then i best = 1 else i best =7: if k mod K W = 0 then 8:
Revised: >

Original: Copy the weights of θ MA ibest to the other. Acc 1 ← 0 ,Acc 2 ← 0 . Increase γ MA 1 & γ MA 2 by a factor of δ if i best = 1, and decrease otherwise.
Revised: , MA weight update interval K W .

Original: 10: end if 11: end for and choose the best MA model for inference anytime during OCL.
Revised: and choose the best MA model for inference anytime during OCL.

Original: We use the information retention validation accuracy/loss both in Algorithm 1 and later for learning rate control.
Revised: We use the information retention validation accuracy/loss both in Algorithm 1 and later for learning rate control.

Original: We propose several strategies to reduce the overhead of MA and population-based search.
Revised: We propose several strategies to reduce the overhead of MA and population-based search.

Original: The overhead from MA lies in 1) updating the MA model and 2) recomputing the batch normalization (BN) statistics (Ioffe & Szegedy, 2015).
Revised: The overhead from MA lies in 1) updating the MA model and 2) recomputing the batch normalization (BN) statistics (Ioffe & Szegedy, 2015).

Original: To solve 1, we perform the MA update only every K M iterations.
Revised: To solve 1, we perform the MA update only every K M iterations.

Original: We set K M = 10 in all experiments and do not observe a performance drop.
Revised: We set K M = 10 in all experiments and do not observe a performance drop.

Original: To solve 2, we replace BN with group normalization (GN)
Revised: To solve 2, we replace BN with group normalization (GN)

Original: + weight standardization (WS) (Qiao et al., 2019), which does not store normalization statistics.
Revised: + weight standardization (WS) (Qiao et al., 2019), which does not store normalization statistics.

Original: We evaluate its impact in Appx.
Revised: We evaluate its impact in Appx.

Original: C.1.
Revised: C.1.

Original: The overhead from population-based search lies in the online validation operation (Line 5).
Revised: The overhead from population-based search lies in the online validation operation (Line 5).

Original: Each operation requires one forward pass on a batch of validation data.
Revised: Each operation requires one forward pass on a batch of validation data.

Original: To reduce the overhead, we execute Line 5 every
Revised: To reduce the overhead, we execute Line 5 every

Original: K V = 20 iterations.
Revised: K V = 20 iterations.

Original: See Appx.
Revised: See Appx.

Original: B for a detailed version of Algorithm 1 with the above strategies.
Revised: B for a detailed version of Algorithm 1 with the above strategies.

Original: 4. M OVING -A VERAGE - BASED L EARNING R ATE S CHEDULE (MALR)
Revised: 4. M OVING -A VERAGE - BASED L EARNING R ATE S CHEDULE (MALR)

Original: As discussed in ( P3 ), monitoring only the validation accuracy is not sufficient for learning rate control as the effects of learning and distribution shifts counteract each other.
Revised: As discussed in ( P3 ), monitoring only the validation accuracy is not sufficient for learning rate control as the effects of learning and distribution shifts counteract each other.

Original: Interestingly, AMA provides a better signal for learning rate control as a by-product.
Revised: Interestingly, AMA provides a better signal for learning rate control as a by-product.

Original: Specifically, we use the difference between information retention validation loss/accuracy of the MA model θ MA ibest and the SGD model θ k .
Revised: Specifically, we use the difference between information retention validation loss/accuracy of the MA model θ MA ibest and the SGD model θ k .

Original: We denote this difference at iteration k by σ k .
Revised: We denote this difference at iteration k by σ k .

Original: Intuitively, since AMA performs better than SGD given non-minimal learning rates, σ k can be used to estimate the performance gain obtainable by learning rate reduction.
Revised: Intuitively, since AMA performs better than SGD given non-minimal learning rates, σ k can be used to estimate the performance gain obtainable by learning rate reduction.

Original: To control the learning rate reduction speed, we do not reduce learning rates unless σ k plateaus, indicating that we cannot achieve better long-term performance gain.
Revised: To control the learning rate reduction speed, we do not reduce learning rates unless σ k plateaus, indicating that we cannot achieve better long-term performance gain.

Original: On the other hand, when the learning rate is too small, potentially incapable of adapting to new data, σ k will also be very small.
Revised: On the other hand, when the learning rate is too small, potentially incapable of adapting to new data, σ k will also be very small.

Original: Hence, we do not reduce the learning rate unless σ k exceeds a certain threshold.
Revised: Hence, we do not reduce the learning rate unless σ k exceeds a certain threshold.

Original: In summary, MALR reduces the learning rate when the following conditions are satisfied.
Revised: In summary, MALR reduces the learning rate when the following conditions are satisfied.

Original: C1 :
Revised: C1 :

Original: The information retention validation performance (of the MA model) does not improve for K R iterations.
Revised: The information retention validation performance (of the MA model) does not improve for K R iterations.

Original: (This is the original RWP condition.)
Revised: (This is the original RWP condition.)

Original: C2 : σ k does not increase for K R iterations.
Revised: C2 : σ k does not increase for K R iterations.

Original: C3 :
Revised: C3 :

Original: σ k > ϵ .
Revised: σ k > ϵ .

Original: Computational complexity.
Revised: Computational complexity.

Original: The average computational complexity of SGD+RWP per iteration is c SGD + RWP =
Revised: The average computational complexity of SGD+RWP per iteration is c SGD + RWP =

Original: ( K V +1 K V )
Revised: ( K V +1 K V )

Original: c F + c G + c U , where c F is the complexity of one forward pass, K V is the interval to do one online validation step, c G and c U are respectively the complexity for computing the gradient and updating the model.
Revised: c F + c G + c U , where c F is the complexity of one forward pass, K V is the interval to do one online validation step, c G and c U are respectively the complexity for computing the gradient and updating the model.

Original: The complexity of AMA+MALR is c AMA + MALR =
Revised: The complexity of AMA+MALR is c AMA + MALR =

Original: ( K V +3 K V )
Revised: ( K V +3 K V )

Original: c F + c G +
Revised: c F + c G +

Original: ( K M +2 K M ) c U , where K M is the interval to do one MA update step.
Revised: ( K M +2 K M ) c U , where K M is the interval to do one MA update step.

Original: When K V and
Revised: When K V and

Original: K M are large, the overhead of AMA+MALR is small.
Revised: K M are large, the overhead of AMA+MALR is small.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Implementation details.
Revised: Implementation details.

Original: We mainly use CLOC (Cai et al., 2021) to study OCL at large scale.
Revised: We mainly use CLOC (Cai et al., 2021) to study OCL at large scale.

Original: Similar to the setting of Cai et al.
Revised: Similar to the setting of Cai et al.

Original: (2021), the model receives 256 images at each time step.
Revised: (2021), the model receives 256 images at each time step.

Original: We use the
Revised: We use the

Original: ResNet-50 architecture, a batch size of 256 and tune the initial hyperparameters on the preprocessing data.
Revised: ResNet-50 architecture, a batch size of 256 and tune the initial hyperparameters on the preprocessing data.

Original: We set the initial learning rate α 0 to 0 .
Revised: We set the initial learning rate α 0 to 0 .

Original: 025 , the momentum of SGD to 0 .
Revised: 025 , the momentum of SGD to 0 .

Original: 9 , the weight decay to
Revised: 9 , the weight decay to

Original: [Figure]
Revised: [Figure]

Original: RWP and CLR.

Original: (c) & (d): Under distribution shifts, the validation loss increased for constant learning rates in both long and short terms.
Revised: d): Under distribution shifts, the validation loss increased for constant learning rates in both long and short terms.

Original: This made RWP reduce the learning rate too fast.
Revised: This made RWP reduce the learning rate too fast.

Original: (e): Given a constant learning rate, σ k in MALR first increased and then plateaued.
Revised: (e): Given a constant learning rate, σ k in MALR first increased and then plateaued.

Original: σ k declined with the learning rate reduction.
Revised: σ k declined with the learning rate reduction.

Original: This trend was stable even under distribution shifts.
Revised: This trend was stable even under distribution shifts.

Original: 1 e − 4 , and the loss to cross-entropy.
Revised: 1 e − 4 , and the loss to cross-entropy.

Original: For RWP, we reduce the learning rate by half if the information retention validation loss stops decreasing for K R = 60000 training iterations.
Revised: For RWP, we reduce the learning rate by half if the information retention validation loss stops decreasing for K R = 60000 training iterations.

Original: For MALR, we set σ k as the difference of the information retention validation accuracy between the MA and the SGD models, and set ϵ in C3 to 3 . 0% .
Revised: For MALR, we set σ k as the difference of the information retention validation accuracy between the MA and the SGD models, and set ϵ in C3 to 3 . 0% .

Original: To decouple the effect of the storage limit and better focus on studying the problem of limited computation in large scale OCL, we use a replay buffer size ofmillion images in main experiments.
Revised: To decouple the effect of the storage limit and better focus on studying the problem of limited computation in large scale OCL, we use a replay buffer size ofmillion images in main experiments.

Original: Appx.
Revised: Appx.

Original: C.4 ablates the effect of replay buffer sizes and shows that large scale OCL only requires a reasonably large replay buffer (4 million images for CLOC).
Revised: C.4 ablates the effect of replay buffer sizes and shows that large scale OCL only requires a reasonably large replay buffer (4 million images for CLOC).

Original: We use the same set of hyperparameters introduced by AMA/MALR in all experiments to verify their robustness.
Revised: We use the same set of hyperparameters introduced by AMA/MALR in all experiments to verify their robustness.

Original: (See Appx.
Revised: (See Appx.

Original: C.11 for further details.)
Revised: C.11 for further details.)

Original: All models are trained with 8 Nvidia Geforce
Revised: All models are trained with 8 Nvidia Geforce

Original: Rtx 2080 Ti GPUs.
Revised: Rtx 2080 Ti GPUs.

Original: We refer to the information retention validation loss/accuracy throughout the experiments as the validation loss/accuracy.
Revised: We refer to the information retention validation loss/accuracy throughout the experiments as the validation loss/accuracy.

Original: The arrows ( ↑ / ↓ ) in the caption of each figure point towards the direction of improvement.
Revised: The arrows ( ↑ / ↓ ) in the caption of each figure point towards the direction of improvement.

Original: 5. M AIN R ESULT : E FFECTIVENESS OF AMA AND MALR
Revised: 5. M AIN R ESULT : E FFECTIVENESS OF AMA AND MALR

Original: To study the effectiveness of AMA + MALR, we compare it against SGD + RWP (SGD is used as the base optimizer due to high performance, see Appx.
Revised: To study the effectiveness of AMA + MALR, we compare it against SGD + RWP (SGD is used as the base optimizer due to high performance, see Appx.

Original: C.5 for similar experiments on ADAM).
Revised: C.5 for similar experiments on ADAM).

Original: To separate the effects of optimizers and learning rate schedules, we also compare against AMA + RWP, AMA + constant learning rate (CLR), the SGD model in AMA+MALR, and SGD+CLR (using training + validation data for training since no online hyperparameter adaptation is needed).
Revised: To separate the effects of optimizers and learning rate schedules, we also compare against AMA + RWP, AMA + constant learning rate (CLR), the SGD model in AMA+MALR, and SGD+CLR (using training + validation data for training since no online hyperparameter adaptation is needed).

Original: We allow 80 training iterations per time step, so that the number of training iterations is large enough and learning rate annealing is necessary.
Revised: We allow 80 training iterations per time step, so that the number of training iterations is large enough and learning rate annealing is necessary.

Original: We evaluate information retention using P IR ( t ) .
Revised: We evaluate information retention using P IR ( t ) .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1a, AMA+MALR outperformed all competitors.
Revised: 1a, AMA+MALR outperformed all competitors.

Original: The “jumps” of the performance curves were due to the learning rate change in RWP or MALR.
Revised: The “jumps” of the performance curves were due to the learning rate change in RWP or MALR.

Original: Note that the information retention in
Revised: Note that the information retention in

Original: Fig.
Revised: Fig.

Original: 1a was not monotonically increasing because of the distribution shifts.
Revised: 1a was not monotonically increasing because of the distribution shifts.

Original: In terms of the optimizer,
Revised: In terms of the optimizer,

Original: AMA was better than SGD in all learning rate schedules.
Revised: AMA was better than SGD in all learning rate schedules.

Original: Moreover, the performance gap was especially large for large learning rates, validating P1 .
Revised: Moreover, the performance gap was especially large for large learning rates, validating P1 .

Original: In terms of the learning rate schedule, MALR outperformed both RWP and CLR.
Revised: In terms of the learning rate schedule, MALR outperformed both RWP and CLR.

Original: Specifically, the distribution shift made the validation loss increase over time for constant learning rates (Fig. 1c and 1d).
Revised: Specifically, the distribution shift made the validation loss increase over time for constant learning rates (Fig. 1c and 1d).

Original: This phenomenon caused RWP to reduce the learning rate too fast (Fig. 1b).
Revised: This phenomenon caused RWP to reduce the learning rate too fast (Fig. 1b).

Original: As a result, AMA+RWP and SGD+RWP performed even worse than
Revised: As a result, AMA+RWP and SGD+RWP performed even worse than

Original: SGD+CLR in the long term.
Revised: SGD+CLR in the long term.

Original: In contrast, MALR effectively controlled the learning rate reduction speed and the minimum learning rate with the help of the new signal σ k (Fig. 1e), which was more robust to distribution shifts than the validation loss/accuracy.
Revised: In contrast, MALR effectively controlled the learning rate reduction speed and the minimum learning rate with the help of the new signal σ k (Fig. 1e), which was more robust to distribution shifts than the validation loss/accuracy.

Original: 5. A BLATIONS
Revised: 5. A BLATIONS

Original: MALR extends RWP by introducing C2 and C3 .
Revised: MALR extends RWP by introducing C2 and C3 .

Original: To understand the effect of each condition, we compare AMA+MALR to versions where C2 is removed (AMA+MALR+No C2), C3 is removed (AMA+MALR+No C3), and both are removed (AMA+RWP).
Revised: To understand the effect of each condition, we compare AMA+MALR to versions where C2 is removed (AMA+MALR+No C2), C3 is removed (AMA+MALR+No C3), and both are removed (AMA+RWP).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2a, AMA+MALR+No C3 failed to limit the minimum learning rate ( P2 ), and performed worse than AMA+MALR in the long term.
Revised: 2a, AMA+MALR+No C3 failed to limit the minimum learning rate ( P2 ), and performed worse than AMA+MALR in the long term.

Original: AMA+MALR+No C2 reduced the learning rate too quickly ( P3 ) at the early stage compared to AMA+MALR.
Revised: AMA+MALR+No C2 reduced the learning rate too quickly ( P3 ) at the early stage compared to AMA+MALR.

Original: With both C2 and C3 removed,
Revised: With both C2 and C3 removed,

Original: AMA+RWP performed the worst in all methods.
Revised: AMA+RWP performed the worst in all methods.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2c, the performance of all competitors dropped when σ k was too low, which happened when the learning rate was too small
Revised: 2c, the performance of all competitors dropped when σ k was too low, which happened when the learning rate was too small

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: vs EMA .

Revised: AMA performed better than EMA by adapting the MA weight over time.

Original: (Fig. 2b).
Revised: (Fig. 2b).

Original: Hence, σ k is an effective learning rate annealing signal for OCL.
Revised: Hence, σ k is an effective learning rate annealing signal for OCL.

Original: Appx.
Revised: Appx.

Original: C.2 further analyzes the effect of using suboptimal learning rates.
Revised: C.2 further analyzes the effect of using suboptimal learning rates.

Original: To summarize, large learning rates led to poor convergence and hurt performance on data from all time ranges .
Revised: To summarize, large learning rates led to poor convergence and hurt performance on data from all time ranges .

Original: On the other hand, small learning rates prevented the model from adapting to new data.
Revised: On the other hand, small learning rates prevented the model from adapting to new data.

Original: AMA vs EMA.
Revised: AMA vs EMA.

Original: We study the effect of population-based MA weight adaptation.
Revised: We study the effect of population-based MA weight adaptation.

Original: We compare AMA with EMA and use the learning rate generated by AMA+MALR on both models to remove the effect of learning rate annealing.
Revised: We compare AMA with EMA and use the learning rate generated by AMA+MALR on both models to remove the effect of learning rate annealing.

Original: Due to resource limitations, we train EMA only for 700 thousand iterations.
Revised: Due to resource limitations, we train EMA only for 700 thousand iterations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, AMA outperformed EMA by adapting the MA weight over time.
Revised: 3, AMA outperformed EMA by adapting the MA weight over time.

Original: Further analyses in the appendix.
Revised: Further analyses in the appendix.

Original: Due to the space limit, we provide the following analyses in the appendix.
Revised: Due to the space limit, we provide the following analyses in the appendix.

Original: Effect of batch sizes in Appx.
Revised: Effect of batch sizes in Appx.

Original: C.3 :
Revised: C.3 :

Original: Heavy parallelization by increasing batch sizes and decreasing the number of training iterations has negative impact on all OCL metrics.
Revised: Heavy parallelization by increasing batch sizes and decreasing the number of training iterations has negative impact on all OCL metrics.

Original: Effect of training objectives, i.e., (5) vs (4) , in Appx.
Revised: Effect of training objectives, i.e., (5) vs (4) , in Appx.

Original: C.6 :
Revised: C.6 :

Original: Optimizing information retention (using (5)) is more suitable for both information retention and long-term forward transfer, at the cost of sacrificing learning efficacy.
Revised: Optimizing information retention (using (5)) is more suitable for both information retention and long-term forward transfer, at the cost of sacrificing learning efficacy.

Original: More importantly, increasing the computation budget for optimizing information retention improved all OCL metrics, whereas increasing the computation budget for optimizing learning efficacy (using (4)) hurt information retention and forward transfer.
Revised: More importantly, increasing the computation budget for optimizing information retention improved all OCL metrics, whereas increasing the computation budget for optimizing learning efficacy (using (4)) hurt information retention and forward transfer.

Original: 5.2. E FFECTIVENESS OF AMA BEYOND CLOC
Revised: 5.2. E FFECTIVENESS OF AMA BEYOND CLOC

Original: Google Landmarks.
Revised: Google Landmarks.

Original: We apply AMA+MALR to Google Landmarks v2 (Weyand et al., 2020), which contains images from different landmarks (each representing a class).
Revised: We apply AMA+MALR to Google Landmarks v2 (Weyand et al., 2020), which contains images from different landmarks (each representing a class).

Original: We construct our continual dataset using a subset of 580 thousand training images with time stamps.
Revised: We construct our continual dataset using a subset of 580 thousand training images with time stamps.

Original: We refer to the constructed dataset as C ontinual G oogle L and M arks (CGLM) .
Revised: We refer to the constructed dataset as C ontinual G oogle L and M arks (CGLM) .

Original: Similar to CLOC, 256 new images are revealed at each time step, and 80 training iterations are allowed per time step (see Appx.
Revised: Similar to CLOC, 256 new images are revealed at each time step, and 80 training iterations are allowed per time step (see Appx.

Original: C.7 for more implementation details).
Revised: C.7 for more implementation details).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4-a, AMA performed better than SGD in CGLM.
Revised: 4-a, AMA performed better than SGD in CGLM.

Original: Since
Revised: Since

Original: CGLM is small-scale, the learning rate of both RWP and MALR remained constant throughout most of the training.
Revised: CGLM is small-scale, the learning rate of both RWP and MALR remained constant throughout most of the training.

Original: Hence, we only reported the performance of AMA+MALR and the SGD model.
Revised: Hence, we only reported the performance of AMA+MALR and the SGD model.

Original: To further check this limitation, we also looked at the first 580K images of CLOC in Figs.
Revised: To further check this limitation, we also looked at the first 580K images of CLOC in Figs.

Original: 4-c and 4-d.
Revised: 4-c and 4-d.

Original: Similar to CGLM, the relative merit of different learning rate schedules was unclear.
Revised: Similar to CGLM, the relative merit of different learning rate schedules was unclear.

Original: Hence, a large-scale dataset is necessary for analyzing OCL learning rate schedules.
Revised: Hence, a large-scale dataset is necessary for analyzing OCL learning rate schedules.

Original: ImageNet.
Revised: ImageNet.

Original: We test our method on task-based benchmarks synthesized from ImageNet.
Revised: We test our method on task-based benchmarks synthesized from ImageNet.

Original: Specifically, we create a 100-task continual learning problem by randomly dividing ImageNet classes, so that each task contains data from 10 classes.
Revised: Specifically, we create a 100-task continual learning problem by randomly dividing ImageNet classes, so that each task contains data from 10 classes.

Original: During training, we allow 120 | X t |training iterations per task,
Revised: During training, we allow 120 | X t |training iterations per task,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The final validation accuracy is close to the supervised learning model.
Revised: The final validation accuracy is close to the supervised learning model.

Original: Due to the implicit computation increase in each model update iteration, OCL methods performed worse than simple SGD+Cyclic when the memory is sufficiently large and the computation is explicitly limited.
Revised: Due to the implicit computation increase in each model update iteration, OCL methods performed worse than simple SGD+Cyclic when the memory is sufficiently large and the computation is explicitly limited.

Original: where | X t | is the number of images of task t and 256 is the training batch size.
Revised: where | X t | is the number of images of task t and 256 is the training batch size.

Original: In this setting, the total number of SGD steps of a continual learner trained on all tasks is the same as the standard supervised learning model trained for 120 epochs.
Revised: In this setting, the total number of SGD steps of a continual learner trained on all tasks is the same as the standard supervised learning model trained for 120 epochs.

Original: During evaluation, we compute the average accuracy on validation data from all revealed tasks.
Revised: During evaluation, we compute the average accuracy on validation data from all revealed tasks.

Original: To demonstrate the online performance, we perform multiple evaluations during each task.
Revised: To demonstrate the online performance, we perform multiple evaluations during each task.

Original: Since clear task boundaries are available, we also compare against
Revised: Since clear task boundaries are available, we also compare against

Original: SGD+Cyclic, where the cosine learning rate schedule is used in each task.
Revised: SGD+Cyclic, where the cosine learning rate schedule is used in each task.

Original: We further compare against methods designed for handling storage limitations: GDumb (Prabhu et al., 2020), ER (Aljundi et al., 2018), and MIR (Aljundi et al., 2019a).
Revised: We further compare against methods designed for handling storage limitations: GDumb (Prabhu et al., 2020), ER (Aljundi et al., 2018), and MIR (Aljundi et al., 2019a).

Original: We explicitly limit the computation budget of ER, GDumb, and MIR to be similar to other methods.
Revised: We explicitly limit the computation budget of ER, GDumb, and MIR to be similar to other methods.

Original: See Appx.
Revised: See Appx.

Original: C.8 for more details about the experimental setup and hyperparameter settings.
Revised: C.8 for more details about the experimental setup and hyperparameter settings.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, AMA+MALR outperformed both SGD+RWP and SGD+Cyclic.
Revised: 5, AMA+MALR outperformed both SGD+RWP and SGD+Cyclic.

Original: Comparing to SGD+Cyclic, AMA+MALR had better intermediate and final performance during each task, making it more suitable for online applications.
Revised: Comparing to SGD+Cyclic, AMA+MALR had better intermediate and final performance during each task, making it more suitable for online applications.

Original: Using the same number of SGD steps, the final validation accuracy of AMA+MALR was close to supervised learning.
Revised: Using the same number of SGD steps, the final validation accuracy of AMA+MALR was close to supervised learning.

Original: On the other hand, ER, MIR, and GDumb performed worse than SGD+Cyclic since implicit computation increase in each model update step made them require more computation/time to converge.
Revised: On the other hand, ER, MIR, and GDumb performed worse than SGD+Cyclic since implicit computation increase in each model update step made them require more computation/time to converge.

Original: Note that here we allow the replay buffer to store all past data.
Revised: Note that here we allow the replay buffer to store all past data.

Original: This result demonstrates that limited computation and limited storage are orthogonal problems in OCL.
Revised: This result demonstrates that limited computation and limited storage are orthogonal problems in OCL.

Original: The are both important and require dedicated treatments.
Revised: The are both important and require dedicated treatments.

Original: See Appx.
Revised: See Appx.

Original: C.9 for similar results with limited storage.
Revised: C.9 for similar results with limited storage.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We studied the problem of improving information retention in online continual learning (OCL).
Revised: We studied the problem of improving information retention in online continual learning (OCL).

Original: We proposed to use pure replay as the objective for optimizing information retention.
Revised: We proposed to use pure replay as the objective for optimizing information retention.

Original: We provided theoretical analysis of the convergence of SGD in OCL.
Revised: We provided theoretical analysis of the convergence of SGD in OCL.

Original: We also proposed an Adaptive Moving Average (AMA) Optimizer and a Moving-Average-based Learning Rate Schedule (MALR) to optimize the pure replay objective online.
Revised: We also proposed an Adaptive Moving Average (AMA) Optimizer and a Moving-Average-based Learning Rate Schedule (MALR) to optimize the pure replay objective online.

Original: Experiments on several large-scale continual learning benchmarks demonstrate the effectiveness of AMA+MALR.
Revised: Experiments on several large-scale continual learning benchmarks demonstrate the effectiveness of AMA+MALR.

Original: In terms of limitations and future work, we used two MA models to adapt the MA weights, which increases the GPU memory footprint to store the network weights for MA models (but not the gradient information as in the SGD model).
Revised: In terms of limitations and future work, we used two MA models to adapt the MA weights, which increases the GPU memory footprint to store the network weights for MA models (but not the gradient information as in the SGD model).

Original: Designing more memory-efficient techniques to adapt MA weights is an interesting future research direction.
Revised: Designing more memory-efficient techniques to adapt MA weights is an interesting future research direction.

Original: Furthermore, this work mainly focused on studying the problem of limited computation in OCL.
Revised: Furthermore, this work mainly focused on studying the problem of limited computation in OCL.

Original: Developing effective solutions to address the computation and storage aspects together is an interesting and important future direction. 
Revised: Developing effective solutions to address the computation and storage aspects together is an interesting and important future direction. R

Original: A C ONVERGENCE A NALYSIS OF SGD
Revised: A C ONVERGENCE A NALYSIS OF SGD

Original: This section analyzes the convergence of SGD in both supervised learning (SL) and online continual learning (OCL).
Revised: This section analyzes the convergence of SGD in both supervised learning (SL) and online continual learning (OCL).

Original: The purpose of the analysis is to demonstrate the difference between the optimization problem in SL and OCL and explain intuitively why SGD+RWP works in SL but can fail in OCL.
Revised: The purpose of the analysis is to demonstrate the difference between the optimization problem in SL and OCL and explain intuitively why SGD+RWP works in SL but can fail in OCL

Original: A. C ONVERGENCE ANALYSIS IN S UPERVISED L EARNING
Revised: A. C ONVERGENCE ANALYSIS IN S UPERVISED L EARNING

Original: In supervised learning, we are given a fixed set of training data, and we solve the following stationary optimization problem
Revised: In supervised learning, we are given a fixed set of training data, and we solve the following stationary optimization problem

Original: [Equation]
Revised: [Equation]

Original: In each iteration k, SGD updates the model θ k using the stochastic gradient, from a batch of training data, via,
Revised: In each iteration k, SGD updates the model θ k using the stochastic gradient, from a batch of training data, via,

Original: [Equation]
Revised: [Equation]

Original: where E k denotes the random variable that determines the noise of the stochastic gradient.
Revised: where E k denotes the random variable that determines the noise of the stochastic gradient.

Original: The standard assumptions (Ghadimi & Lan, 2013) for analyzing the non-convex case of supervised learning is
Revised: The standard assumptions (Ghadimi & Lan, 2013) for analyzing the non-convex case of supervised learning is

Original: A5 Lipschitz Smoothness with constant L
Revised: A5 Lipschitz Smoothness with constant L

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: , where ∥·∥ is the Euclidean Norm.
Revised: , where ∥·∥ is the Euclidean Norm.

Original: A6 Unbiased gradient estimation: 
Revised: A6 Unbiased gradient estimation: E E

Original: [Equation]
Revised: [Equation]

Original: R d {+. A
+}
Revised: d {+. A
+}

Original: [Equation]

Original: A7 Bounded gradient noise with constant ρ ≥ 0 :
Revised: Bounded gradient noise with constant ρ ≥ 0 :

Original: ∥ ▽ l ( θ ) − g [Equation] θ 
Revised: ∥ ▽ l ( θ ) − g ( θ |E ) ∥

Original: [Equation] E ) ∥ 2 ≤ ρ [Equation] ∀ θ ∈ R d .
Revised: 2 ≤ ρ 2 , ∀ θ ∈ R d .

Original: Under these assumptions, we re-produce the standard convergence result of SGD in the non-convex
Revised: Under these assumptions, we re-produce the standard convergence result of SGD in the non-convex

Original: SL case:
Revised: SL case:

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume A5 to A7 , and let α k <
Revised: Assume A5 to A7 , and let α k <

Original: L 2 ,min
Revised: L 2 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Following the proof of Theorem 2.1 until (2.11) in (Ghadimi & Lan, 2013), we have k (cid:88) 
Revised: Following the proof of Theorem 2.1 until (2.11) in (Ghadimi & Lan, 2013), we have k (cid:88) j

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: And since (
Revised: And since (

Original: [Equation]
Revised: [Equation]

Original: we immediately have (12) by combining (13) with (14) and dividing both sides by (cid:80) kj =0 (2 α j +1 −
Revised: we immediately have (12) by combining (13) with (14) and dividing both sides by (cid:80) kj =0 (2 α j +1 −

Original: Lα 2 j +1 ) .
Revised: Lα 2 j +1 ) .

Original: Theorem 2 tells us that the convergence of SGD can be guaranteed by properly setting the learning rate α k .
Revised: Theorem 2 tells us that the convergence of SGD can be guaranteed by properly setting the learning rate α k .

Original: Denote T 4 = 2( l ( θ 0 ) − E E 1 ,..., E k +

Revised: Denote T 4 = 2( l ( θ 0 ) − E E 1 ,..., E k +

Original: [ l ( θ k +1 )])
Revised: [ l ( θ k +1 )

Original: (cid:80) kj

Original: [Equation]
Revised: [Equation]

Original: Lα 2 j +
Revised: ) (cid:80) kj

Original: [Equation]
Revised: [Equation]

Original: and T 5 =
Revised: and T 5 =

Original: [Equation]
Revised: [Equation]

Original: 1 − Lα 2 j
Revised: (2 α j

Original: [Equation]
Revised: [Equation]

Original: 1  ) as the first and the second terms at the right hand side of (12), which are of the same form as T 1 and T 2 of (7).
Revised: 1 − Lα 2 j +1 ) as the first and the second terms at the right hand side of (12), which are of the same form as T 1 and T 2 of (7).

Original: To ensure that T 4 goes to 0, we need to make sure that
Revised: To ensure that T 4 goes to 0, we need to make sure that

Original: [Equation]
Revised: [Equation]

Original: And to ensure that T 5 goes to 0, we need to ensure that
Revised: And to ensure that T 5 goes to 0, we need to ensure that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: For example, one can set α k
Revised: For example, one can set α k

Original: = O ( √ k ) so that both (15) and (16) are satisfied.
Revised: = O ( √ k ) so that both (15) and (16) are satisfied.

Original: Theorem 2 also provides insights of why RWP is effective in SL.
Revised: Theorem 2 also provides insights of why RWP is effective in SL.

Original: Given a constant learning rate α k
Revised: Given a constant learning rate α k

Original: = α , we can see that T 

Revised: = α , we can see that T

Original: =
Revised: =

Original: O ( 1 k ) after k iterations, i.e., it converges with linear speed.
Revised: O ( 1 k ) after k iterations, i.e., it converges with linear speed.

Original: However, the second term
Revised: However, the second term

Original: T 

Revised: T

Original: =
Revised: =

Original: Lρ 2 α 2 − Lα is a constant over time.
Revised: Lρ 2 α 2 − Lα is a constant over time.

Original: Hence, the gradient norm will first decrease and then converge to a constant given a constant learning rate.
Revised: Hence, the gradient norm will first decrease and then converge to a constant given a constant learning rate.

Original: And the plateau of the gradient norm is a signal that we need to reduce the learning rate to further improve the model.
Revised: And the plateau of the gradient norm is a signal that we need to reduce the learning rate to further improve the model.

Original: In practice, the validation loss/accuracy is often used as a surrogate of the gradient norm.
Revised: In practice, the validation loss/accuracy is often used as a surrogate of the gradient norm.

Original: A. C ONVERGENCE ANALYSIS IN OCL
Revised: A. C ONVERGENCE ANALYSIS IN OCL

Original: We provide a proof of Theorem 1 in this section.
Revised: We provide a proof of Theorem 1 in this section.

Original: Before the proof, we first re-state the problem, the assumptions and the theory in a more detailed format.
Revised: Before the proof, we first re-state the problem, the assumptions and the theory in a more detailed format.

Original: At each iteration k , we are given a non-stationary objective
Revised: At each iteration k , we are given a non-stationary objective

Original: [Equation]
Revised: [Equation]

Original: In each iteration k , SGD updates the model θ k using the stochastic gradient g k ( θ k − 1 |E k ) on l k ( θ k − 1 ) , from a batch of training data, via,
Revised: In each iteration k , SGD updates the model θ k using the stochastic gradient g k ( θ k − 1 |E k ) on l k ( θ k − 1 ) , from a batch of training data, via,

Original: [Equation]
Revised: [Equation]

Original: where E k denotes the random variable that decides the noise of the stochastic gradient.
Revised: where E k denotes the random variable that decides the noise of the stochastic gradient.

Original: It is unsurprising that no algorithm can achieve a reasonable performance if l k ( · ) can change arbitrarily between consecutive iterations, since we can change the training data in a completely adversarial fasion.
Revised: It is unsurprising that no algorithm can achieve a reasonable performance if l k ( · ) can change arbitrarily between consecutive iterations, since we can change the training data in a completely adversarial fasion.

Original: One reasonable assumption is that l k ( · ) does not change significantly between consecutive time steps.
Revised: One reasonable assumption is that l k ( · ) does not change significantly between consecutive time steps.

Original: For example, when we want to optimize information retention, the training data pool S t does not change significantly between consecutive iterations.
Revised: For example, when we want to optimize information retention, the training data pool S t does not change significantly between consecutive iterations.

Original: Given this intuition, we formalize our assumptions in the non-stationary case, with the definition of the mean operation E (which is omitted in the main paper) as follows:
Revised: Given this intuition, we formalize our assumptions in the non-stationary case, with the definition of the mean operation E (which is omitted in the main paper) as follows:

Original: A1 Lipschitz Smoothness with constant L ≥ 0 : ∥ ▽ l k ( θ ) − ▽ l k ( θ ′ )
Revised: [Equation]

Original: ∥ ≤ L ∥
Revised: ∥ ≤ L ∥

Original: θ − θ ′ ∥ , ∀ θ , θ ′ ∈ R d and k .
Revised: θ − θ ′ ∥ , ∀ θ , θ ′ ∈ R d and k .

Original: A2 Unbiased gradient estimation
Revised: A2 Unbiased gradient estimation:

Revised: E E [ g k ( θ |E )]

Revised: =

Revised: ▽ l k ( θ ) , ∀ θ ∈ R d and k .

Original: A
Revised: A

Revised: Bounded gradient noise

Revised: with constant ρ ≥ 0 : ∥ ▽ l k ( θ ) − g k ( θ |E ) ∥ 2 ≤ ρ

Original:  χ 
Revised: ∀ θ ∈ d and k . A Bounded non-stationarity with constants χ k ≥ 0 : | l k ( θ ) − l k +1 ( θ ) | ≤ χ k , ∀ θ ∈ R d and k .

Original: Intuitively, A1 to A3 extend A5 to A7 to non-stationary objectives, and A4 constraints the change of the objective function value between consecutive iterations, which is similar to the analysis of SGD in the convex case (Besbes et al., 2015).
Revised: Intuitively, A1 to A3 extend A5 to A7 to non-stationary objectives, and A4 constraints the change of the objective function value between consecutive iterations, which is similar to the analysis of SGD in the convex case (Besbes et al., 2015).

Original: Note that l k ( θ ) is computed over the replay data, i.e., not on the discarded historical data when the replay buffer/storage is limited.
Revised: Note that l k ( θ ) is computed over the replay data, i.e., not on the discarded historical data when the replay buffer/storage is limited.

Original: Hence the analysis in this section naturally includes limited storage as a special case.
Revised: Hence the analysis in this section naturally includes limited storage as a special case.

Original: Under these assumptions, we can prove the following convergence result of SGD in the non-stationary case:
Revised: Under these assumptions, we can prove the following convergence result of SGD in the non-stationary case:

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume A1 to A4 , and let α k <
Revised: Assume A1 to A4 , and let α k <

Original: L 2 ,
Revised: L 2 ,

Original: [Equation]
Revised: [Equation]

Original: (19) where T=
Revised: [Equation] where T=

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: With A1 and Lemma 1.2.3 of (Nesterov, 1998)
Revised: With A1 and Lemma 1.2.3 of (Nesterov, 1998)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where (26) is due to 
Revised: where (25) is due to A

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms in (27) and using A4 , we have
Revised: Rearranging the terms in (26) and using A4 , we have

Original: [Equation]
Revised: [Equation]

Original: Summing both sides of (30) from j = 0 to k 
Revised: Summing both sides of (29) from j = 0 to k , we have

Original: [Equation]
Revised: [Equation]

Original: we have k (cid:88)

Original: j

Original: = [Equation] [Equation] With a similar derivation as in the proof of Theorem 2, we immediately have (19).
Revised: With a similar derivation as in the proof of Theorem 2, we immediately have (19).

Original: Intuitively, to make T 3 converge to 0 , we need to set the learning rate such that
Revised: Intuitively, to make T 3 converge to 0 , we need to set the learning rate such that

Original: [Equation]
Revised: [Equation]

Original: i.e., we need (cid:80) kj =

Revised: i.e., we need (cid:80) kj =

Original: (2 α j +1 − Lα 2 j +1 ) to increase faster than (cid:80) kj =0 χ j +1 .
Revised: (2 α j +1 − Lα 2 j +1 ) to increase faster than (cid:80) kj =0 χ j +1 .

Original: In practice, the value of χ k depends on the application.
Revised: In practice, the value of χ k depends on the application.

Original: Given a constant learning rate α k
Revised: Given a constant learning rate α k

Original: = α , T 3 can increase for an increasing sequence of χ k .
Revised: = α , T 3 can increase for an increasing sequence of χ k .

Original: This explains why P3 can happen in practice.
Revised: This explains why P3 can happen in practice.

Original: If χ k ≥ χ for a constant χ , we cannot have a learning rate schedule that makes T 1 , T 2 and T 3 converge to 0 at the same time.
Revised: If χ k ≥ χ for a constant χ , we cannot have a learning rate schedule that makes T 1 , T 2 and T 3 converge to 0 at the same time.

Original: Moreover, when α k < 1 , decreasing α k will increase of T 3 .
Revised: Moreover, when α k < 1 , decreasing α k will increase of T 3 .

Original: In the extreme case, if α k is reduced to 0, T 3 will increase in linear speed, causing long term failure of SGD+RWP.
Revised: In the extreme case, if α k is reduced to 0, T 3 will increase in linear speed, causing long term failure of SGD+RWP.

Original: This explains P2 , i.e., why reducing the learning rate to 0 using RWP hurts the performance of OCL.
Revised: This explains P2 , i.e., why reducing the learning rate to 0 using RWP hurts the performance of OCL.

Original: And demonstrates why we need to keep the learning rate sufficiently large, i.e., P1 .
Revised: And demonstrates why we need to keep the learning rate sufficiently large, i.e., P1 .

Original: Note that currently, there is no lower bound to demonstrate the tightness of (19).
Revised: Note that currently, there is no lower bound to demonstrate the tightness of (19).

Original: Hence, the analysis in this section can only be viewed as an intuition.
Revised: Hence, the analysis in this section can only be viewed as an intuition.

Original: Interpreting Theorem 1 under limited storage In practice, both supervised learning and continual learning use a limited amount of (though may be in a large scale) training/replay data.
Revised: Interpreting Theorem 1 under limited storage In practice, both supervised learning and continual learning use a limited amount of (though may be in a large scale) training/replay data.

Original: In such cases, the loss ( l k ( θ ) in (17) and l ( θ ) (10)) is only computed over the stored data.
Revised: In such cases, the loss ( l k ( θ ) in (17) and l ( θ ) (10)) is only computed over the stored data.

Original: And our assumptions in the theories are only w.r.t.
Revised: And our assumptions in the theories are only w.r.t.

Original: the loss over the training/replay data.
Revised: the loss over the training/replay data.

Original: For supervised learning, each training data point can be viewed as a random sample from the original data distribution.
Revised: For supervised learning, each training data point can be viewed as a random sample from the original data distribution.

Original: For continual learning where not all past data can be stored, one can construct a reservoir replay buffer (Chaudhry et al., 2019) online, so that at each time step t , each data point in the replay buffer is also a random sample from the original data distribution.
Revised: For continual learning where not all past data can be stored, one can construct a reservoir replay buffer (Chaudhry et al., 2019) online, so that at each time step t , each data point in the replay buffer is also a random sample from the original data distribution.

Original: Hence, the bound in Theorem 1 is directly comparable to the bound in Theorem 2 when the replay buffer size in OCL is the same as the dataset size of supervised learning.
Revised: Hence, the bound in Theorem 1 is directly comparable to the bound in Theorem 2 when the replay buffer size in OCL is the same as the dataset size of supervised learning.

Original: The performance of a (continual) learner over the original data distribution is the problem of generalization, which is out of the scope of convergence analysis.
Revised: The performance of a (continual) learner over the original data distribution is the problem of generalization, which is out of the scope of convergence analysis.

Original: B F URTHER D ETAILS OF AMA AND O NLINE V ALIDATIONAlgorithm 2 Adaptive Moving Average (AMA)
Revised: B F URTHER D ETAILS OF AMA AND O NLINE V ALIDATION

Original: Require: Initial model θ 0 , the initial moving average weight γ 0 , MA weight adjustment coefficient δ ( > 0) , model update interval K M , validation interval K V , MA weight update interval K W , number of training iterations per time step p . 
Revised: Require: Initial model θ 0 , the initial moving average weight γ 0 , MA weight adjustment coefficient δ ( > 0) , model update interval K M , validation interval K V , MA weight update interval K W , number of training iterations per time step p . γ MA MA

Original:  do {+4: ) 5: if k mod K M = 
+}
Revised: : Acc : for k , ∞} do {+4: ) 5: if k mod K M = 
+}

Original:  8: end if 9: if  then 10:
Revised: then 6: θ ← 7: θ θ k 8: end if 9: if k mod K V = 0 then 10:

Original: Acc  n · Acc 1 + acc ( B V ⌊ kp ⌋ , θ MA 1 ) n + , Acc 2 ← n · Acc 2 + acc ( B V ⌊ kp ⌋ , θ MA 2 ) n + , n ← n +11:
Revised: Acc 1 ← n · Acc 1 + acc ( B V ⌊ kp ⌋ , θ MA 1 ) n + , Acc 2 ← n · Acc 2 + acc ( B V ⌊ kp ⌋ , θ MA 2 ) n + , n ← n +11:

Original: If Acc  > Acc 2 then i best = 1 else i best =12: end if 13: if k mod K W = 0 then 14:
Revised: If Acc 1 > Acc 2 then i best = 1 else i best =12: end if 13: if k mod K W = 0 then 14:

Original: Acc 1 ← 0 , Acc 2 ← 0 , n ←15: if i best = 1 then ▷ Increase both MA weights but limit them to be ≤16: γ MA 1 ← min(1 , δγ MA 1 ) , γ MA 2 ← γ MA 1 δ 17: θ MA 2 ← θ MA 1 , i best =18: else ▷ Decrease both MA weights 19: γ MA 1 ← γ MA 1 δ , γ MA 2 ← γ MA 2 δ 20: θ MA 1 ← θ MA 2 , i best =21: end if 22: end if 23: end for
Revised: Acc 1 ← 0 , Acc 2 ← 0 , n ←15: if i best = 1 then ▷ Increase both MA weights but limit them to be ≤16: γ MA 1 ← min(1 , δγ MA 1 ) , γ MA 2 ← γ MA 1 δ 17: θ MA 2 ← θ MA 1 , i best =18: else ▷ Decrease both MA weights 19: γ MA 1 ← γ MA 1 δ , γ MA 2 ← γ MA 2 δ 20: θ MA 1 ← θ MA 2 , i best =21: end if 22: end if 23: end for

Original: Alg. 2 describes the complete version of Alg. 1, with overhead reduction strategies added.
Revised: Alg. 2 describes the complete version of Alg. 1, with overhead reduction strategies added.

Original: Each online validation step (Line 9 to Line 12) computes the accuracy on a batch of data B V ⌊ kp ⌋ sampled from the information retention validation set S V
Revised: Each online validation step (Line 9 to Line 12) computes the accuracy on a batch of data B V ⌊ kp ⌋ sampled from the information retention validation set S V

Original: ⌊ kp ⌋ of the current training iteration k , where p is the number of SGD updates allowed per time step.
Revised: ⌊ kp ⌋ of the current training iteration k , where p is the number of SGD updates allowed per time step.

Original: We average the computed accuracy on the fly between each K W iterations, and use the averaged accuracy as the online information retention validation accuracy.
Revised: We average the computed accuracy on the fly between each K W iterations, and use the averaged accuracy as the online information retention validation accuracy.

Original: The benefit of this online validation process is that the validation steps can be parallelized completely with model updates, and we can also select the best MA model for inference anytime during OCL.
Revised: The benefit of this online validation process is that the validation steps can be parallelized completely with model updates, and we can also select the best MA model for inference anytime during OCL.

Original: C F URTHER E XPERIMENTS C. BN VS GN+WS
Revised: C F URTHER E XPERIMENTS C. BN VS GN+WS

Original: As mentioned in Sec.
Revised: [Figure] a t i o n R e t e n t i o n ( % )

Original: 4.2, we replace BN with GN+WS to remove the overhead of re-computing BN statistics in AMA.
Revised: GN+WS [Figure] [Figure] [Figure]

Original: To show the effect of changing the normalization layer, we compare the model
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: As mentioned in Sec. 4.

Original: [Figure]
Revised: [Figure]

Revised: we replace BN with GN+WS to remove the overhead of re-computing BN statistics in AMA.

Revised: To show the effect of changing the normalization layer, we compare the model trained with BN against the one trained with GN+WS.

Original:  resource limitations, we only train the
Revised: Due to resource limitations, we only train the

Original: BN model for 700 thousand iterations.
Revised: BN model for 700 thousand iterations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, GN+WS performed slightly better than
Revised: 6, GN+WS performed slightly better than

Original: BN.
Revised: BN.

Original: Combined with the efficiency, GN+WS is suitable for information retention in OCL.
Revised: Combined with the efficiency, GN+WS is suitable for information retention in OCL.

Original: C. S MALL L EARNING R ATES S TYMIE A DAPTATION TO N EW D ATA
Revised: C. S MALL L EARNING R ATES S TYMIE A DAPTATION TO N EW D ATA

Original: To understand the effect of using suboptimal learning rates on data from different ranges, we take the models of different methods at two distinct training iterations ( k = 5 .
Revised: To understand the effect of using suboptimal learning rates on data from different ranges, we take the models of different methods at two distinct training iterations ( k = 5 .

Original: 76e5 and k = 4 . 8e8 ), and plot their accuracy on evaluation data from different time ranges.
Revised: 76e5 and k = 4 . 8e8 ), and plot their accuracy on evaluation data from different time ranges.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, AMA+CLR performed worse than AMA+MALR on all data ranges due to the use of a large learning rate.
Revised: 7, AMA+CLR performed worse than AMA+MALR on all data ranges due to the use of a large learning rate.

Original: The performance gap was larger on historical data than on future data.
Revised: The performance gap was larger on historical data than on future data.

Original: SGD+RWP and AMA+RWP performed well only on early data and failed to improve the performance for new data when the learning rate was too low.
Revised: SGD+RWP and AMA+RWP performed well only on early data and failed to improve the performance for new data when the learning rate was too low.

Original: C. H EAVY PARALLELISM H URT OCL
Revised: C. H EAVY PARALLELISM H URT OCL

Original: Cai et al.
Revised: Cai et al.

Original: (2021) have shown that increasing batch sizes for optimizing learning efficacy hurts all
Revised: (2021) have shown that increasing batch sizes for optimizing learning efficacy hurts all

Original: OCL performance metrics.
Revised: OCL performance metrics.

Original: We perform a similar analysis to study whether this phenomenon still happens when optimizing information retention.
Revised: We perform a similar analysis to study whether this phenomenon still happens when optimizing information retention.

Original: Specifically, we optimize information retention with batch sizes varying from 64 to 1024.
Revised: Specifically, we optimize information retention with batch sizes varying from 64 to 1024.

Original: When we increase the batch size, we reduce the number of training iterations per time step and increase the learning rate by the same factor.
Revised: When we increase the batch size, we reduce the number of training iterations per time step and increase the learning rate by the same factor.

Original: We allowtraining iteration per time step for a batch size of 256.
Revised: We allowtraining iteration per time step for a batch size of 256.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8, increasing batch sizes for optimizing information retention also hurt all OCL performance metrics.
Revised: 8, increasing batch sizes for optimizing information retention also hurt all OCL performance metrics.

Original: C. A R EASONABLY L ARGE R EPLAY B UFFER I S S UFFICIENT
Revised: C. A R EASONABLY L ARGE R EPLAY B UFFER I S S UFFICIENT

Original: We now analyze the effect of replay buffer sizes, by training models with replay buffer sizes ofthousand, 4 million, and 40 million images.
Revised: We now analyze the effect of replay buffer sizes, by training models with replay buffer sizes ofthousand, 4 million, and 40 million images.

Original: To optimize information retention, we use a reservoirbuffer (Chaudhry et al., 2019) in this experiment.
Revised: To optimize information retention, we use a reservoir

Original: We allow 5 training iterations per time step for each replay buffer size.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F o r w a r d T r a n s f e r
Revised: buffer (Chaudhry et al., 2019) in this experiment.

Original: [Figure]
Revised: We allow 5 training iterations per time step for each replay buffer size.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 9, reducing the replay buffer size from 40 million to 400 thousand images significantly hurt all performance metrics.
Revised: 9, reducing the replay buffer size from 40 million to 400 thousand images significantly hurt all performance metrics.

Original: However, the performance remained similar when the replay buffer size was 4 million or larger.
Revised: However, the performance remained similar when the replay buffer size was 4 million or larger.

Original: Therefore, given a fixed model and computational budget, OCL only requires a reasonably large replay buffer.
Revised: Therefore, given a fixed model and computational budget, OCL only requires a reasonably large replay buffer.

Original: C. A PPLY AMA TO O THER B ASE O PTIMIZERS
Revised: C. A PPLY AMA TO O THER B ASE O PTIMIZERS

Original: Besides SGD, AMA+MALR can also be combined with other base optimizers.
Revised: Besides SGD, AMA+MALR can also be combined with other base optimizers.

Original: As an example, we apply it to ADAM and test the performance on CLOC.
Revised: As an example, we apply it to ADAM and test the performance on CLOC.

Original: Specifically, we compare (ADAM-based)
Revised: Specifically, we compare (ADAM-based)

Original: AMA+MALR with ADAM+MALR and ADAM+RWP, with an initial learning rate tuned to 0.001.
Revised: AMA+MALR with ADAM+MALR and ADAM+RWP, with an initial learning rate tuned to 0.001.

Original: Due to resource limitations, we only run all algorithms on partial data.
Revised: Due to resource limitations, we only run all algorithms on partial data.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 10, (ADAM)
Revised: 10, (ADAM)

Original: AMA+MALR outperformed ADAM+RWP.
Revised: AMA+MALR outperformed ADAM+RWP.

Original: However, ADAM methods performed worse than SGD.
Revised: However, ADAM methods performed worse than SGD.

Original: We conjecture that the adaptive scaling in ADAM reduced the effective learning rate (similar to the
Revised: We conjecture that the adaptive scaling in ADAM reduced the effective learning rate (similar to the

Original: SGD+RWP case), and leave further investigations as future works.
Revised: SGD+RWP case), and leave further investigations as future works.

Original: Due to the poor performance of
Revised: Due to the poor performance of

Original: ADAM-based methods
Revised: ADAM-based methods

Original: , we use SGD-based methods in the main experiments.
Revised: , we use SGD-based methods in the main experiments.

Original: C. I MPACT OF O PTIMIZATION O BJECTIVES TO OCL
Revised: C. I MPACT OF O PTIMIZATION O BJECTIVES TO OCL

Original: An ideal OCL algorithm should perform well for all three metrics, i.e., learning efficacy (eq.
Revised: An ideal OCL algorithm should perform well for all three metrics, i.e., learning efficacy (eq.

Original: (1)), information retention (eq.
Revised: (1)), information retention (eq.

Original: (2)), and forward transfer (eq.
Revised: (2)), and forward transfer (eq.

Original: However, optimizing one objective can hurt others.
Revised: However, optimizing one objective can hurt others.

Original: To demonstrate this point, we compare the model trained to optimize information retention (with AMA+MALR), and the model trained to optimize learning efficacy (with the method of Cai et al.
Revised: To demonstrate this point, we compare the model trained to optimize information retention (with AMA+MALR), and the model trained to optimize learning efficacy (with the method of Cai et al.

Original: Following Cai et al.
Revised: Following Cai et al.

Original: (2021), we allow different computation budgets (1 andtraining iterations) per time step.
Revised: (2021), we allow different computation budgets (1 andtraining iterations) per time step.

Original: To fairly compare pure replay and mixed replay, we double the batch size for AMA+MALR, so that the average number of gradient descent steps per image is the same for the two models.
Revised: To fairly compare pure replay and mixed replay, we double the batch size for AMA+MALR, so that the average number of gradient descent steps per image is the same for the two models.

Original: We set k 1 and k 2 in the forward transfer metric to 10% and 25% of the dataset size respectively.
Revised: We set k 1 and k 2 in the forward transfer metric to 10% and 25% of the dataset size respectively.

Original: As shown in Fig. 11a, given the same budget, optimizing learning efficacy (LE Model) performed the best for learning efficacy.
Revised: As shown in Fig. 11a, given the same budget, optimizing learning efficacy (LE Model) performed the best for learning efficacy.

Original: Optimizing information retention (IR Model) performed the best for information retention (Fig. 11b) and forward transfer (Fig. 11c).
Revised: Optimizing information retention (IR Model) performed the best for information retention (Fig. 11b) and forward transfer (Fig. 11c).

Original: More importantly, unlike optimizinginformation retention, where increasing the computation budget improved all OCL metrics, increasing the computation budget for optimizing learning efficacy hurt information retention and forward transfer. This phenomenon is consistent with the finding in the language modeling setting (Hu et al.,
Revised: More importantly, unlike optimizing

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) vs optimizing information retention (IR

Original: Model).

Original: Optimizing information retention hurt learning efficacy (a), but improved information retention (b) and forward transfer (c).
Revised: Optimizing information retention hurt learning efficacy (a), but improved information retention (b) and forward transfer (c).

Original: More importantly, increasing the budget for optimizing learning efficacy hurt both information retention and forward transfer.
Revised: More importantly, increasing the budget for optimizing learning efficacy hurt both information retention and forward transfer.

Original: In contrast, increasing the budget for optimizing information retention improved all OCL metrics.
Revised: In contrast, increasing the budget for optimizing information retention improved all OCL metrics.

Revised: information retention, where increasing the computation budget improved all OCL metrics, increasing the computation budget for optimizing learning efficacy hurt information retention and forward transfer.

Revised: This phenomenon is consistent with the finding in the language modeling setting (Hu et al., 2020).

Original: Hence in terms of improving long term transfer in OCL, information retention is a better objective than learning efficacy.
Revised: Hence in terms of improving long term transfer in OCL, information retention is a better objective than learning efficacy.

Original: C. D ETAILS ON G OOGLE L ANDMARKS V
Revised: C. D ETAILS ON G OOGLE L ANDMARKS V

Original: Similar to CLOC, we construct the OCL version of Google Landmarks v2 using the subset of images from Flickrs, which have time stamps that can be retrieved from flickr API.
Revised: Similar to CLOC, we construct the OCL version of Google Landmarks v2 using the subset of images from Flickrs, which have time stamps that can be retrieved from flickr API.

Original: Similar to the original paper (Weyand et al., 2020), we address the noisy label and class imbalance issues by using only the data from the cleaned set, and filter out the classes with fewer than 25 images.
Revised: Similar to the original paper (Weyand et al., 2020), we address the noisy label and class imbalance issues by using only the data from the cleaned set, and filter out the classes with fewer than 25 images.

Original: This results in 645666 images in total.
Revised: This results in 645666 images in total.

Original: We further divide the dataset into 90% of training data (roughly 580 thousand images) and 10% of evaluation data (roughly 65 thousand images).
Revised: We further divide the dataset into 90% of training data (roughly 580 thousand images) and 10% of evaluation data (roughly 65 thousand images).

Original: In terms of hyper-parameters, we use the same model as the CLOC dataset.
Revised: In terms of hyper-parameters, we use the same model as the CLOC dataset.

Original: We tune other hyperparameters on the first 20% of the training data (we still use this part of data during training because otherwise the dataset will be even smaller), resulting in the learning rate of 0.135, the momentum of 0.9 for SGD, the weight decay of 1e-4, and the cross-entropy loss.
Revised: We tune other hyperparameters on the first 20% of the training data (we still use this part of data during training because otherwise the dataset will be even smaller), resulting in the learning rate of 0.135, the momentum of 0.9 for SGD, the weight decay of 1e-4, and the cross-entropy loss.

Original: C. D ETAILS ON I MAGE N ET C ONTINUAL L EARNING E XPERIMENTS
Revised: C. D ETAILS ON I MAGE N ET C ONTINUAL L EARNING E XPERIMENTS

Original: We mostly follow the hyper-parameter settings of supervised learning, i.e., we use ResNet50, with batch size of 256, initial learning rate of 0.1, the momentum of 0.9 for SGD, the weight decay of 1e-4.
Revised: We mostly follow the hyper-parameter settings of supervised learning, i.e., we use ResNet50, with batch size of 256, initial learning rate of 0.1, the momentum of 0.9 for SGD, the weight decay of 1e-4.

Original: For ER, GDumb and MIR, we use the same basic hyper-parameter as SGD+Cyclic, including the optimizer, learning rates, the batch size etc.
Revised: For ER, GDumb and MIR, we use the same basic hyper-parameter as SGD+Cyclic, including the optimizer, learning rates, the batch size etc.

Original: For ER, we mix 128 samples from the current task and 128 samples from replay for each model udpate step.
Revised: For ER, we mix 128 samples from the current task and 128 samples from replay for each model udpate step.

Original: For MIR, we select 128 examples fromcandidates (using the s MI − 1 criterion) during the replay data sampling procedure, and mix them with another 128 examples from the current task to update the model.
Revised: For MIR, we select 128 examples fromcandidates (using the s MI − 1 criterion) during the replay data sampling procedure, and mix them with another 128 examples from the current task to update the model.

Original: For a fair comparison, we limit the computation of each method to be the same as SGD+Cyclic.
Revised: For a fair comparison, we limit the computation of each method to be the same as SGD+Cyclic.

Original: Specifically, MIR requires at least two
Revised: Specifically, MIR requires at least two

Original: SGD iterations in each model update step, hence, the total number of model update steps is limited to  half of the SGD steps in SGD+Cyclic.
Revised: SGD iterations in each model update step, hence, the total number of model update steps is limited to [Figure] [Figure] half of the SGD steps in SGD+Cyclic.

Original: GDumb trains a model from scratch for each task, hence the number of SGD updates of GDumb and SGD+Cyclic is limit to be the same in each task.
Revised: GDumb trains a model from scratch for each task, hence the number of SGD updates of GDumb and SGD+Cyclic is limit to be the same in each task.

Original: To make the final validation accuracy directly comparable to the standard ResNet50 supervised learning result,[Figure] [Figure] we use BN as the normalization layer in this experiment (The GN+WS model performed slightly better than the BN model).
Revised: To make the final validation accuracy directly comparable to the standard ResNet50 supervised learning result, we use BN as the normalization layer in this experiment (The GN+WS model performed slightly better than the BN model).

Original: To update BN statistics on the fly, we do one extra forward pass pertraining iterations to accumulate the BN statistics for MA models.
Revised: To update BN statistics on the fly, we do one extra forward pass pertraining iterations to accumulate the BN statistics for MA models.

Original: For both RWP and AMA, we decay the learning rate by half when the corresponding conditions are satisfied.
Revised: For both RWP and AMA, we decay the learning rate by half when the corresponding conditions are satisfied.

Original: We set K R in RWP and AMA to 15000, which is smaller than in CLOC, so that the effect of learning rate reduction can be clearly seen.
Revised: We set K R in RWP and AMA to 15000, which is smaller than in CLOC, so that the effect of learning rate reduction can be clearly seen.

Original: We also show in Fig.
Revised: We also show in Fig.

Original: 12 the performance comparison between the BN model and the GN+WS model.
Revised: 12 the performance comparison between the BN model and the GN+WS model.

Original: Similar as in CLOC, GN+WS also performed slightly better than BN.
Revised: Similar as in CLOC, GN+WS also performed slightly better than BN.

Original: And its best validation accuracy in the final task ( 76 . 8% ) is similar as the standard BN model trained with supervised learning.
Revised: And its best validation accuracy in the final task ( 76 . 8% ) is similar as the standard BN model trained with supervised learning.

Original: This result confirmed the generalization of our method across problems and datasets.
Revised: This result confirmed the generalization of our method across problems and datasets.

Original: C. S MALL REPLAY BUFFER ON I MAGE N ET
Revised: C. S MALL REPLAY BUFFER ON I MAGE N ET

Original: In this section, we show the ImageNet continual learning result with limited storage.
Revised: In this section, we show the ImageNet continual learning result with limited storage.

Original: Specifically, we only allow 120K images to be stored in the replay buffer, and compare AMA+MALR against
Revised: Specifically, we only allow 120K images to be stored in the replay buffer, and compare AMA+MALR against

Original: SGD+Cyclic, ER, MIR and GDumb.
Revised: SGD+Cyclic, ER, MIR and GDumb.

Original: For AMA+MALR, SGD+Cyclic and GDumb, the replay buffer is formed using reservoir sampling.
Revised: For AMA+MALR, SGD+Cyclic and GDumb, the replay buffer is formed using reservoir sampling.

Original: For ER and MIR, we assign half of the replay buffer for storing data from previous tasks (using reservoir sampling), and half of the replay buffer for storing data from the current task.
Revised: For ER and MIR, we assign half of the replay buffer for storing data from previous tasks (using reservoir sampling), and half of the replay buffer for storing data from the current task.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 15, AMA+MALR still outperformed all competitors in this case.
Revised: 15, AMA+MALR still outperformed all competitors in this case.

Original: Interestingly, the performance gap between SGD+Cyclic and ER/MIR became smaller compared to the case of large replay buffers.
Revised: Interestingly, the performance gap between SGD+Cyclic and ER/MIR became smaller compared to the case of large replay buffers.

Original: C. P ERFORMANCE ON S UPERVISED L EARNING
Revised: C. P ERFORMANCE ON S UPERVISED L EARNING

Original: To understand the performance of AMA + MALR on supervised learning, we apply it to ImageNet, and compare against SGD + cosine schedule (denoted as SGD+Cosine), which is the standard for ImageNet.
Revised: To understand the performance of AMA + MALR on supervised learning, we apply it to ImageNet, and compare against SGD + cosine schedule (denoted as SGD+Cosine), which is the standard for ImageNet.

Original: To understand the effect of each component, we also show the performance of
Revised: To understand the effect of each component, we also show the performance of

Original: AMA+Cosine and the SGD model in AMA+MALR.
Revised: AMA+Cosine and the SGD model in AMA+MALR.

Original: To replicate the standard ResNet50 result on
Revised: To replicate the standard ResNet50 result on

Original: ImageNet, we use BN as the normalization layer, and train all models for 120 epochs. 
Revised: ImageNet, we use BN as the normalization layer, and train all models for 120 epochs. Since limiting [Figure] [Figure]

Revised: ↑ )

Original: Since limiting the minimum learning rate hurts supervised learning, we only use C1 and C2 in MALR for this experiment.
Revised: Due to the small number of training iterations in total, allowing 1 training iteration per time step does not reveal the problem of SGD+RWP in the long term, as in the main experiments. And the advantage of AMA+MALR is also invisible since MA needs a reasonable number of iterations to track the SGD trajectory. The learning rate of SGD+RWP and AMA+MALR remained constant throughout training, making this experiment not proper to validate Theorem 7 and demonstrate the complete behavior of different algorithms in large scale OCL. the minimum learning rate hurts supervised learning, we only use C1 and C2 in MALR for this experiment.

Original: We reduce the learning rate by 10 times once C1 and C2 are true.[Figure] [Figure]
Revised: We reduce the learning rate by 10 times once C1 and C2 are true.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 14, though not better in terms of the final performance, AMA performed much better than SGD at early training stages, where the learning rate was large.
Revised: 14, though not better in terms of the final performance, AMA performed much better than SGD at early training stages, where the learning rate was large.

Original: MALR adaptively reduced the learning rate over time, providing a similar final performance as the cosine schedule.
Revised: MALR adaptively reduced the learning rate over time, providing a similar final performance as the cosine schedule.

Original: C. AMA HYPER - PARAMETER TUNING AND SENSITIVITY
Revised: C. AMA HYPER - PARAMETER TUNING AND SENSITIVITY

Original: As described in Sec.4.
Revised: As described in Sec.

Original:  we set the MA weight adjustment coefficient δ to 5 and the interval K W to adjust MA weights to 10000 iterations.
Revised: 4.2, we set the MA weight adjustment coefficient δ to 5 and the interval K W to adjust MA weights to 10000 iterations.

Original: We empirically observed that setting K W in between 1-10000, and δ between 2-10 does not have strong effect to the performance (i.e., the classification accuracy).
Revised: We empirically observed that setting K W in between 1-10000, and δ between 2-10 does not have strong effect to the performance (i.e., the classification accuracy).

Original: Setting K W to 10000 and δ to 5 worked across problems (ImageNet, CLOC, Google
Revised: Setting K W to 10000 and δ to 5 worked across problems (ImageNet, CLOC, Google

Original: Landmarks) and provides AMA a similar speed as SGD in practice.
Revised: Landmarks) and provides AMA a similar speed as SGD in practice.

Original: For classification tasks, we set ϵ in C3 to 3% accuracy.
Revised: For classification tasks, we set ϵ in C3 to 3% accuracy.

Original: This value worked for CLOC, ImageNet and
Revised: This value worked for CLOC, ImageNet and

Original: CGLM (only tuned on the CLOC preprocessing data).
Revised: CGLM (only tuned on the CLOC preprocessing data).

Original: It can be used as a reasonable default for other datasets as well.
Revised: It can be used as a reasonable default for other datasets as well.

Original: For tasks other than classification, one can tune ϵ by:
Revised: For tasks other than classification, one can tune ϵ by:

Original: Finding the best initial learning rate on preprocessing data.
Revised: Finding the best initial learning rate on preprocessing data.

Original: Using a constant learning rate schedule, with learning rate set to the best initial learning rate.
Revised: Using a constant learning rate schedule, with learning rate set to the best initial learning rate.

Original: Compute the performance gap between AMA and SGD models on pre-processing data, denoted as g. 3.
Revised: Compute the performance gap between AMA and SGD models on pre-processing data, denoted as g. 3.

Original: Setting ϵ to roughly 0.3g (this is how we set our ϵ ), we observe that the performance difference is small by setting ϵ in between 0.5-0.2g.
Revised: Setting ϵ to roughly 0.3g (this is how we set our ϵ ), we observe that the performance difference is small by setting ϵ in between 0.5-0.2g.

Original: C. W HY NOT SMALLER DATASETS LIKE CIFAR OR 1 TRAINING ITERATION PER TIME STEP ?
Revised: C. W HY NOT SMALLER DATASETS LIKE CIFAR OR 1 TRAINING ITERATION PER TIME STEP ?