Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning
Abstract
Online continual learning (OCL) aims to train neural networks incrementally from a non-stationary data stream with a single pass through data.
Rehearsal-based methods attempt to approximate the observed input distributions over time with a small memory and revisit them later to avoid forgetting.
Despite their strong empirical performance, rehearsal methods still suffer from a poor approximation of past data’s loss landscape with memory samples.
This paper revisits the rehearsal dynamics in online settings.
We provide theoretical insights on the inherent memory overfitting risk from the viewpoint of biased and dynamic empirical risk minimization, and examine the merits and limits of repeated rehearsal.
Inspired by our analysis, a simple and intuitive baseline, repeated augmented rehearsal (RAR), is designed to address the underfitting-overfitting dilemma of online rehearsal.
Surprisingly, across four rather different OCL benchmarks, this simple baseline outperforms vanilla rehearsal by 9%-17% and also significantly improves the state-of-the-art rehearsal-based methods MIR, ASER, and SCR.
We also demonstrate that RAR successfully achieves an accurate approximation of the loss landscape of past data and high-loss ridge aversion in its learning trajectory.
Extensive ablation studies are conducted to study the interplay between repeated and augmented rehearsal, and reinforcement learning (RL) is applied to dynamically adjust the hyperparameters of RAR to balance the stability-plasticity trade-off online.
Introduction
Despite its recent success, deep learning largely relies on the assumption of independent and identically distributed (i.i.d.) data that can be repeatedly revisited during training.
Non-i.i.d settings are challenging for neural networks due to catastrophic forgetting: previously learned knowledge can easily be overwritten when training on new data because this data may follow a different distribution [Li and Hoiem, 2017, Rebuffi et al., 2017, Delange et al., 2021].
Online continual learning (OCL or Online CL) studies how to enable deep learning in an online manner from a non-stationary data stream.
As the data stream can be vast or even infinite, it is infeasible to store and shuffle the dataset for multiple epochs of training.
Therefore, a fundamental assumption is that the data stream can only be accessed one batch at a time and training is performed with a single pass over the data [Aljundi et al., 2019].
Experience replay (ER), also known as rehearsal [Chaudhry et al., 2019, Delange et al., 2021], is a key method in OCL.
It stores a subset of previously seen data D in a fixed-size memory M and revisits the memorized samples during training to mitigate forgetting of previous tasks.
To update the model, a batch sampled from the memory is combined with the incoming batch from the stream to compute the gradient [Chaudhry et al., 2019].
Different variants of ER have been developed to improve memory sampling policies and representation learning, achieving state-of-the-art performance in a number of standard OCL benchmarks [Aljundi et al., 2019, Mai et al., 2021, Shim et al., 2021].
However, whether rehearsal is appropriate for continual learning, considering the risk of overfitting the memory when using data from the memory to directly contribute to the gradient computation, has been debated vigorously [Lopez-Paz and Ranzato, 2017, Chaudhry et al., 2019, Verwimp et al., 2021].
The potential for overfitting has motivated the development of constraint-based replay methods [LopezPaz and Ranzato, 2017], e.g., GEM and A-GEM, which use memory samples solely to constrain the gradient direction.
However, there is empirical evidence suggesting that rehearsal-based methods consistently outperform methods that do not train directly on the memory [Chaudhry et al., 2019].
This indicates rehearsal on the memory does not necessarily prevent effective generalization, possibly due to the implicit regularization effect of incoming data [Chaudhry et al., 2019].
Nevertheless, recent work analyzing the loss landscape when applying rehearsal to offline continual learning finds that memory samples indeed provide a poor approximation of the loss landscape of past tasks, especially near a high-loss ridge.
As a result, “instead of ending up near the high-loss ridge in perspective of the rehearsal memory, the solution in reality resides on the high-loss ridge for the training data” [Verwimp et al., 2021].
This latest finding poses the question of how to better approximate the loss surface of past data L ( D ; θ ) with memory samples’ loss L ( M ; θ ) .
To better approximate past data’s loss surface, previous work studies which samples should be memorized for rehearsal.
Instead, we study how to effectively perform rehearsal with the memorized samples.
Focused on online CL, our study extends the previous understanding of rehearsal along two directions.
First, we provide theoretical considerations that reveal two insights regarding the extent of overfitting to memory: it is a) related to the inherent attributes of the OCL problem concerned and b) varies across the different stages of continual learning.
Second, we highlight the limits of applying rehearsal with multiple iterations—a trick used to maximally utilize the incoming batch [Aljundi et al., 2019]—and identify an underfitting-overfitting dilemma for online rehearsal.
Based on our analysis, we design a simple baseline to deal with the underfitting-overfitting dilemma in online CL problems, dubbed repeated augmented rehearsal (RAR), that can be easily integrated into existing rehearsal-based methods.
Surprisingly, this simple baseline leads to a large performance boost for ER, as well as state-of-the-art ER-based approaches, across four OCL benchmarks.
More importantly, the loss landscape analysis shows that RAR can help memory samples reliably approximate the distribution of past data and successfully avoids the high-loss ridge of past tasks.
To better understand the behavior of RAR, we further investigate the interplay between repeated rehearsal and augmented rehearsal via an ablation study.
We also propose a reinforcement learning-based method to dynamically adjust the hyperparameters of RAR and balance the stability-plasticity trade-off in an online manner.
Related Work
Online continual learning : We consider the online continual learning setting with a non-stationary (potentially infinite) stream of data: at each time step t , the continual learning agent receives an incoming batch of data samples B t = { x i , y i } i =1 ,.., |B t | that are drawn from the current data distribution D t .
The period of time where the data distribution stays the same is often called a task or experience in the continual learning literature.
Abrupt change in the data distribution occurs when the task changes.
The standard objective during training is to minimize the empirical risk on all the data seen so far:
[Equation]
with loss function L , the CL network function f , and its associated parameters θ .
Metrics :
A common metric is the end accuracy after training on T tasks,
defined
as A T = 1 T (cid:80) j = T j =1 a T,j , where a i,j denotes the model’s accuracy on the
held
out test set of task
j after training on task i [-=
-]. Other metrics
are “forgetting”
[Chaudhry et al
, 2018], which is defined as
[Figure]
F T = [Equation] [Equation] i
[Equation]
i ) and the related metric “backward transfer” [LopezPaz and Ranzato, 2017]:
[Equation]
=
[Equation]
(cid:80)
[Equation] − [Equation] Online setting : A key difference between online and offline CL is that the latter assumes full access to the whole training data for the task that is currently being processed.
Therefore, it allows training on each single task with multiple epochs (e.g., 70-200 epochs) [Rebuffi et al., 2017, Wu et al., 2019,
Cha et al., 2021].
The online CL setting is more challenging because the agent can only access the current batch of incoming data and performs training with a single pass through the data.
Experience Replay (Rehearsal) : Chaudhry et al.
(2019) propose experience replay (ER), which performs joint training on memory samples and incoming samples.
A simple but strong baseline approach to sampling in ER is reservoir sampling Vitter [1985].
Aljundi et al.
(2019) propose
Maximally Interfered Retrieval (MIR), which retrieves the samples that will be most negatively impacted by the foreseen parameter updates.
Shim et al.
(2021) propose ASER, which selects samples to best preserve existing memory-based class boundaries.
In terms of model training, Mai et al.
(2021) propose to replace the cross-entropy loss with the supervised contrastive loss to learn a better representation.
We consider all these variants of ER in our experiments.
Augmentation :
In the standard i.i.d.
setting, data augmentation is a widely used method to improve deep learning [Cubuk et al., 2020].
In the offline CL setting, Mai et al.
[2021] use augmentation to construct a supervised contrastive loss and Bang et al.
[2021] employ it together with an uncertaintybased memory management strategy.
However, these papers apply augmentation together with other advanced techniques, and there is no ablation study on the effect of augmentation per se.
Thus, it is unclear whether augmentation itself helps rehearsal or not, especially in the online setting.
Repeated Rehearsal (Multiple Iterations) : Vanilla online continual learning employs a single gradient update given an incoming batch of data.
To maximally utilize the current incoming batch, Aljundi et al.
[2019] propose to perform multiple gradient updates instead.
Their experiment with CIFAR shows using MIR with five iterations leads to a 1.7% improvement in accuracy.
In this paper, we systematically analyze the effect of multiple iterations on online rehearsal and provide theoretical and empirical insights on when this trick may improve or harm performance.
Hyperparameter Tuning for OCL :
Hyperparameter tuning is a particular challenge in OCL due to the lack of a dedicated validation set and the constraint of a single pass through the data.
Chaudhry et al.
[2018] and Mai et al.
[2022] employ a hyperparameter tuning protocol that uses an external validation data stream with a small number of tasks.
Offline hyperparameter tuning is applied to this validation data with multiple passes to identify optimal values, which are then used for the actual online continual learning tasks.
A limitation of this method is that it relies on external validation data.
Revisiting Online Rehearsal: Is Repeated Rehearsal a Good Idea?
We revisit rehearsal from two directions.
First, while previous work demonstrates its strong empirical performance [Chaudhry et al., 2019] and provides conceptual analysis for offline CL [Verwimp et al., 2021], we focus on the online setting and provide theoretical insights through the lens of empirical risk minimization (ERM).
Second, we examine the dynamics of employing rehearsal with multiple iterations.
This has been proposed as a trick for online CL to maximally utilize the incoming batch already [Mai et al., 2022, Aljundi et al., 2019]; we investigate whether it is always better than rehearsal with a single iteration.
: a Biased and Dynamic Objective
For rehearsal in OCL, at each iteration t , a batch of data B t is obtained from the incoming task, where
B t ∼ D T and B t
= { x i , y i } i =1 ,... |B
t | , and a batch B M t is sampled from memory, where B M t ∼ D t M and B M t = { x i , y i } i =1 ,... |B
M t | .
The gradient-based update rule of ER is:
[Equation]
Given this update rule, we would like to establish the corresponding objective function, but this is not straightforward to derive because the memory is immediately updated after each incoming batch, which means D t M changes all the time.
For a widely used memory management policy, where the memory is updated using reservoir sampling [Vitter, 1985, Chaudhry et al., 2019], we prove in the appendix that the empirical risk for online ER follows Proposition 1.
Proposition 1 (ERM for online rehearsal) : Assume an incoming task stream D T and an initial memory set D 0 M with different data distribution P ( D T )
̸
= P ( D 0 M ) .
Assume further that the memory is updated at the end of each iteration using reservoir sampling.
Then, Eq 2 implements unbiased stochastic gradient descent for the following loss function:
[Equation]
( f θ ( x ) , y ) , (3) where λ :=
|D T | |D 0 M | and |D
M | and |D T | are the memory size and incoming task data size respectively; β t :
= 1 / (1 + 2 N tcur N T past ) , and N tcur =
(cid:80) i
= t i =1 |B i | denotes the number of samples of the current task that have been seen so far and N T past =
(cid:80)
j = T j =1 |D j | denotes the number of samples pertaining to the tasks so far, excluding the current task.
We immediately have β t ∈ (0 , 1] .
The proposition reveals several interesting properties: • Bias : compared with the true objective in continual learning in Eq 1, the loss of online ER in Eq 3 is actually a biased approximation of the former, as it puts a different weight ( β t λ ) on memory samples while the true objective treats all samples with equal weight.
This bias, introduced by ER’s objective function, can contribute to the risk of memory overfitting.
• Problem-dependence : given that β t ∈ (0 , 1] , the biased weight on the memory samples is mostly influenced by λ .
In other words, the memory overfitting risk is related to an inherent property of the CL problem concerned: the ratio λ between the current task data size and the memory data size.
While previous works extensively report empirical results on the influence of memory size, to our knowledge, we are the first to point out that the relative data size of an incoming task also plays an important role.
Empirical evidence is provided in Section 6.3 to support this claim.
With a 2k memory, performing rehearsal on the CORE dataset with a larger task data size ( λ = 6 ) faces a high level of memory overfitting while the CLRS dataset with a smaller task data size ( λ = 1 . 12 ) enjoys a lower risk of memory overfitting.
• Dynamic : ER in online CL optimizes towards a dynamic objective, which varies with each incoming batch t , as the weight on the memory sample depends on N tcur , and N T past .
This analysis shows that memory overfitting may be relatively slight for the first few tasks when N T past is small.
As more tasks arrive, memory overfitting worsens as β t increases with N T past .
In the case of an infinite data stream, lim N T past →∞ β t = 1 , the memory weight is solely determined by λ .
3.2 Repeated Rehearsal: The Decaying Regularization Effect of Incoming Data
We now investigate whether and why performing multiple iterations is indeed beneficial to rehearsal.
We refer to applying multiple iterations in ER as “repeated experience replay” (Repeated ER) or “repeated rehearsal” and formalize it as follows: for each incoming data batch B t ∼ D T from the data stream, we perform multiple gradient updates ( K in total) using stochastic gradient descent (SGD) or variants thereof.
At each gradient update k = 1 , ...
K , a data batch B M t,k is chosen from memory M and concatenated with the current incoming batch B t to perform a replay iteration as follows:
[Equation]
Note that when k = 1 , this update rule provides an unbiased stochastic gradient for Eq 3; in the case of k > 1 , it provides a biased gradient estimate as the same incoming batch is used for consecutive gradient updates.
To study the influence of this biased gradient update in repeated rehearsal, we examine the internal dynamics of repeated rehearsal by studying the loss landscape.
Memory Overfitting :
We compare the loss surfaces regarding the memory samples and the test data of past tasks during repeated rehearsal.
Following the visualization method used in Verwimp et al. [2021], Mirzadeh et al.
[2020], we examine the learning process on the first two tasks in the Split
Mini-ImageNet dataset and plot the loss landscape in the 2D plane defined by three model parameter vectors 2 : the model w 1 obtained by training on the first task until convergence, the model w 2 obtained on the second task using experience replay, and the model w 2 ,ft obtained after training on the second task using finetuning without replay.
Verwimp et al.
(2021) use this method to demonstrate the memory overfitting in ER for offline continual learning setting with 10 epochs.
Our results show that applying ER in the online setting yields severe memory overfitting with 10 iterations (see Fig 1 (a)).
In other words, increasing the number of iterations means the loss landscape of the rehearsal memory provides a poorer approximation of the loss landscape of previous tasks, as shown by the differences in the positions and the shapes between the left and right red contours in Fig 1 (a).
As a result, the learning trajectory of R-ER avoids the high-loss ridge region for the memory data but goes right into the high-loss ridge region for the past tasks’ test data.
Regularization Effect of Incoming Data : To investigate why repeated rehearsal may suffer from even more memory overfitting than vanilla rehearsal, we analyze the regularization effects of incoming data during repeated ER.
To this end, we examine the training process given an incoming batch and compare the training loss on the memory batch and incoming batch with respect to memory iteration k (see Fig 2 ).
An interesting observation is that during the training session of a given incoming data batch, the decrease in the training loss on the incoming batch is much faster than the decrease in loss on the memory batches.
One intuitive explanation is that the former is computed over a fixed batch during multiple iterations and the latter is computed over different memory batch samples.
As a result, even though the incoming loss is larger than the memory loss at the start of a training session ( k = 1 ), at later iterations (i.e., k > 5 ) the training loss of the incoming batch becomes 10 − 10times lower than that of the memory batch.
This means that at this stage the regularization effect of the incoming data batch is greatly undermined: the joint training on the memory batch and the incoming batch becomes similar to training on memory only.
In summary, our findings imply that the performance of online rehearsal is constrained by the dilemma between overfitting locally and underfitting globally.
Specifically, online rehearsal faces the challenge of underfitting of the large data stream but overfitting of a small memorized data subset.
Applying repeated rehearsal ameliorates the former problem but aggravate the latter problem.
Therefore, the performance gain of repeated rehearsal is quite limited.
Repeated Augmented Rehearsal
To deal with the overfitting-underfitting dilemma, we explore a simple strategy, “repeated augmented rehearsal” (RAR), which combines repeated rehearsal with data augmentation.
Consider a group of transforms G that acts on the input space X and is invariant under function f , i.e., f ( g x ) =
f
( x ) , g ∈ G, x ∈ X .
Given an incoming batch from the data stream, B t , multiple replay iterations are conducted using this batch.
At each replay iteration k , a random memory batch B M t,k is sampled and concatenated with the incoming batch.
Then, a random transform g t,k ∈ G is sampled and applied to each data point x i in the concatenated minibatch.
The model parameters are updated as :
[Equation]
Intuitively, the augmentation can help alleviate memory overfitting in two ways.
First, we observe that applying augmentation on the incoming batch helps strengthen the regularization effect.
As shown in Fig 2(b), the decaying regularization effect of incoming data is alleviated in RAR, as the loss of the incoming batch stays comparable to the loss of the memory batch during multiple iterations.
Second, rehearsal on augmented memory batches can help to more accurately reflect past tasks’ data distributions.
With RAR, the loss landscapes of memory data and past tasks’ test data become very similar (see Fig 1 (b)), which suggests that the model ends up in a part of the parameter space where the rehearsal memory approximates the past tasks’ distribution well.
Moreover, the continual learning solution identified with RAR avoids the high-loss ridge not only in the memory data loss landscape but also in the test data loss landscape.
Theoretically, we prove that augmented rehearsal reduces the generalization error in OCL.
Specifically, assume the augmentation group G is a compact topological group and follows a probability distribution Q .
Similar to Proposition 1 , it can be easily proven that the augmented rehearsal gradient corresponds to unbiased SGD on an augmented empirical risk 3 (see Proposition 3 in Appendix A):
[Equation]
This result shows that applying augmented rehearsal is equivalent to performing an averaging operation of the loss of rehearsal in Eq 3 over the orbits of a certain group that keeps the data distribution approximately invariant.
In the standard i.i.d.
learning setting, Chen et al.
[2020] found that such an orbit-averaging operation can reduce both the variance and generalization error.
Based on
Eq 3 and Eq 5, we show that this theoretical benefit of using augmentation to boost model invariance is also applicable to rehearsal in continual learning.
In fact, as discussed in Section 3.1, rehearsal in online CL has a biased empirical risk Eq 3, which leads to inherent memory overfitting and poor generalization ability.
Thus, this benefit of augmentation in reducing generalization error is particularly important when applying rehearsal in online CL.
The modifications required for the RAR procedure are summarized in Lines 3 and 5 of Algorithm 1.
It uses a general framework for ER-based continual learning that consists of three key components: sampling from memory, joint training on memory data and incoming data, and updating of the memory.
As mentioned in the related work section, different ER variants have been proposed to improve these components.
RAR can flexibly be combined with any of these ER variants, and we investigate the effectiveness of RAR on these different ER variants in the experiment section.
Reinforcement Learning-based Adaptive Repeated Augmented Rehearsal
There are two key components in RAR: repeated rehearsal and augmented rehearsal.
The interplay of the two is determined by the number of memory iterations and the strength of augmentation.
A key question is how to choose these hyperparameters.
In general, hyperparameter tuning (HPT) still remains an unsolved challenge for online CL due to the single-pass assumption [Chaudhry et al., 2018].
Finding suitable RAR hyperparameters needs to account for the severity of memory overfitting and poses extra challenges.
In particular, as shown in the ERM analysis, the extent of memory overfitting is related to the CL problem features (e.g., task data size and memory size) and also varies at different training stages of the continual learning process.
To automatically select suitable
RAR hyperparameters for the different CL problems and different training stages, we propose to use reinforcement learning to adaptively adjust the hyperparameters (see Algorithm 1).
In particular, we design the hyperparameters of RAR as the action space and use the training statistics as the reward (see lines 2, 6, and 10 in Algorithm 1).
A major challenge of applying RL in online
M is the memory with fixed size, B t is the incoming batch from the current task, θ are the parameters of the CL network, w are the parameters of the RL agent, K is the number of memory iterations, P, Q are the augmentation hyperparameters 1: procedure RAR( M t , B t , θ t , w t ) 2:
K t , P t , Q t = SampleAction ( w t ) 3: for k = 1 , ..., K t do 4:
B M t,k ∼ MemRetrieval ( M t ) 5:
B aug ← aug ( B M t,k ∪ B t , P t , Q t ) 6: r t ← ComputeReward ( B aug , θ t,k ) 7: θ t,k +1 ← SGD ( B aug , θ t,k ) 8: end for 9:
M t +1 ← MemUpdate ( M t , B t ) 10: w t +1 ← UpdateRL ( r t ) 11: end procedure
HPT is sample efficiency.
The exploration horizon (i.e, training steps) in the OCL environment is quite limited due to the constraint of a single pass through the data and poor action choices (undesirable hyperparameters) may lead to a bad gradient update step and hurt the OCL training process.
To address the sample efficiency issue, we employ the multi-armed bandit framework and apply bootstrapped policy gradient (BPG)
[Zhang and Goh, 2019].
The key idea of BPG is to incorporate prior knowledge to bootstrap the policy gradient to achieve stable and fast convergence with limited samples.
To obtain prior knowledge in OCL problems, we use the training accuracy on the memory batch as the overfitting feedback, as a higher training memory accuracy suggests a higher chance of memory overfitting.
Reward is defined as the distance between the current memory accuracy and target memory accuracy.
Compared against a target memory accuracy (e.g., 0.9), the current memory accuracy is used to indicate whether the current choice of rehearsal iteration or augmentation causes too much memory overfitting and then the action selection probability is adjusted following BPG (see Appendix C.2 for more RL design and implementation details). The algorithm details of applying BPG as a specific RL method for hyperparameter tuning is summarized in Algorithm 2 of Appendix C.2).
Experiments 6.1 Experiment Setup
Baseline : We apply RAR to four ER-based continual learning algorithms: ER
[Chaudhry et al., 2019],
MIR [Aljundi et al., 2019], ASER [Shim et al., 2021], and SCR [Mai et al., 2021].
We also compare it with other continual learning methods, including the regularization-based method LWF [Li and
Hoiem, 2017] and the constrained optimization-based method A-GEM [Chaudhry et al., 2018].
Dataset : Four CL benchmarks are used in the experiments: Seq-CIFAR100 (20 tasks), SeqMiniImageNet (10 tasks)
Vinyals et al.
[2016], CORE50-NC (9 tasks)
Lomonaco and Maltoni [2017] and CLRS25-NC (5 tasks)
Li et al.
[2020](see Appendix B for more details).
Additionally, we also investigate ER and RAR on the large-scale ImageNet-1k dataset in Appendix D.3.
Implementation : We use a reduced ResNet-18 for all datasets following Mai et al.
[2021], Aljundi et al.
Single-head evaluation is employed with a shared final layer trained for all the tasks.
RandAugmentation Cubuk et al.
[2020] is used for auto augmentation.
Given a set of augmentation operations, it randomly selects P augmentation operations and exerts an augmentation magnitude of Q for all the selected augmentation operations on each image.
All the experimental results we present are averages of three runs.
We summarize all hyperparameter details in Appendix C.
The running time of different algorithms is shown in Appendix D.5. [Table]
6.2 Main Results
We first analyze RAR’s performance with a pre-defined hyperparameter set ( K = 10 , P = 1 , Q = 14 ).As
shown in Table 1, RAR greatly improves the ER method on the four datasets, by +8 . 8% ∼ +17 .
Moreover, RAR also leads to substantial gains for the other algorithmic variants of ER for all datasets (MIR: +9 . 1% ∼ +17 . 7% , ASER: +7 . 2% ∼ +14 . 7% , SCR:
+5 .
1% ∼ +17 . 9% ).
These results suggest that even with advanced memory management strategies, such as MIR or ASER, or representation learning techniques, e.g., SCR, OCL still benefits substantially from repeated augmented rehearsal. RAR with Modified Rehearsal Loss Besides using the vanilla online rehearsal loss in Eq 3, we investigate the effectiveness of RAR with another two, more advanced, rehearsal loss designs: 1) Reweighted memory loss : ER-rw introduces a reweighting hyperparameter α in the gradient of Eq 2 to deal with the biased ER loss by balancing the weight of the memory loss and incoming loss; 2) Distillation-based memory loss : DER [Buzzega et al., 2020] employs the logits-based distillation loss for memory samples, instead of the cross entropy loss.
The results in Table 1 show
RAR leads to large performance gains for ER-rw (for the best α choice; further results can be found in
Appendix D.2) and DER for all four datasets (ER
-rw: +9 . 4% ∼ +23 .
0% , DER: +11 .
9% ∼ +32 . 8% ).
This suggests that even with more advanced rehearsal loss designs, repeated augmented rehearsal is important for online rehearsal.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] Stability and Plasticity Trade-off Based on the definition of accuracy A T , forgetting F T and backward transfer B T in Section 2, we find that these three metrics have the following relationship:
[Equation]
Interestingly, this finding shows that accuracy is related to the ability to learn new tasks, quantified by 1 T (cid:80)
Ti =1 a
i,i , and the ability to avoid forgetting past tasks, quantified by T − 1 T B T , which draws a connection to the more general problem of the stability-plasticity trade-off in neural networks and continual learning [Grossberg, 2012, Delange et al., 2021].
Plasticity refers to the ability to integrate new knowledge and stability refers to the ability to retain old knowledge.
Fig 3 presents the stability and plasticity trade-off in RAR.
Generally, we observe increasing the repeated rehearsal iterations ( K ) leads to a higher level of plasticity.
However, this may also cause a decrease of stability, i.e., introduce forgetting.
On the other hand, the use of augmentation generally improves stability.
More importantly, the use of augmentation in repeated rehearsal shifts the stability-plasticity trade-off curve towards the upper right, thus creating a better stability-plasticity trade-off frontier.
Hyperparameter Tuning for RAR We compare the RL-based hyperparameter tuning method with the hyperparameter tuning framework for continual learning (HTOCL) method used in Chaudhry et al.
[2018], Mai et al.
[2022] (see Section 2).
The results in Table 2 show that RL-based RAR significantly outperforms using HTOCL to select hyperparameters for RAR.
One reason is that the extent of memory overfitting varies at the different training stages of CL.
HTOCL only uses the first few tasks to select hyperparameters for RAR, which may not optimal for later stages of CL.
In fact, we observe HTOCL tends to select a large number of repeats and small augmentation strength.
This selection strategy may be desirable for problems with short task sequences but problematic for long task sequences with increased memory overfitting risk.
In contrast, the RL-based method can take into account the latest feedback (e.g., train memory accuracy) to adjust hyperparameter choices.
The selected iteration numbers and augmentation are shown in Appendix D.4.
6.3 Ablation Studies
Interplay between Repeated and Augmented Rehearsal We investigate the interaction of the number of replay iterations with the augmentation strength in Fig 4.
For three out of four datasets, using augmentation alone without repeated rehearsal leads to even worse performances than rehearsal without augmentation (see K = 1 in Fig 4).
One explanation is the underfitting challenge of online CL.
Training on augmented samples can make model underfitting even worse.
The only exception is the CORE50 dataset, which inherently has a high memory overfitting risk with λ = 6 making it benefit more from augmentation.
Similarly, employing repeated rehearsal alone (see the blue solid line in Fig 4) also harms performance in three out of four datasets, with the CLRS dataset as the only exception, which enjoys a low memory overfitting risk with λ = 1 . 12 .
An important takeaway message is that repeated rehearsal or augmented rehearsal is not always helpful in OCL settings and whether they will benefit or harm the performance is dependent on the structure of the OCL problem at hand (e.g., the task data size and memory data size).
RAR’s Robustness to Large Numbers of Repeats Although the performance curve flattens out around 10 iterations, there is no evident drop in performance even with 20 iterations.
This result reinforces how RAR can help with the underfitting-overfitting dilemma: it can support the use of more training in OCL without having to worry about the performance drop introduced by memory overfitting.
In comparison, for repeated rehearsal without augmentation (solid lines), the accuracy starts to drop quickly on CIFAR100 and MiniImageNet when using more than two iterations.
Augmenting the Memory vs. Augmenting the Incoming Data RAR applies augmentation to both the memory batch and the incoming batch.
We examine the effectiveness of the two separately.
Table 2 presents the performance of RAR applied (a) solely with memory augmentation (RARmem) and (b) solely with incoming data augmentation (RAR-inc).
We find that RAR achieves the best performance compared to RAR-mem and RAR-inc.
This shows that it is beneficial to apply augmentation to both the memory batch and the incoming batch.
Interestingly, RAR-inc itself also achieves consistent performance improvements over R-ER, e.g., a gain of 7.7% in CORE50.
As discussed in Section 3, adding augmentation on the incoming batch can strengthen the regularization effect of the incoming task and indirectly alleviate memory overfitting.
RAR with MIR, ASER,
SCR We also perform ablation studies to investigate RAR’s strong performance with the ER variants (see Appendix D.1).
Similar to ER, the performance gains for RAR-MIR and RAR-ASER come from the combination of repeated rehearsal and augmented rehearsal.
However, for SCR, the repeated rehearsal itself also leads to a consistent performance boost.
One reason is that
SCR already includes a strong augmentation procedure with four augmentation operations to construct the supervised contrastive loss used in this method.
It also works poorly without augmentation.
Augmentation for Offline Rehearsal Although this paper focuses on online CL, the (augmented) empirical risk analysis is also relevant to offline CL (see Propositions 2 and 3 in Appendix A).
This suggests the memory overfitting risk in offline rehearsal is related to the ratio of task-to-memory size λ and can be alleviated by augmented rehearsal (see more empirical results in Table 8 of Appendix D.6).
Discussion and Conclusion
Rehearsal-based methods play a central role in fighting catastrophic forgetting when learning from non-stationary data streams.
Compared to offline rehearsal, online rehearsal has faced particular challenges in tackling complex CL datasets due to the single-pass-through data constraint.
This work tries to analyze the internal workings of online rehearsal from a theoretical and conceptual perspective and identifies the fundamental challenge that it faces as the dilemma between overfitting locally and underfitting globally.
To deal with this challenge, we propose a simple baseline: repeated and augmented rehearsal (RAR).
Surprisingly, despite its simplicity, RAR achieves a large performance boost for a set of different rehearsal-based methods.
Additionally, we propose an RL-based method to tune the hyperparameters of RAR to balance the stability-plasticity trade-off in an online manner.
It achieves promising results compared to hyperparameter tuning based on validation data.
Also, this work is focused on continual learning for classification problems with image data.
An interesting future research direction is to look at other CL domains, e.g., text/audio inputs or RL problems.
Acknowledgments and Disclosure of Funding
This research is funded by the New Zealand MBIE TAIAO data science programme.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Theoretical Analysis
This section contains the theoretical analysis of the loss functions of offline experience replay ( Proposition 2 ), augmented experience replay ( Proposition 3 ), and online experience replay with reservoir sampling ( Proposition 1 ).
Proposition 2 (Empirical risk minimization for experience replay) :
We assume a memory set
D M and an incoming task stream D T with different data distribution P ( D T )
̸
= P ( D M ) .
At each iteration t , a batch of data is sampled from memory B M t ∼ D M with B M t = { x i , y i } i =1 ,... |B
M t | , and a batch of data is sampled from incoming task B t ∼ D T with B t = { x i , y i } i =1 ,... |B
t | .
To update the parameter θ of a function f θ : f θ × X → Y based on loss function L , consider the parameter update rule defined by θ = θ − η |B t | (cid:80)
x
i ,y i ∈B t ∇L ( f θ ( x i ) , y i ) −
[Equation]
| (cid:80) x
i ,
[Equation] [Equation]
( x
[Equation]
) , y i ) , then it is an unbiased stochastic gradient descent update rule for the following empirical risk:
[Equation]
Proof :
Given the empirical gradient
[Equation]
[Equation]
|
[Equation]
M t |
[Equation]
x i ,y [Equation] t ∇L ( f θ ( x i ) , y i ) during stochastic optimization, we can derive the gradient expectation as follows:
[Equation]
Proposition 3 (Augmented empirical risk minimization for experience replay) : Assume a memory set D M = { x i , y i } i =1 ,.., |D M | and an incoming task stream D T = { x i , y i } i =1 ,.., |D T | with different data distribution P ( D T )
̸
= P ( D M ) , and a compact topological group of transform G with a probability distribution Q that acts on the input space X and is invariant under function f , i.e., f ( gx ) =
f ( x ) , g ∈ G, x ∈ X .
At each iteration t , a batch of data is sampled from memory, B M t ∼ D M and B M t = { x i , y i } i =1 ,... |B
M t | , a batch of data is sampled from the incoming task, B t ∼ D T and B t = { x i , y i } i =1 ,... |B t | , and a group operation g t ∼ G is randomly selected.
To update the parameter θ of the function f θ based on loss function L , consider the parameter update rule defined by θ = θ − η |B t | (cid:80)
x
i ,y i ∈B t ∇L ( f θ ( g t x i ) , y i ) − η | B M t | (cid:80) x i ,y i ∈B M t ∇L ( f θ ( g t x i ) , y i ) , then it is an unbiased stochastic gradient descent update rule for the loss function
[Equation]
Proof :
Given the augmented empirical gradient during stochastic optimization, we can derive the gradient expectation as follows:
[Equation]
The second equality is based on the results of Proposition 1, which we prove next.
Proposition 1 (ERM for online rehearsal) : Assume an initial memory set D 0 M and an incoming task stream D T with different data distribution P ( D T )
̸
= P ( D M ) .
At each iteration t , t = 1 , ..T , a batch of data is sampled from the incoming task, B t ∼ D T and B t = { x i , y i } i =1 ,... |B t | , and a batch of data is sampled from the memory, B M t ∼ D t M and B M t = { x i , y i } i =1 ,... |B
M t | .
To update the parameter θ of the function f θ : f θ ×X → Y based on loss function L , consider a parameter update rule defined by
[Equation] [Equation]
Assume the memory is updated at the end of each iteration using reservoir sampling Vitter [1985], Chaudhry et al.
D t +1 M ← RS ( D t M , B t ).
Then the gradient update rule is an unbiased stochastic gradient descent update rule for the loss function
[Equation]
where β t = 1+2 ∗ NtcurNpast , N tcur = (cid:80) i
=
t i =1 |B i | denotes the number of samples of the current task that have been seen so far and N past = (cid:80)
j = T j =1 |D j | denotes the number of samples of past tasks.
Note 1 : The objective function R t ( θ ) changes with respect to the batch number t due to the changes in β t .
Note 2 : 1 1+2 ∗ |DT |
Npast ≤ β t ≤ 1 and β t is decreasing with the batch number t .
At the start of a task, β t =
= β max = 1 .
At the end of a task, β t = T = β min = 1 1+2 ∗ |DT | Npast .
Note 3 : Consider a balanced continual learning dataset (e.g., Split-CIFAR100, Split-Mini-ImageNet) where |D
j | =
|D T | , j = 1 , .. T .
Then, we have β min =
T − 1 T +1 =
(1 − 2 T +1 )
and lim T →∞ β min = 1 .
Note 4 : Consider general continual learning datasets.
As CL learns more tasks, N past increases, and lim N past →∞ β t =
lim N past →∞
[Equation]
Proof :
Given the empirical gradient
[Equation]
∇L ( f θ ( x i ) , y i ) during stochastic optimization, we can derive the gradient expectation as follows:
[Equation]
i
[Equation]
The third equality is based on the condition that D t M is updated using reservoir sampling so that all the data seen so far have an equal probability of being stored in the memory.
In other words, at a given time t , the memory contains a fraction of
N past N past
+ N tcur samples coming from past tasks and a fraction of
N
tcurr N past + N t cur samples coming from the current task
B Dataset details
Table 3 lists the image size, the number of classes, the number of tasks, and data size per task of the four CL benchmarks.
[Table]
C Implementation Details C.1 Continual Learning Implementation
The hyperparameter settings are summarized in Table 4.
All models are optimized using vanilla SGD.
For all experiments, we use the learning rate of 0.1 following the same setting as in Aljundi et al. [2019], Shim et al.
[2021], and the Nearest-Class-Mean (NCM) classifier is used for evaluation, as
Mai et al. reported (2021) considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier.
Each mini batch during training consists of 10 new and 10 memory samples, except for the SCR method, which employs 100 memory samples and 10 new incoming samples Mai et al.
By default, the repeated rehearsal parameter for all the results is K =and the augmented rehearsal parameters are P = 1 , Q = 14 .
This paper uses Randaugment [Cubuk et al., 2020], which is an auto augmentation method.
It randomly selects P augmentation operators from a set of 14 operators and applies them to the images.
The augmentation operator set includes:’Identity’, ’AutoContrast’, ’Equalize’, ’Rotate’, ’Solarize’, ’Color’,’Posterize’, ’Contrast’, ’Brightness’, ’Sharpness’,’ShearX’, ’ShearY’,’TranslateX’, ’TranslateY’.
[Table]
ASER
[Table]
C.2 RL-based hyperparameter tuning implementation
The memory iteration choices are from 1 to 20 and the augmentation choices are (1 , 5) , (1 , 14) , (2 , 14) , (3 , 14) , (4 , 14) .
Action selection probabilities are modeled with softmax weight π w ( a i ) =
e wi (cid:80) k e wk .
Bootstrapped policy gradient is used to adjust action weights:
[Equation]
where ˆ π + w ( a i ) :
= Σ a k ∈X + ai π w ( a k ) and ˆ π − w ( a i ) :=
Σ a k ∈X − ai π w ( a k )
Better/Worse action set The key of the BPG idea is to incorporate prior information into the construction of better and worse action sets.
To apply BPG in the OCL environment, we propose to determine the better/worse action set based on the feedback in the form of current memory batch accuracy A M , which reflects the memory overfitting level of the CL agent.
We want the training memory accuracy to be neither too high nor too low.
The desirable training memory accuracy is defined by A ∗M .
We find a higher repeated replay iteration leads to higher memory accuracy.
Therefore, the better/worse action set is defined as follows:
[Equation]
Reward A challenge in the hyperparameter tuning for the OCL setting is that the CL agent may face new tasks with unseen data distribution.
Therefore, it is often infeasible to assume the existence of an external validation data containing all the tasks in advance for hyperparameter tuning.
To achieve online hyperparameter tuning without external validation data, we propose to define the reward based on the accuracy on the memory.
Given a target memory accuracy A ∗M , the reward is defined as
[Equation]
| A M − A ∗M |
Non-stationarity To address the non-stationary nature in the CL environment, we reset the weight of
BPG to a uniform weight at the start of each task.
The analysis of the selected action can be found in Figures 9 and 10 in Section D.4.
M is the memory with fixed size,
B t is the incoming batch from the current task, θ are the parameters of the CL network, w are the parameters of the RL agent,
K is the number of memory iterations,
P, Q are the augmentation hyperparameters
A ∗M target memory accuracy 1: procedure RAR( M t , B t , θ t , w t ) 2:
K t ∼ π w itert 3:
P t , Q t ∼ π w augt 4: for k = 1 , ..., K t do 5:
B M t,k ∼ MemRetrieval ( M t ) 6:
B aug ← aug ( B M t,k ∪ B t , P t , Q t ) 7:
A M ← MemAcc ( M , θ t,k ) 8:
X + , X − ← ActionSet
( A M , A ∗M ) 9: θ t,k +1 ← SGD ( B aug , θ t,k ) 10: end for 11:
M t +1 ← MemUpdate ( M t , B t ) 12: w itert +1 , w augt +1 ← UpdateRL
( A M , X + , X − ) 13: end procedure
D Supplementary Experimental Results D.1 Ablation studies for MIR-RAR, ASER-RAR and SCR-RAR
As shown in Table 1, RAR improves ER and ER variants (MIR, ASER and SCR).
The detailed results of the ablation studies of RAR with ER variants are presented in this section.
In particular, Fig 5 and 6 show the comparison of only using repeated rehearsal or augmented rehearsal for MIR and ASER respectively.
Neither of them alone consistently improves the performance of the baseline.
This result suggests the influence of RAR for MIR/ASER is similar to ER.
The performance gain comes from the combination of the repeated rehearsal and augmented rehearsal.
Fig 7 shows the results for SCR.
Repeated rehearsal leads to consistent performance gains because SCR already includes augmentation.
D.2 Reweighted ER
To deal with the biased ER loss, one straightforward way is to balance the weight of the memory loss and incoming loss by introducing a reweighting hyperparameter α in the gradient of Eq 2.
Specifically, the gradient for reweighted ER is implemented as
[Equation]
The performance of ER and RAR with respect to different reweighting hyperparameter values α ∈ [0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9] is shown in Fig 8, where α = 0 .
5 denotes vanilla ER loss with the equal weighting of memory loss and incoming loss.
To keep the learning rate comparable to vanilla ER, the learning rate of ER-rw is twice that of ER.
A key observation is that similar to vanilla ER, reweighted
ER (ER-rw) also significantly benefits from repeated and augmented rehearsal, as ER-rw-rar greatly improves over ER-rw (see the red line and blue line in Fig 8) for all four datasets.
D.3 Large-scale online CL
To examine the effectiveness of RAR in a large-scale continual learning problem, we apply RAR to
ImageNet-1k.
ImageNet-1k is split into 10 tasks and each task contains 100 classes.
The training dataset of 10 tasks contains 1,281,167 images in total.
Considering the task size, we evaluate with a memory size of M= 20k (with the task to memory size ratio λ ≈ 6 . 4 ) and a memory size of
M=100k (with the task to memory size ratio λ ≈ 1 . 28 ).
These choices are similar to Seq-CIFAR ( λ = 1 . 25 ) and CORE50 ( λ = 6 ) with a 2k memory.
We randomly crop the images to size 224x224, and use the ResNet-18 architecture for training on ImageNet-1k with a single epoch.
An incoming batch size of 32 and a memory batch size of 32 are used.
We employ a learning rate of 0.1 with 0.001 of weight decay.
For the RAR method, we use 10 memory iterations with RandAugment and parameters P=1 and Q=14.
The results are shown in the Table 5.
[Figure] We observe that repeated augmented rehearsal (RAR) is effective in this large-scale online continual learning problem and improves the vanilla rehearsal from 2.1% to 15.1% with a 20k memory and from 8.7% to 34.7% with a 100k memory.
The discussion in Buzzega et al.
[2020] suggests that to deal with complex continual learning datasets, it is necessary to employ multiple epochs of training with offline CL to avoid the underfitting problem present in online learning.
Our results show that RAR can greatly improve online rehearsal for largescale CL problems.
Although several offline CL algorithms have been evaluated on ImageNet-1k, to our knowledge, our result is the first attempt to apply online CL to this problem.
We aim to investigate other online rehearsal-based methods on ImageNet-1k in future work.
[Table]
D.4 Action Selection in Reinforcement Learning
The selected hyperparameters for four datasets are shown in Figure 9.
Interestingly, the RL-based method assigns a stronger augmentation and lower iteration for a dataset with a higher λ attribute.
As discussed in the ERM analysis, a dataset with a higher λ suggests a higher risk of overfitting.
The RL-based method successfully takes this into account to tune the hyperparameters.
In contrast, the
OCL-HT method selects the weakest augmentation and highest iteration for three datasets.
Figure 10 presents the selected hyperparameters at different CL training stages.
Generally, as the continual learning proceeds, the RL-based method selects a stronger augmentation strength and lower iterations, to balance off the increasing risk of memory overfitting. [Figure]
D.5 Running time D.5.1 Running time of RAR
The running time of ER and ER-RAR is shown in Table 6.
Experiments are conducted using an
Nvidia GeForce RTX 2080 TI Graphics Card on Mini-ImageNet.
The running time of RAR grows linearly with respect to the number of replay iterations.
Hence, compared to vanilla experience replay, RAR requires more running time due to the multiple iteration design.
However, RAR is still much more computationally efficient compared with offline CL with multiple epochs.
An interesting future research direction is to study how to dynamically adjust the iteration number and augmentation strength of RAR to balance the trade-off of accuracy and running time.
D.5.2 Running time of RL
RL-based hyperparameter optimization is an online method that does not require repeated running over different hyperparameter choices.
Therefore, RL-based HPO is much more computationally efficient than offline hyperparameter selection methods.
More specifically, the hyperparameter search space in our problem is 100 with 5 augmentation strength levels and 20 memory iteration numbers.
Grid search would need to run the OCL algorithm 100 times while RL only needs to run it once.
[Table]
Nevertheless, the training of RL agents indeed introduces extra computation.
In practice, we observe the running time of RL-RAR is about two times slower than that of RAR, as shown in the Table
[Table]
D.6 Offline Continual Learning
Although this paper is mostly focused on online continual learning, some of the analysis and discussion are also of independent interest to offline continual learning.
Specifically, from a theoretical perspective, the empirical risk minimization of offline rehearsal is shown in Proposition 2 in Section A and its augmented risk is shown in Proposition 3.
Two conclusions can be drawn from this analysis.
First, the risk of memory overfitting in offline rehearsal is also related to the problem characteristic, the ratio λ between task size and memory size.
Second, augmentation can help with offline rehearsal since the orbit-averaging operation in the augmented empirical risk can reduce both the model variance and generalization error.
From an empirical perspective, Table 8 shows the performance of offline ER with and without augmentation in four datasets.
We use 50 epochs and a memory size of 2000.
For all four datasets, offline ER with augmentation achieves a significant performance gain over offline ER without augmentation.
More interestingly, compared to datasets with a small λ (e.g., CLRS with λ = 1 . 125 )
we observe that the datasets with a higher task-to-memory size ratio (e.g., CORE50 with λ = 6 ) tend to benefit more from augmentation, due to the increased risk of memory overfitting. [Table]