Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Online continual learning (OCL) aims to train neural networks incrementally froma non-stationary data stream with a single pass through data.
Rehearsal-based methods attempt to approximate the observed input distributions over time with a smallmemory and revisit them later to avoid forgetting.
Despite its strong empirical performance, rehearsal methods still suffer from a poor approximation of past data’sloss landscape with memory samples.
This paper revisits the rehearsal dynamics inonline settings.
We provide theoretical insights on the inherent memory overfittingrisk from the viewpoint of biased and dynamic empirical risk minimization, and examine the merits and limits of repeated rehearsal.
Inspired by our analysis, a simpleand intuitive baseline, Repeated Augmented Rehearsal (RAR), is designed to address the underfitting-overfitting dilemma of online rehearsal.
Surprisingly, acrossfour rather different OCL benchmarks, this simple baseline outperforms vanillarehearsal by 9%-19% and also significantly improves state-of-the-art rehearsalbased methods MIR, ASER, and SCR.
We also demonstrate that RAR successfullyachieves an accurate approximation of the loss landscape of past data and high-lossridge aversion in its learning trajectory.
Extensive ablation studies are conducted tostudy the interplay between repeated and augmented rehearsal and reinforcementlearning (RL) is applied to dynamically adjust the hyperparameters of RAR tobalance the stability-plasticity trade-off online.
Introduction
Despite its recent success, deep learning largely relies on the assumption of independent and identically distributed (i.i.d.) data that can be repeatedly revisited during training.
Non-i.i.d settings arechallenging for neural networks due to catastrophic forgetting: previously learned knowledge caneasily be overwritten when training on new data since this data may follow a different distribution [Liand Hoiem, 2017, Rebuffi et al., 2017, Delange et al., 2021].
Online continual learning (OCL or
Online CL) studies how to enable deep learning in an online manner from a non-stationary datastream.
As the data stream can be vast or even infinite, it is infeasible to store and shuffle the datasetfor multiple epochs of training.
Therefore, a fundamental assumption is that the data stream can onlybe accessed one batch at a time and training is performed with a single pass over the data [Aljundiet al., 2019].
Experience replay (ER), also known as rehearsal [Chaudhry et al., 2019, Delange et al., 2021], is a keymethod in OCL.
It stores a subset of previously seen data D in a fixed-size memory M and revisitsthe memorized samples during training to mitigate forgetting of previous tasks.
To update the model,a batch sampled from the memory is combined with the incoming batch from the stream to computethe gradient [Chaudhry et al., 2019].
Different variants of ER have been developed to improve memory sampling policies and representation learning, achieving state-of-the-art performance in anumber of standard OCL benchmarks [Aljundi et al., 2019, Mai et al., 2021, Shim et al., 2021].
However, whether rehearsal is appropriate for continual learning, considering the risk of overfitting thememory when using data from the memory to directly contribute to the gradient computation, has beendebated vigorously [Lopez-Paz and Ranzato, 2017, Chaudhry et al., 2019, Verwimp et al., 2021].
Thepotential for overfitting has motivated the development of constraint-based replay methods [LopezPaz and Ranzato, 2017], e.g. GEM and A-GEM, which use memory samples solely to constrain thegradient direction.
However, there is empirical evidence suggesting that rehearsal-based methodsconsistently outperform methods that do not train directly on the memory [Chaudhry et al., 2019].
This indicates rehearsal on the memory does not necessarily prevent effective generalization, possiblydue to the implicit regularization effect of incoming data [Chaudhry et al., 2019].
Nevertheless,recent work analyzing the loss landscape when applying rehearsal to offline continual learning findsthat memory samples indeed provide a poor approximation of the loss landscape past task’s data,especially near a high-loss ridge region.
As a result, " instead of ending up near the high-loss ridgein perspective of the rehearsal memory, the solution in reality resides on the high-loss ridge forthe training data " [Verwimp et al., 2021].
This latest finding poses the question of how to betterapproximate the loss surface of past data L ( D ; θ ) with memory samples’ loss L ( M ; θ ) .
To better approximate past data’s loss distribution, previous work studies which samples should bememorized for rehearsal.
Instead, we study how to effectively perform rehearsal with the memorizedsamples.
Focused on online CL, our study extends the previous understanding of rehearsal along twodirections.
First, we provide theoretical considerations that reveal two insights regarding the extent ofoverfitting to memory: it is a) related to the inherent attributes of the OCL problem concerned and b)varies across the different stages of continual learning.
Second, we highlight the limits of applyingrehearsal with multiple iterations—a trick used to maximally utilize the incoming batch [Aljundiet al., 2019]—and identify an underfitting-overfitting dilemma for online rehearsal.
Based on our analysis, we design a simple baseline to deal with the underfitting-overfitting dilemmain online CL problems, dubbed repeated augmented rehearsal (RAR), that can be easily integratedinto existing rehearsal-based methods.
Surprisingly, this simple baseline leads to a large performanceboost for ER, as well as state-of-the-art ER-based approaches, across four OCL benchmarks.
Moreimportantly, the loss landscape analysis shows that RAR can help memory samples reliably approximate the distribution of past data and successfully avoids the high-loss ridge of past tasks.
To betterunderstand the behavior of RAR, we further investigate the interplay between repeated rehearsal andaugmented rehearsal via an ablation study.
We also propose a reinforcement learning-based methodto dynamically adjust the hyperparameters of RAR and balance the stability-plasticity trade-off in anonline manner.
Related Work
Online continual learning : We consider the online continual learning setting with a non-stationary(potentially infinite) stream of data: at each time step t , the continual learning agent receivesan incoming batch of data samples B t = { x i , y i } i =1 ,.., |B t | that are drawn from the current data 74 distribution D t .
The period of time where the data distribution stays the same is often called a task orexperience in the continual learning literature.
Abrupt change in the data distribution occurs whenthe task changes.
The goal is to minimize the empirical risk on all the data seen so far:
[Equation]
with loss function L , the CL network function f , and its associated parameters θ .
Metrics : A common metric is the end accuracy after training on T tasks, defined as A T =1 T (cid:80) j = T j =1 a T,j , where a i,j denotes the model’s accuracy on the held-out test set of task j aftertraining on task i .
Other metrics are “forgetting” [Chaudhry et al., 2018], which is defined as
F T = − 1 T − 1 (cid:80) T − 1 i =1 ( a T,i − max l ∈ 1 ...
T − 1 a l,i ) and the related metric "backward transfer" [LopezPaz and Ranzato, 2017]: B T = 1 T − 1 (cid:80) T − 1 i =
a T,i − a i,i .
Online setting : A key difference between online and offline CL is that the latter assumes full accessto the whole training data for the task that is currently being processed.
Therefore, it allows training [Figure] on each single task with multiple epochs (e.g., 70-200 epochs) [Rebuffi et al., 2017, Wu et al., 2019,
Cha et al., 2021].
The online CL setting is more challenging because the agent can only access thecurrent batch of incoming data and performs training with a single pass through the data.
Experience Replay (Rehearsal) : Chaudhry et al.
(2019) propose experience replay (ER), whichperforms joint training on memory samples and incoming samples.
A simple but strong baselineapproach to sampling in ER is reservoir sampling Vitter [1985].
Aljundi et al.
(2019) propose
Maximally Interfered Retrieval (MIR), which retrieves the samples that will be most negativelyimpacted by the foreseen parameters update.
Shim et al.
(2021) propose ASER, which selectssamples to best preserve existing memory-based class boundaries.
In terms of model training, Mai etal.
(2021) propose to replace the cross-entropy loss with supervised contrastive loss to learn a betterrepresentation.
We consider all these variants of ER in our experiments.
Augmentation :
In the standard i.i.d.
setting, data augmentation is a widely used method to improvedeep learning [Cubuk et al., 2020].
In the offline CL setting, Mai et al.
[2021] uses augmentation toconstruct a supervised contrastive loss and Bang et al.
[2021] employs it together with an uncertaintybased memory management strategy.
However, these papers apply augmentation together with otheradvanced techniques, and there is no ablation study on the effect of augmentation per se.
Thus, it isunclear whether augmentation itself helps rehearsal or not, especially in the online setting.
Repeated Rehearsal (Multiple Iterations) : Vanilla online continual learning employs a single gradientupdate given an incoming batch of data.
To maximally utilize the current incoming batch, Aljundiet al.
[2019] proposes to perform multiple gradient updates instead.
Their experiment with CIFAR10shows using MIR with 5 iterations leads to a slight accuracy improvement of 1.7%.
In this paper, wesystematically analyze the effect of multiple iterations on online rehearsal and provide theoreticaland empirical insights on when this trick may improve or harm performance.
Hyperparameter Tuning for OCL :
Hyperparameter tuning is a particular challenge in OCL due tothe lack of a dedicated validation set and the constraint of a single pass through the data.
Chaudhryet al.
[2018] and Mai et al.
[2022] employ a hyperparameter tuning protocol that uses an externalvalidation data stream with a small number of tasks.
Offline hyperparameter tuning is applied to thisvalidation data with multiple passes to identify optimal values, which are then used for the actualonline continual learning tasks.
A limitation of this method is that it relies on external validation data.
Revisiting Online Rehearsal: Is Repeated Rehearsal a Good Idea?
We revisit rehearsal from two directions.
First, while previous work demonstrates its strong empiricalperformance [Chaudhry et al., 2019] and provides conceptual analysis for offline CL [Verwimp et al.
,2021], we focus on the online setting and provide theoretical insights through the lens of empiricalrisk minimization (ERM).
Second, we examine the dynamics of employing rehearsal with multipleiterations.
This has been proposed as a trick for online CL to maximally utilize the incoming batchalready [Mai et al., 2022, Aljundi et al., 2019]; we investigate whether it is always better thanrehearsal with a single iteration.
Empirical Risk Minimization in Online Rehearsal: a Biased and Dynamic Objective
For rehearsal in OCL, at each iteration t , a batch of data B t is obtained from the incoming task, where
B t ∼ D T and B t
= { x i , y i } i =1 ,... |B
t | , and a batch B M t is sampled from memory, where B M t ∼ D t M and B M t = { x i , y i } i =1 ,... |B
M t | .
The gradient update rule of ER is:
[Equation]
Given this update rule, we would like to establish the corresponding objective function, but this isnot straightforward to derive because the memory is immediately updated after each incoming batch,which means D t M changes all the time.
For a widely used memory management policy, where thememory is updated using reservoir sampling [Vitter, 1985, Chaudhry et al., 2019], we prove in theappendix that the empirical risk for online ER follows Theorem 1.
Theorem 1 (ERM for online rehearsal) : Assume an incoming task stream D T and an initialmemory set D 0 M with different data distribution P
( D T ) ̸
= P ( D M ) .
Assume further that the memoryis updated at the end of each iteration using reservoir sampling.
Then, Eq 2 implements unbiasedstochastic gradient descent for the following loss function:
[Equation]
where |D 0 M | and |D[-
 M | and |D-] T | are the memory size and incoming task data size, N tcur =
(cid:80) i
= t i {+=
+} |B i | denotesthe number of samples of the current task that have been seen so far and N T past =
(cid:80)
j = T j =1 |D j | denotesthe number of samples of past tasks.
For the purposes of discussing this ERM theorem, let us define λ := |D T | |D 0 M | and β t : = 1 / (1 + 2 N tcur N T past ) ,where β t ∈ (0 , 1] .
The theorem reveals several interesting properties: 1) Bias : compared with the trueobjective in continual learning in Eq 1, the loss of online ER in Eq 3 is actually a biased approximationof the former, as it puts a different weight ( β t λ ) on memory samples while the true objective treatsall samples with equal weight.
This bias, introduced by ER’s objective function, can contribute tothe risk of memory overfitting.
2) Problem-dependence : given that β t ∈ (0 , 1] , the biased weighton the memory samples is heavily influenced by λ .
In other words, the memory overfitting risk isrelated to an inherent property of the CL problem concerned: the ratio λ between the current taskdata size and the memory data size.
While previous works extensively report empirical results on theinfluence of memory size, to our knowledge, we are the first to point out that the relative data sizeof an incoming task also plays an important role.
Empirical evidence is provided in Section 6.3 tosupport this claim.
With a 2k memory, performing rehearsal on the CORE50 dataset with a largertask data size ( λ = 6 ) faces a high level of memory overfitting while the CLRS dataset with a smallertask data size ( λ = 1 . 12 ) enjoys a lower risk of memory overfitting.
3) Dynamic : ER in online CLoptimizes towards a dynamic objective, which varies with each incoming batch t , as the weight on thememory sample depends on N tcur , and N T past .
This analysis shows that memory overfitting may berelatively slight for the first few tasks when N T past is small.
As more tasks arrive, memory overfittingworsens as β t increases with N T past .
In the case of an infinite data stream, lim N T past →∞ β t = 1 , thememory weight is solely determined by λ .
3.2 Repeated Rehearsal: The Decaying Regularization Effect of Incoming Data
We now investigate whether and why performing multiple iterations is indeed beneficial to rehearsal.
We refer to applying multiple iterations in ER as “repeated experience replay” (Repeated ER) or“repeated rehearsal” and formalize it as follows: for each incoming data batch B t ∼ P t from the datastream, we perform multiple gradient updates ( K in total) using stochastic gradient descent (SGD) orvariants thereof.
At each gradient update k = 1 , ...
K , a data batch B M t,k is chosen from memory Mand concatenated with the current incoming batch B t to perform a replay iteration as follows.
[Equation]
Note that when k = 1 , this update rule is an unbiased stochastic gradient for Eq3; in the case ofk > 1 , it is a biased gradient estimate as the same incoming batch is used for consecutive gradientupdates.
To study the influence of this biased gradient update in repeated rehearsal, we examine theinternal dynamics of repeated rehearsal by studying the loss landscape.
Memory Overfitting :
We compare the loss surfaces regarding the memory samples and the test dataof past tasks during repeated rehearsal.
Following the visualization method used in Verwimp et al.[2021], Mirzadeh et al.
[2020], we examine the learning process on the first two tasks in the Split
Mini-ImageNet dataset and plot the loss landscape in the 2D plane defined by three model parametervectors 1 : the model w 1 obtained by training on the first task until convergence, the model w 2 obtainedon the second task using experience replay, and the model w 2 ,ft obtained after training on the secondtask using finetuning without replay.
Verwimp et al.
(2021) use this method to demonstrate thememory overfitting in ER for offline continual learning setting with 10 epochs.
Our results showthat applying ER in the online setting only leads to slight memory overfitting when the number ofiterations is 5 (see Fig 1 (a)) but yield severe memory overfitting for 10 iterations (see Fig 1 (b)).
Inother words, increasing the number of iterations means the loss landscape of the rehearsal memoryprovides a poorer approximation of the loss landscape of previous tasks, as shown by the differencesin the positions and the shapes between the left and right red contours in Fig 1.
As a result, thelearning trajectory of R-ER avoids the high-loss ridge region for the memory data but goes right intothe high-loss ridge region for the past tasks’ test data.
Regularization Effect of Incoming Data : To investigate why repeated rehearsal may suffer fromeven more memory overfitting than vanilla rehearsal, we analyze the regularization effects of incomingdata during repeated ER.
To this end, we examine the training process given an incoming batch andcompare the training loss on the memory batch and incoming batch with respect to memory iterationk (see Fig 2 ).
An interesting observation is that during the training session of a given incoming databatch, the decrease in the training loss on the incoming batch is much faster than the decrease inloss on the memory batches.
One intuitive explanation is that the former is computed over a fixedbatch during multiple iterations and the latter is computed over different memory batch samples.
Asa result, even though the incoming loss is larger than the memory loss at the start of a training session( k = 1 ), at later iterations (i.e., k > 5 ) the training loss of the incoming batch becomes 10 − 10 2times lower than that of the memory batch.
This means that at this stage the regularization effectof the incoming data batch is greatly undermined: the joint training on the memory batch and theincoming batch becomes similar to training on memory only.
In summary, our findings imply that the performance of online rehearsal is constrained by aunderfitting-overfitting dilemma.
Online rehearsal faces the challenge of underfitting of the large datastream but overfitting of a small memorized data subset.
Applying repeated rehearsal amelioratesthe former problem but aggravate the latter problem.
Therefore, the performance gain of repeatedrehearsal is quite limited.
Repeated Augmented Rehearsal
To deal with the underfitting-overfitting dilemma, we explore a simple strategy, “repeated augmentedrehearsal” (RAR), which combines repeated rehearsal with data augmentation.
Consider a groupof transforms G that acts on the input space X and is invariant under function f , i.e., f ( gx ) =
f
( x ) , g ∈ G, x ∈ X .
Given an incoming batch from the data stream, B t , multiple replay iterationsare conducted using this batch.
At each replay iteration n , a random memory batch B M t,k is sampledand concatenated with the incoming batch.
Then, a random transform g k,t ∈ G is sampled andapplied to each data point
x i in the concatenated minibatch.
The model parameters are updated as :
[Equation]
Intuitively, the augmentation can help alleviate memory overfitting in two ways.
First, we observeapplying augmentation on the incoming batch helps strengthen the regularization effect.
As shown in Fig 2(b), the decaying regularization effect of incoming of repeated rehearsal is alleviated in RAR,as the loss of the incoming batch stays comparable to the loss of memory batch during multipleiterations.
Second, rehearsal on augmented memory batch can help depict a more accurate past tasks’data distribution.
With RAR, the loss landscapes of memory data and past tasks’ test data becomevery similar (see Fig 1 (c)), which suggests that the model ends up in a part of the parameter spacewhere the rehearsal memory approximates the past tasks’ distribution well.
Moreover, the continuallearning solution identified with RAR avoids the high-loss ridge not only in the memory data losslandscape but also in the test data loss landscape.
M is a memory with fixed size,
B t is the incoming batch data from current task, θ are the parameters of CL network, w is the parameter of RL agent,
K is the number of memory iterations,
P, Q are the augmentation hyperparameters 1: procedure RAR( M t , B t , θ t , w t ) 2:
K t , P t , Q t = SampleAction ( w t ) 3: for k = 1 , ..., K t do 4:
B M t,k ∼ MemRetrieval ( M t ) 5:
B aug ← aug ( B M t,k ∪ B t , P t , Q t ) 6: r t ← ComputeReward ( B aug , θ t,k ) 7: θ t,k +1 ← SGD ( B aug , θ t,k ) 8: end for 9:
M t +1 ← MemUpdate ( M t , B t ) 10: w t +1 ← UpdateRL ( r t ) 11: end procedure Theoretically, we prove that augmented rehearsal reduces the generalization error in OCL. Specifically, assume the augmentation group G is a compact topological group and follows a probabilitydistribution Q . Similar to Theorem 1 , it can be easily proven that the augmented rehearsal gradientcorresponds to unbiased SGD on an augmented empirical risk (see Proposition 2 in Appendix A):¯ R t ( θ ) =
(cid:88)
[Equation] [Equation]
This result shows that applying augmented rehearsal is equivalent to performing an averagingoperation of the loss of rehearsal in Eq 3 over the orbits of a certain group that keeps the datadistribution approximately invariant.
In the standard i.i.d.
learning setting, Chen et al.
[2020] foundthat such an orbit-averaging operation can reduce both the variance and generalization error.
Based on
Eq 3 and Eq 5, we show that this theoretical benefit of using augmentation to boost model invarianceis also applicable to rehearsal in continual learning.
In fact, as discussed in Section 3.1, rehearsalin online CL has a biased empirical risk Eq 3, which leads to inherent memory overfitting andpoor generalization ability.
Thus, this benefit of augmentation in reducing generalization error isparticularly important when applying rehearsal in online CL.
The modifications required for the RAR procedure are summarized in Lines 3 and 5 of Algorithm 1.
It uses a general framework for ER-based continual learning that consists of three key components:sampling from memory, joint training of memory data and incoming data, and updating of thememory.
As mentioned in the related work section, different ER variants have been proposed toimprove these components.
RAR can flexibly be combined with any of these ER variants, and weinvestigate the effectiveness of RAR on these different ER variants in the experiment section.
Reinforcement Learning-based Adaptive Repeated Augmented Rehearsal
There are two key components in RAR: repeated rehearsal and augmented rehearsal.
The interplayof the two is determined by the number of memory iterations and the strength of augmentation.
Akey question is how to choose these hyperparameters.
In general, hyperparameter tuning (HPT) stillremains an unsolved challenge for online CL due to the single-pass assumption [Chaudhry et al.,2018].
Finding suitable RAR hyperparameters needs to account for the severity of memory overfittingand poses extra challenges.
In particular, as shown in the ERM analysis, the extent of memoryoverfitting is related to the CL problem features (e.g., task data size and memory size) and alsovaries at different training stages of the continual learning process.
To automatically select suitable
RAR hyperparameters for the different CL problems and different training stages, we propose to usereinforcement learning to adaptively adjust the hyperparameters (see Algorithm 1). [Table] [Table]
In particular
[Table]
we design the hyperparameter of RAR as the action space and use the training statisticsas the reward (see line 2, 6, 10 in Algorithm 1).
A major challenge of applying RL in online HPT issample efficiency.
The exploration horizon (i.e training steps) in the OCL environment is quite limiteddue to the constraint of single-pass of data and poor action choices (undesirable hyperparameters) maylead to bad gradient update step and hurt the OCL training process.
To address the sample efficiencyissue, we employ multi-arm bandit framework and apply bootstrapped policy gradient (BPG)
[Zhangand Goh, 2019].
The key idea of BPG is to incorporate prior knowledge to bootstrap the policygradient to achieve stable and fast convergence with limited samples.
To obtain prior knowledge in
OCL problem, we use the training accuracy on the memory batch as the stability-plasticity feedback,as a higher memory accuracy suggests a high level of stability.
Compared against a target memoryaccuracy (e.g. 0.9), the current memory accuracy is used to indicate whether the current choice ofrehearsal iteration or augmentation causes too much stability (and less plasticity) and then the actionselection probability is adjusted according following BPG (see Appendix C for more details).  Experiments
6.1 Experiment Setup
Baseline : We apply RAR to four ER-based continual learning algorithms: ER
[Chaudhry et al., 2019],
MIR [Aljundi et al., 2019], ASER [Shim et al., 2021], and SCR [Mai et al., 2021].
We also compareit with other continual learning methods, including the regularization-based method LWF [Li and
Hoiem, 2017] and the constraint optimization-based method A-GEM [Chaudhry et al., 2018].
Dataset : Four continual learning benchmarks are used in the experiments: Split-CIFAR100 (20tasks), Split-MiniImageNet (10 tasks)
Vinyals et al.
[2016], CORE50-NC (9 tasks)
Lomonaco and Maltoni [2017] and CLRS-NC (5 tasks)
Li et al.
[2020](see Appendix B for more details).
Implementation : All datasets are trained on a reduced Resnet-18 following Mai et al.
Aljundi et al.
Single-head evaluation is employed with a shared final layer trained forall the tasks.
A Nearest-Class-Mean classifier is used for evaluation following Mai et al.
RandAugmentation Cubuk et al.
[2020] is used for auto augmentation due to its simplicity.
Givena set of augmentation operations, it randomly selects P augmentation operations and exerts anaugmentation magnitude of Q for all the selected augmentation operations on each image.
All theexperimental results we present are the average of three runs.
We summarize all hyperparameterdetails in Appendix C. Supplementary results using Resnet-18 can be found in Appendix D.3.
Therunning time of different algorithms is shown in Appendix D.6. [Figure]
6.2 Main Results
We first analyze RAR’s performance with a pre-defined hyperparameter set ( K = 10 , P = 1 , Q =14 ).As
shown in Table 1, RAR greatly improves the ER method on the four datasets, by 8.7% - 18.8%.
Moreover, RAR also leads to substantial gains for the other algorithmic variants of ER for all datasets(MIR: 9.0% - 17.7%, ASER: 7.2% - 14.7%, SCR: 5.1% - 17.9%).
These results suggest that evenwith advanced memory management, OCL still benefits a lot from repeated augmented rehearsal.
Stability and Plasticity Trade-off Based on the definition of accuracy A T , forgetting F T and backward transfer B T in section 2, we find that these three metrics have the following relationship:
[Equation]
Interestingly, this finding shows that accuracy is related to the ability of learning the new task1 T (cid:80)
Ti =1 a
i,i and the forgetting of past tasks, which draws a connection to the more general problemof the stability-plasticity trade-off in neural networks and continual learning [Grossberg, 2012,
Delange et al., 2021].
Plasticity refers to the ability to integrate new knowledge and stability refersto the ability to retain old knowledge.
Fig 3 presents the stability and plasticity trade-off in RAR.
Generally, we observe increasing the repeated rehearsal iterations ( K ) leads to a higher level ofplasticity.
However, this may also cause a decrease of stability, i.e., introducing forgetting.
Onthe other hand, the use of augmentation generally improves stability.
More importantly, the use ofaugmentation in repeated rehearsal shifts the stability-plasticity trade-off curve towards the upperright thus creating a better stability-plasticity trade-off frontier.
Hyperparameter Tuning for RAR We compare the RL based hyperparameter tuning method withthe hyperparameter tuning framework for continual learning (HTOCL) method used in Chaudhryet al.
[2018], Mai et al.
[2022] (see Section 2).
The results in Table 2 show that RL-based RARsignificantly outperforms using HTOCL to select hyperparameters for RAR.
One reason is that theextent of memory overfitting varies at the different training stages of CL.
HTOCL only uses the firstfew tasks to select hyperparameters for RAR, which may not optimal for later stages of CL.
In fact,we observe HTOCL tends to select a large number of repeats and small augmentation strength.
Thisselection strategy may be desirable for problems with short task sequences but problematic for longtask sequences with increased memory overfitting risk.
In contrast, RL-based method can take intoaccount the latest feedback (e.g. train memory accuracy) to adjust hyperparameter choices.
Theselected iteration numbers and augmentation are shown in Appendix D.5.
6.3 Ablation Studies
Interplay between Repeated and Augmented Rehearsal We investigate the interaction of thenumber of replay iterations N with the augmentation strength in Fig 4.
For three out of fourdatasets, using augmentation with rehearsal leads to even worse performances than rehearsal withoutaugmentation (see K = 1 in Fig 4).
One explanation is the underfitting challenge of online CL.
Training on augmented samples can make model underfitting even worse.
The only exception is the
CORE50 dataset, which inherently has a high memory overfitting risk with λ = 6 making it benefitmore from augmentation.
Similarly, employing repeated rehearsal alone also harms performancein three out of four datasets, with the CLRS dataset as the only exception, which enjoys a lowmemory overfitting risk with λ = 1 . 12 .
An important takeaway message is that repeated rehearsal oraugmented rehearsal is not always helpful in OCL settings and whether they will benefit or harm theperformance is dependent on the structure of the OCL problem at hand (e.g., the task data size andmemory data size).
RAR’s Robustness to Large Numbers of Repeats Although the performance curve flattens outaround 10 iterations, there is no evident drop in performance even with 20 iterations.
This resultreinforces how RAR can help with the underfitting-overfitting dilemma: it can support the use ofmore training in OCL without having to worry about the performance drop introduced by memoryoverfitting.
In comparison, for repeated rehearsal without augmentation (solid lines), the accuracystarts to drop quickly when using more than two iterations in CIFAR100.
Augmenting the Memory vs. Augmenting the Incoming Data RAR applies augmentation to boththe memory batch and the incoming batch.
We examine the effectiveness of the two separately.
Table 2 presents the performance of RAR applied (a) solely with memory augmentation, RARmem and (b) solely with incoming data augmentation, RAR-inc.
We find that RAR achieves thebest performance compared to RAR-mem and RAR-inc.
This shows that it is beneficial to applyaugmentation to both the memory batch and the incoming batch.
Interestingly, RAR-inc itself alsoachieves consistent performance improvements over R-ER, e.g. a gain of 7.7% in CORE50.
Asdiscussed in Section 3, adding augmentation on the incoming batch can strengthen the regularizationeffect of the incoming task and indirectly alleviate memory overfitting.
RAR with MIR, ASER,
SCR We also perform ablation studies to investigate RAR’s strong performance with the ER variants (see Appendix D.2).
Similar to ER, the performance gains for RAR-MIRand RAR-ASER come from the combination of repeated rehearsal and augmented rehearsal.
However,for SCR, the repeated rehearsal itself also leads to a consistent performance boost.
One reason is that
SCR already includes a strong augmentation procedure with four augmentation operations to constructthe supervised contrastive loss used in this method.
It also works poorly without augmentation.
Discussion and Conclusion
Rehearsal-based methods play a central role in fighting catastrophic forgetting when learning fromnon-stationary data streams.
However, apart from empirical discussions, what loss function rehearsalbased methods are actually minimizing has not been well studied in the literature. Our theoreticalstudy reveals that rehearsal optimizes a biased and dynamic objective. This explains the inherentmemory overfitting risk of rehearsal and suggests the extent of memory overfitting is dependenton the OCL problem attributes and varies at different training stages.
Based on the biased natureof rehearsal-based optimization, we design a simple baseline algorithm: repeated and augmentedrehearsal (RAR), which achieves a large performance boost for different OCL problems.
Additionally,to account for the dynamic nature of rehearsal-based optimization, we propose a RL-based methodto tune the hyperparameters of RAR to balance the stability-plasticity trade-off online.
It achievespromising results compared to the validation data based hyperparameter tuning.
A limitation of this
RL-based framework is that it is focused on the hyperparameters of RAR.
A remaining question ishow to apply RL to tune different types of OCL hyperparameters of jointly.
Also, the non-stationarynature of the OCL environment poses special challenge for RL. Another interesting future directionis to explore non-stationary bandits techniques to address this.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
b) Did you mention the license of the assets?
[Yes]
) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table] ) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table] you crowdsourcing or conducted research with human subjects [Table]
you include the full text of instructions given to participants and screenshots,
[Table]
[Table]
you describe any potential participant risks,
to Institutional Review
approvals,
[-At the start of a task,β t =
-]
applicable
Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N
A]