Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Online continual learning (OCL) aims to train neural networks incrementally froma non-stationary data stream with a single pass through data.
Rehearsal-based methods attempt to approximate the observed input distributions over time with a smallmemory and revisit them later to avoid forgetting.
Despite its strong empirical performance, rehearsal methods still suffer from a poor approximation of past data’sloss landscape with memory samples.
This paper revisits the rehearsal dynamics inonline settings.
We provide theoretical insights on the inherent memory overfittingrisk from the viewpoint of biased and dynamic empirical risk minimization, and examine the merits and limits of repeated rehearsal.
Inspired by our analysis, a simpleand intuitive baseline, Repeated Augmented Rehearsal (RAR), is designed to address the underfitting-overfitting dilemma of online rehearsal.
Surprisingly, acrossfour rather different OCL benchmarks, this simple baseline outperforms vanillarehearsal by 9%-19% and also significantly improves state-of-the-art rehearsalbased methods MIR, ASER, and SCR.
We also demonstrate that RAR successfullyachieves an accurate approximation of the loss landscape of past data and high-lossridge aversion in its learning trajectory.
Extensive ablation studies are conducted tostudy the interplay between repeated and augmented rehearsal and reinforcementlearning (RL) is applied to dynamically adjust the hyperparameters of RAR tobalance the stability-plasticity trade-off online.
Introduction
Despite its recent success, deep learning largely relies on the assumption of independent and identically distributed (i.i.d.) data that can be repeatedly revisited during training.
Non-i.i.d settings arechallenging for neural networks due to catastrophic forgetting: previously learned knowledge caneasily be overwritten when training on new data since this data may follow a different distribution [Liand Hoiem, 2017, Rebuffi et al., 2017, Delange et al., 2021].
Online continual learning (OCL or
Online CL) studies how to enable deep learning in an online manner from a non-stationary datastream.
As the data stream can be vast or even infinite, it is infeasible to store and shuffle the datasetfor multiple epochs of training.
Therefore, a fundamental assumption is that the data stream can onlybe accessed one batch at a time and training is performed with a single pass over the data [Aljundiet al., 2019].
Experience replay (ER), also known as rehearsal [Chaudhry et al., 2019, Delange et al., 2021], is a keymethod in OCL.
It stores a subset of previously seen data D in a fixed-size memory M and revisitsthe memorized samples during training to mitigate forgetting of previous tasks.
To update the model,a batch sampled from the memory is combined with the incoming batch from the stream to computethe gradient [Chaudhry et al., 2019].
Different variants of ER have been developed to improve memory sampling policies and representation learning, achieving state-of-the-art performance in anumber of standard OCL benchmarks [Aljundi et al., 2019, Mai et al., 2021, Shim et al., 2021].
However, whether rehearsal is appropriate for continual learning, considering the risk of overfitting thememory when using data from the memory to directly contribute to the gradient computation, has beendebated vigorously [Lopez-Paz and Ranzato, 2017, Chaudhry et al., 2019, Verwimp et al., 2021].
Thepotential for overfitting has motivated the development of constraint-based replay methods [LopezPaz and Ranzato, 2017], e.g. GEM and A-GEM, which use memory samples solely to constrain thegradient direction.
However, there is empirical evidence suggesting that rehearsal-based methodsconsistently outperform methods that do not train directly on the memory [Chaudhry et al., 2019].
This indicates rehearsal on the memory does not necessarily prevent effective generalization, possiblydue to the implicit regularization effect of incoming data [Chaudhry et al., 2019].
Nevertheless,recent work analyzing the loss landscape when applying rehearsal to offline continual learning findsthat memory samples indeed provide a poor approximation of the loss landscape past task’s data,especially near a high-loss ridge region.
As a result, " instead of ending up near the high-loss ridgein perspective of the rehearsal memory, the solution in reality resides on the high-loss ridge forthe training data " [Verwimp et al., 2021].
This latest finding poses the question of how to betterapproximate the loss surface of past data L ( D ; θ ) with memory samples’ loss L ( M ; θ ) .
To better approximate past data’s loss distribution, previous work studies which samples should bememorized for rehearsal.
Instead, we study how to effectively perform rehearsal with the memorizedsamples.
Focused on online CL, our study extends the previous understanding of rehearsal along twodirections.
First, we provide theoretical considerations that reveal two insights regarding the extent ofoverfitting to memory: it is a) related to the inherent attributes of the OCL problem concerned and b)varies across the different stages of continual learning.
Second, we highlight the limits of applyingrehearsal with multiple iterations—a trick used to maximally utilize the incoming batch [Aljundiet al., 2019]—and identify an underfitting-overfitting dilemma for online rehearsal.
Based on our analysis, we design a simple baseline to deal with the underfitting-overfitting dilemmain online CL problems, dubbed repeated augmented rehearsal (RAR), that can be easily integratedinto existing rehearsal-based methods.
Surprisingly, this simple baseline leads to a large performanceboost for ER, as well as state-of-the-art ER-based approaches, across four OCL benchmarks.
Moreimportantly, the loss landscape analysis shows that RAR can help memory samples reliably approximate the distribution of past data and successfully avoids the high-loss ridge of past tasks.
To betterunderstand the behavior of RAR, we further investigate the interplay between repeated rehearsal andaugmented rehearsal via an ablation study.
We also propose a reinforcement learning-based methodto dynamically adjust the hyperparameters of RAR and balance the stability-plasticity trade-off in anonline manner.
Related Work
Online continual learning : We consider the online continual learning setting with a non-stationary(potentially infinite) stream of data: at each time step t , the continual learning agent receivesan incoming batch of data samples B t = { x i , y i } i =1 ,.., |B t | that are drawn from the current data 74 distribution D t .
The period of time where the data distribution stays the same is often called a task orexperience in the continual learning literature.
Abrupt change in the data distribution occurs whenthe task changes.
The standard objective during training is to minimize the empirical risk on all thedata seen so far:
[Equation]
(1) with loss function L , the CL network function f , and its associated parameters θ .
Metrics : A common metric is the end accuracy after training on T tasks, defined as A T =1 T (cid:80) j = T j =1 a T,j , where a i,j denotes the model’s accuracy on the held-out test set of task j aftertraining on task i .
Other metrics are “forgetting” [Chaudhry et al., 2018], which is defined as
F T = − 1 T − 1 (cid:80) T − 1 i =1 ( a T,i − max l ∈ 1 ...
T − 1 a l,i ) and the related metric "backward transfer" [LopezPaz and Ranzato, 2017]: B T = 1 T − 1 (cid:80) T − 1 i =
a T,i − a i,i .
Online setting : A key difference between online and offline CL is that the latter assumes full accessto the whole training data for the task that is currently being processed.
Therefore, it allows training
[Figure]
on each single task with multiple epochs (e.g., 70-200 epochs) [Rebuffi et al., 2017, Wu et al., 2019,
Cha et al., 2021].
The online CL setting is more challenging because the agent can only access thecurrent batch of incoming data and performs training with a single pass through the data.
Experience Replay (Rehearsal) : Chaudhry et al.
(2019) propose experience replay (ER), whichperforms joint training on memory samples and incoming samples.
A simple but strong baselineapproach to sampling in ER is reservoir sampling Vitter [1985].
Aljundi et al.
(2019) propose
Maximally Interfered Retrieval (MIR), which retrieves the samples that will be most negativelyimpacted by the foreseen parameters update.
Shim et al.
(2021) propose ASER, which selectssamples to best preserve existing memory-based class boundaries.
In terms of model training, Mai etal.
(2021) propose to replace the cross-entropy loss with supervised contrastive loss to learn a betterrepresentation.
We consider all these variants of ER in our experiments.
Augmentation :
In the standard i.i.d.
setting, data augmentation is a widely used method to improvedeep learning [Cubuk et al., 2020].
In the offline CL setting, Mai et al.
[2021] uses augmentation toconstruct a supervised contrastive loss and Bang et al.
[2021] employs it together with an uncertaintybased memory management strategy.
However, these papers apply augmentation together with otheradvanced techniques, and there is no ablation study on the effect of augmentation per se.
Thus, it isunclear whether augmentation itself helps rehearsal or not, especially in the online setting.
Repeated Rehearsal (Multiple Iterations) : Vanilla online continual learning employs a single gradientupdate given an incoming batch of data.
To maximally utilize the current incoming batch, Aljundiet al.
[2019] proposes to perform multiple gradient updates instead.
Their experiment with CIFAR10shows using MIR with 5 iterations leads to a slight accuracy improvement of 1.7%.
In this paper, wesystematically analyze the effect of multiple iterations on online rehearsal and provide theoreticaland empirical insights on when this trick may improve or harm performance.
Hyperparameter Tuning for OCL :
Hyperparameter tuning is a particular challenge in OCL due tothe lack of a dedicated validation set and the constraint of a single pass through the data.
Chaudhryet al.
[2018] and Mai et al.
[2022] employ a hyperparameter tuning protocol that uses an externalvalidation data stream with a small number of tasks.
Offline hyperparameter tuning is applied to thisvalidation data with multiple passes to identify optimal values, which are then used for the actualonline continual learning tasks.
A limitation of this method is that it relies on external validation data.
Revisiting Online Rehearsal: Is Repeated Rehearsal a Good Idea?
We revisit rehearsal from two directions.
First, while previous work demonstrates its strong empiricalperformance [Chaudhry et al., 2019] and provides conceptual analysis for offline CL [Verwimp et al.
,2021], we focus on the online setting and provide theoretical insights through the lens of empiricalrisk minimization (ERM).
Second, we examine the dynamics of employing rehearsal with multipleiterations.
This has been proposed as a trick for online CL to maximally utilize the incoming batchalready [Mai et al., 2022, Aljundi et al., 2019]; we investigate whether it is always better thanrehearsal with a single iteration.
3.1 Empirical Risk Minimization in Online Rehearsal: a Biased and Dynamic Objective
For rehearsal in OCL, at each iteration t , a batch of data B t is obtained from the incoming task, where
B t ∼ D T and B t
= { x i , y i } i =1 ,... |B
t | , and a batch B M t is sampled from memory, where B M t ∼ D t M and B M t = { x i , y i } i =1 ,... |B
M t | .
The gradient update rule of ER is:
[Equation]
Given this update rule, we would like to establish the corresponding objective function, but this isnot straightforward to derive because the memory is immediately updated after each incoming batch,which means D t M changes all the time.
For a widely used memory management policy, where thememory is updated using reservoir sampling [Vitter, 1985, Chaudhry et al., 2019], we prove in theappendix that the empirical risk for online ER follows Proposition 1.
Proposition 1 (ERM for online rehearsal) : Assume an incoming task stream D T and an initialmemory set D 0 M with different data distribution P
( D T ) ̸
= P ( D M ) .
Assume further that the memoryis updated at the end of each iteration using reservoir sampling.
Then, Eq 2 implements unbiasedstochastic gradient descent for the following loss function:
[Equation]
where λ :=
|D T | |D 0 M | and |D
M | and |D T | are the memory size and incoming task data size; β t :
=1 / (1 + 2 N tcur N T past ) , and N tcur =
(cid:80) i
= t i =1 |B i | denotes the number of samples of the current task thathave been seen so far and N T past =
(cid:80)
j = T j =1 |D j | denotes the number of samples of past tasks.
Weimmediately have β t ∈ (0 , 1] .
The proposition reveals several interesting properties: 1) Bias : compared with the true objectivein continual learning in Eq 1, the loss of online ER in Eq 3 is actually a biased approximation ofthe former, as it puts a different weight ( β t λ ) on memory samples while the true objective treatsall samples with equal weight.
This bias, introduced by ER’s objective function, can contribute tothe risk of memory overfitting.
2) Problem-dependence : given that β t ∈ (0 , 1] , the biased weighton the memory samples is heavily influenced by λ .
In other words, the memory overfitting risk isrelated to an inherent property of the CL problem concerned: the ratio λ between the current taskdata size and the memory data size.
While previous works extensively report empirical results on theinfluence of memory size, to our knowledge, we are the first to point out that the relative data sizeof an incoming task also plays an important role.
Empirical evidence is provided in Section 6.3 tosupport this claim.
With a 2k memory, performing rehearsal on the CORE50 dataset with a largertask data size ( λ = 6 ) faces a high level of memory overfitting while the CLRS dataset with a smallertask data size ( λ = 1 . 12 ) enjoys a lower risk of memory overfitting.
3) Dynamic : ER in online CLoptimizes towards a dynamic objective, which varies with each incoming batch t , as the weight on thememory sample depends on N tcur , and N T past .
This analysis shows that memory overfitting may berelatively slight for the first few tasks when N T past is small.
As more tasks arrive, memory overfittingworsens as β t increases with N T past .
In the case of an infinite data stream, lim N T past →∞ β t = 1 , thememory weight is solely determined by λ .
3.2 Repeated Rehearsal: The Decaying Regularization Effect of Incoming Data
We now investigate whether and why performing multiple iterations is indeed beneficial to rehearsal.
We refer to applying multiple iterations in ER as “repeated experience replay” (Repeated ER) or“repeated rehearsal” and formalize it as follows: for each incoming data batch B t ∼ P t from the datastream, we perform multiple gradient updates ( K in total) using stochastic gradient descent (SGD) orvariants thereof.
At each gradient update k = 1 , ...
K , a data batch B M t,k is chosen from memory Mand concatenated with the current incoming batch B t to perform a replay iteration as follows.
[Equation]
Note that when k = 1 , this update rule is an unbiased stochastic gradient for Eq3; in the case ofk > 1 , it is a biased gradient estimate as the same incoming batch is used for consecutive gradientupdates.
To study the influence of this biased gradient update in repeated rehearsal, we examine theinternal dynamics of repeated rehearsal by studying the loss landscape.
Memory Overfitting :
We compare the loss surfaces regarding the memory samples and the test dataof past tasks during repeated rehearsal.
Following the visualization method used in Verwimp et al.
[2021], Mirzadeh et al.
[2020], we examine the learning process on the first two tasks in the Split
Mini-ImageNet dataset and plot the loss landscape in the 2D plane defined by three model parametervectors 1 : the model w 1 obtained by training on the first task until convergence, the model w 2 obtainedon the second task using experience replay, and the model w 2 ,ft obtained after training on the secondtask using finetuning without replay.
Verwimp et al.
(2021) use this method to demonstrate thememory overfitting in ER for offline continual learning setting with 10 epochs.
Our results show thatapplying ER in the online setting yields severe memory overfitting for 10 iterations (see Fig 1 (a)).
Inother words, increasing the number of iterations means the loss landscape of the rehearsal memoryprovides a poorer approximation of the loss landscape of previous tasks, as shown by the differencesin the positions and the shapes between the left and right red contours in Fig 1 (a).
As a result, thelearning trajectory of R-ER avoids the high-loss ridge region for the memory data but goes right intothe high-loss ridge region for the past tasks’ test data.
Regularization Effect of Incoming Data : To investigate why repeated rehearsal may suffer fromeven more memory overfitting than vanilla rehearsal, we analyze the regularization effects of incomingdata during repeated ER.
To this end, we examine the training process given an incoming batch andcompare the training loss on the memory batch and incoming batch with respect to memory iterationk (see Fig 2 ).
An interesting observation is that during the training session of a given incoming databatch, the decrease in the training loss on the incoming batch is much faster than the decrease inloss on the memory batches.
One intuitive explanation is that the former is computed over a fixedbatch during multiple iterations and the latter is computed over different memory batch samples.
Asa result, even though the incoming loss is larger than the memory loss at the start of a training session( k = 1 ), at later iterations (i.e., k > 5 ) the training loss of the incoming batch becomes 10 − 10 2times lower than that of the memory batch.
This means that at this stage the regularization effectof the incoming data batch is greatly undermined: the joint training on the memory batch and theincoming batch becomes similar to training on memory only.
In summary, our findings imply that the performance of online rehearsal is constrained by the dilemmabetween overfitting locally and underfitting globally.
Specifically, online rehearsal faces the challengeof underfitting of the large data stream but overfitting of a small memorized data subset.
Applyingrepeated rehearsal ameliorates the former problem but aggravate the latter problem.
Therefore, theperformance gain of repeated rehearsal is quite limited.
Repeated Augmented Rehearsal
To deal with the overfitting-underfitting dilemma, we explore a simple strategy, “repeated augmentedrehearsal” (RAR), which combines repeated rehearsal with data augmentation.
Consider a groupof transforms G that acts on the input space X and is invariant under function f , i.e., f ( g x ) =
f
( x ) , g ∈ G, x ∈ X .
Given an incoming batch from the data stream, B t , multiple replay iterationsare conducted using this batch.
At each replay iteration n , a random memory batch B M t,k is sampledand concatenated with the incoming batch.
Then, a random transform g k,t ∈ G is sampled andapplied to each data point
x i in the concatenated minibatch.
The model parameters are updated as :
[Equation]
Intuitively, the augmentation can help alleviate memory overfitting in two ways.
First, we observeapplying augmentation on the incoming batch helps strengthen the regularization effect.
As shown in
Fig 2(b), the decaying regularization effect of incoming data is alleviated in RAR, as the loss of theincoming batch stays comparable to the loss of memory batch during multiple iterations.
Second,rehearsal on augmented memory batch can help depict a more accurate past tasks’ data distribution.
With RAR, the loss landscapes of memory data and past tasks’ test data become very similar (see
Fig 1 (b)), which suggests that the model ends up in a part of the parameter space where the rehearsalmemory approximates the past tasks’ distribution well.
Moreover, the continual learning solutionidentified with RAR avoids the high-loss ridge not only in the memory data loss landscape but also inthe test data loss landscape.
Theoretically, we prove that augmented rehearsal reduces the generalization error in OCL.
Specifically, assume the augmentation group G is a compact topological group and follows a probability
M is a memory with fixed size,
B t is the incoming batch data from current task, θ are the parameters of CL network, w is the parameter of RL agent,
K is the number of memory iterations,
P, Q are the augmentation hyperparameters 1: procedure RAR( M t , B t , θ t , w t ) 2:
K t , P t , Q t = SampleAction ( w t ) 3: for k = 1 , ..., K t do 4:
B M t,k ∼ MemRetrieval ( M t ) 5:
B aug ← aug ( B M t,k ∪ B t , P t , Q t ) 6: r t ← ComputeReward ( B aug , θ t,k ) 7: θ t,k +1 ← SGD ( B aug , θ t,k ) 8: end for 9:
M t +1 ← MemUpdate ( M t , B t ) 10: w t +1 ← UpdateRL ( r t ) 11: end procedure distribution Q .
Similar to Proposition 1 , it can be easily proven that the augmented rehearsal gradientcorresponds to unbiased SGD on an augmented empirical risk (see Proposition 3 in Appendix A):
[Equation]
This result shows that applying augmented rehearsal is equivalent to performing an averagingoperation of the loss of rehearsal in Eq 3 over the orbits of a certain group that keeps the datadistribution approximately invariant.
In the standard i.i.d.
learning setting, Chen et al.
[2020] foundthat such an orbit-averaging operation can reduce both the variance and generalization error.
Based on
Eq 3 and Eq 5, we show that this theoretical benefit of using augmentation to boost model invarianceis also applicable to rehearsal in continual learning.
In fact, as discussed in Section 3.1, rehearsalin online CL has a biased empirical risk Eq 3, which leads to inherent memory overfitting andpoor generalization ability.
Thus, this benefit of augmentation in reducing generalization error isparticularly important when applying rehearsal in online CL.
The modifications required for the RAR procedure are summarized in Lines 3 and 5 of Algorithm 2.
It uses a general framework for ER-based continual learning that consists of three key components:sampling from memory, joint training of memory data and incoming data, and updating of thememory.
As mentioned in the related work section, different ER variants have been proposed toimprove these components.
RAR can flexibly be combined with any of these ER variants, and weinvestigate the effectiveness of RAR on these different ER variants in the experiment section.
Reinforcement Learning-based Adaptive Repeated Augmented Rehearsal
There are two key components in RAR: repeated rehearsal and augmented rehearsal.
The interplayof the two is determined by the number of memory iterations and the strength of augmentation.
Akey question is how to choose these hyperparameters.
In general, hyperparameter tuning (HPT) stillremains an unsolved challenge for online CL due to the single-pass assumption [Chaudhry et al.,2018].
Finding suitable RAR hyperparameters needs to account for the severity of memory overfittingand poses extra challenges.
In particular, as shown in the ERM analysis, the extent of memoryoverfitting is related to the CL problem features (e.g., task data size and memory size) and alsovaries at different training stages of the continual learning process.
To automatically select suitable
RAR hyperparameters for the different CL problems and different training stages, we propose to usereinforcement learning to adaptively adjust the hyperparameters (see Algorithm 2).
In particular, we design the hyperparameter of RAR as the action space and use the training statisticsas the reward (see line 2, 6, 10 in Algorithm 1).
A major challenge of applying RL in online HPT is
[Table] [Table]
.4 ± 0.
38.6 ± 0.
28.8 ± 1.0 RAR - INC 21.6 ± 0.
24.5 ± 0.
35.7 ± 1.
29.1 ± 1.1 RAR - BOTH 27.8 ± 0.
30.0 ± 0.
39.3 ± 1.
28.8 ± 1.5 RAR-HTOCL 23.4 ± 0.
26.0 ± 0.
40.8 ± 0.
26.9 ± 0.5 RAR-RL 29.6 ± 0.
32.1 ± 1.
44.4 ± 0.
35.0 ± 0.
sample efficiency.
The exploration horizon (i.e training steps) in the OCL environment is quite limiteddue to the constraint of single-pass of data and poor action choices (undesirable hyperparameters) maylead to bad gradient update step and hurt the OCL training process.
To address the sample efficiencyissue, we employ multi-arm bandit framework and apply bootstrapped policy gradient (BPG)
[Zhangand Goh, 2019].
The key idea of BPG is to incorporate prior knowledge to bootstrap the policygradient to achieve stable and fast convergence with limited samples.
To obtain prior knowledge in
OCL problem, we use the training accuracy on the memory batch as the stability-plasticity feedback,as a higher memory accuracy suggests a high level of stability.
Reward is defined as the distancebetween the current memory accuracy and target memory accuracy.
Compared against a targetmemory accuracy (e.g. 0.9), the current memory accuracy is used to indicate whether the currentchoice of rehearsal iteration or augmentation causes too much stability (and less plasticity) and thenthe action selection probability is adjusted according following BPG (see Appendix C.2 for more RLdesign and implementation details).
Experiments
6.1 Experiment Setup
Baseline : We apply RAR to four ER-based continual learning algorithms: ER
[Chaudhry et al., 2019],
MIR [Aljundi et al., 2019], ASER [Shim et al., 2021], and SCR [Mai et al., 2021].
We also compareit with other continual learning methods, including the regularization-based method LWF [Li and
Hoiem, 2017] and the constrained optimization-based method A-GEM [Chaudhry et al., 2018].
Dataset : Four CL benchmarks are used in the experiments: Seq-CIFAR100 (20 tasks), SeqMiniImageNet (10 tasks)
Vinyals et al.
[2016], CORE50-NC (9 tasks)
Lomonaco and Maltoni[2017] and CLRS25-NC (5 tasks)
Li et al.
[2020](see Appendix B for more details).
Additionally, wealso investigate ER and RAR in a large-scale ImageNet-1k dataset in Appendix!D.4.
Implementation : All datasets are trained on a reduced ResNet-18 following Mai et al.
[2021], Aljundiet al.
Single-head evaluation is employed with a shared final layer trained for all the tasks.
RandAugmentation Cubuk et al.
[2020] is used for auto augmentation.
Given a set of augmentationoperations, it randomly selects P augmentation operations and exerts an augmentation magnitudeof Q for all the selected augmentation operations on each image.
All the experimental results wepresent are the average of three runs.
We summarize all hyperparameter details in Appendix C.
Supplementary results using ResNet-18 can be found in Appendix D.5.
The running time of differentalgorithms is shown in Appendix D.8.
6.2 Main Results
We first analyze RAR’s performance with a pre-defined hyperparameter set ( K = 10 , P = 1 , Q =14 ).As
shown in Table 1, RAR greatly improves the ER method on the four datasets, by 8.7% - 18.8%.
Moreover, RAR also leads to substantial gains for the other algorithmic variants of ER for all datasets(MIR: 9.0% - 17.7%, ASER: 7.2% - 14.7%, SCR: 5.1% - 17.9%).
These results suggest that evenwith advanced memory management, OCL still benefits a lot from repeated augmented rehearsal.
Stability and Plasticity Trade-off Based on the definition of accuracy A T , forgetting F T and backward transfer B T in section 2, we find that these three metrics have the following relationship:
[Equation]
Interestingly, this finding shows that accuracy is related to the ability of learning the new task1 T (cid:80)
Ti =1 a
i,i and the forgetting of past tasks, which draws a connection to the more general problemof the stability-plasticity trade-off in neural networks and continual learning [Grossberg, 2012,
Delange et al., 2021].
Plasticity refers to the ability to integrate new knowledge and stability refersto the ability to retain old knowledge.
Fig 3 presents the stability and plasticity trade-off in RAR.
Generally, we observe increasing the repeated rehearsal iterations ( K ) leads to a higher level ofplasticity.
However, this may also cause a decrease of stability, i.e., introducing forgetting.
Onthe other hand, the use of augmentation generally improves stability.
More importantly, the use ofaugmentation in repeated rehearsal shifts the stability-plasticity trade-off curve towards the upperright thus creating a better stability-plasticity trade-off frontier.
Hyperparameter Tuning for RAR We compare the RL based hyperparameter tuning method withthe hyperparameter tuning framework for continual learning (HTOCL) method used in Chaudhryet al.
[2018], Mai et al.
[2022] (see Section 2).
The results in Table 2 show that RL-based RARsignificantly outperforms using HTOCL to select hyperparameters for RAR.
One reason is that theextent of memory overfitting varies at the different training stages of CL.
HTOCL only uses the firstfew tasks to select hyperparameters for RAR, which may not optimal for later stages of CL.
In fact,we observe HTOCL tends to select a large number of repeats and small augmentation strength.
Thisselection strategy may be desirable for problems with short task sequences but problematic for longtask sequences with increased memory overfitting risk.
In contrast, RL-based method can take intoaccount the latest feedback (e.g. train memory accuracy) to adjust hyperparameter choices.
Theselected iteration numbers and augmentation are shown in Appendix D.7.
RAR with Modified Rehearsal Loss Besides using the vanilla online rehearsal loss in Eq 3, weinvestigate the effectiveness of RAR with two advanced rehearsal loss designs: 1) Reweightedmemory loss (ER-rw): one straightforward way to deal with the biased ER loss is to balance theweight of the memory loss and incoming loss by introducing a reweighting hyperparameter α inthe gradient of Eq 2. 2) Distillation-based memory loss : DER [Buzzega et al., 2020] employs thelogits-based distillation loss for memory samples, instead of the cross entropy loss.
The results in
Table 1 show RAR leads to large performance gains for ER-rw (for the best α choice; further resultscan be found in Appendix D.3)) and DER for all four datasets.
This suggests that even with advancedrehearsal loss design, repeated augmented rehearsal is important for online rehearsal.
6.3 Ablation Studies
Interplay between Repeated and Augmented Rehearsal We investigate the interaction of thenumber of replay iterations N with the augmentation strength in Fig 4.
For three out of four
[Figure]
datasets, using augmentation with rehearsal leads to even worse performances than rehearsal withoutaugmentation (see K = 1 in Fig 4).
One explanation is the underfitting challenge of online CL.
Training on augmented samples can make model underfitting even worse.
The only exception is the
CORE50 dataset, which inherently has a high memory overfitting risk with λ = 6 making it benefitmore from augmentation.
Similarly, employing repeated rehearsal alone also harms performancein three out of four datasets, with the CLRS dataset as the only exception, which enjoys a lowmemory overfitting risk with λ = 1 . 12 .
An important takeaway message is that repeated rehearsal oraugmented rehearsal is not always helpful in OCL settings and whether they will benefit or harm theperformance is dependent on the structure of the OCL problem at hand (e.g., the task data size andmemory data size).
RAR’s Robustness to Large Numbers of Repeats Although the performance curve flattens outaround 10 iterations, there is no evident drop in performance even with 20 iterations.
This resultreinforces how RAR can help with the underfitting-overfitting dilemma: it can support the use ofmore training in OCL without having to worry about the performance drop introduced by memoryoverfitting.
In comparison, for repeated rehearsal without augmentation (solid lines), the accuracystarts to drop quickly when using more than two iterations in CIFAR100.
Augmenting the Memory vs. Augmenting the Incoming Data RAR applies augmentation to boththe memory batch and the incoming batch.
We examine the effectiveness of the two separately.
Table 2 presents the performance of RAR applied (a) solely with memory augmentation, RARmem and (b) solely with incoming data augmentation, RAR-inc.
We find that RAR achieves thebest performance compared to RAR-mem and RAR-inc.
This shows that it is beneficial to applyaugmentation to both the memory batch and the incoming batch.
Interestingly, RAR-inc itself alsoachieves consistent performance improvements over R-ER, e.g. a gain of 7.7% in CORE50.
Asdiscussed in Section 3, adding augmentation on the incoming batch can strengthen the regularizationeffect of the incoming task and indirectly alleviate memory overfitting.
RAR with MIR, ASER,
SCR We also perform ablation studies to investigate RAR’s strong performance with the ER variants (see Appendix D.2).
Similar to ER, the performance gains for RAR-MIRand RAR-ASER come from the combination of repeated rehearsal and augmented rehearsal.
However,for SCR, the repeated rehearsal itself also leads to a consistent performance boost.
One reason is that
SCR already includes a strong augmentation procedure with four augmentation operations to constructthe supervised contrastive loss used in this method.
It also works poorly without augmentation.
Discussion and Conclusion
Rehearsal-based methods play a central role in fighting catastrophic forgetting when learning fromnon-stationary data streams.
Compared to offline rehearsal, online rehearsal has faced particularchallenges in tackling complex CL datasets due to the single-pass-through data constraint.
This worktries to analyze the internal workings of online rehearsal from theoretical and conceptual perspectiveand identifies the fundamental challenge that it faces as the dilemma between overfitting locallyand underfitting globally.
To deal with this challenge, we propose a simple baseline: repeated andaugmented rehearsal (RAR).
Surprisingly, despite its simplicity, RAR achieves a large performanceboost for a set of different rehearsal-based methods.
Additionally, we propose a RL-based methodto tune the hyperparameters of RAR to balance the stability-plasticity trade-off online.
It achievespromising results compared to the validation data based hyperparameter tuning.
A remaining questionis how to apply RL to tune different types of OCL hyperparameters of jointly.
Also, this work isfocused on the continual learning for classification problems with image data.
An interesting futureresearch direction is to look at other CL domains, e.g
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that theChecklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Theoretical Analysis
This section contains the theoretical analysis of the loss function of offline experience replay( Proposition 2 ), augmented experience replay ( Proposition 3 ) and online experience replay withreservoir sampling ( Proposition 1 ).
Proposition 2 (Empirical risk minimization for experience replay) :
We assume a memory set
D M and an incoming task stream D T with different data distribution P ( D T )
̸
= P ( D M ) .
At each iteration t , a batch of data is sampled from memory B M t ∼ D M and B M t = { x i , y i } i =1 ,... |B
M t | , and a batch of data from incoming task B t ∼ D T and B t = { x i , y i } i =1 ,... |B
t | .
To update the parameter θ of a function mapping f θ : f θ × X → Y based on loss function L , consider a parameter update rule defined by θ = θ − η |B t | (cid:80)
x
i ,y i ∈B t ∇L ( f θ ( x i ) , y i ) −
[Equation]
|
[Equation]
x i ,
[Equation]
( f θ ( x
[Equation]
) , y i ) , then it is an unbiased stochastic gradient descent for the following empirical risk:
[Equation]
Proof :
Given the empirical gradient
[Equation]
x
[Equation]
) , y
[Equation]
) +1 | B M t | (cid:80) x i ,y i ∈B M t ∇L ( f θ ( x i ) , y i ) during stochastic optimization, we can derive the gradientexpectation as follows:
[Equation]
Proposition 3 (Augmented empirical risk minimization for experience replay) : Assume a memory set D M = { x i , y i } i =1 ,.., |D M | and an incoming task stream D T = { x i , y i } i =1 ,.., |D T | with different data distribution P ( D T )
̸
= P ( D M ) , and a compact topological group of transform G with a probability distribution of Q that acts on the input space X and is invariant under function f , i.e. f ( gx ) =
f ( x ) , g ∈ G, x ∈ X .
At each iteration t , a batch data is sampled from memory B M t ∼ D M and B M t = { x i , y i } i =1 ,... |B
M t | , a batch of data from incoming task B t ∼ D T and B t = { x i , y i } i =1 ,... |B t | , and a randomly selected group operation g t ∼ G .
To update the pa- rameter θ of the function f θ based on loss function L , consider a parameter update rule defined by θ = θ − η |B t | (cid:80)
x
i ,y i ∈B t ∇L ( f θ ( g t x i ) , y i ) − η | B M t | (cid:80) x i ,y i ∈B M t ∇L ( f θ ( g t x i ) , y i ) , then it is an unbiased stochastic gradient descent for the following loss function
[Equation]
Proof :
Given the augmented empirical gradientduring stochastic optimization, we can derive the gradient expectation as follows:
[Equation]
The second equality is based on the results of proposition 1.
Proposition 1 (Empirical risk minimization for online experience replay) : Assume an initial memory set D 0 M and an incoming task stream D T with different data distribution P ( D T )
̸
= P ( D M ) .
At each iteration t , t = 1 , ..T , a batch of data from incoming task B t ∼ D T and B t = { x i , y i } i =1 ,... |B t | and a batch data is sampled from memory B M t ∼ D t M and B M t = { x i , y i } i =1 ,...
|B
M t | .
To update the parameter θ of a function mapping f θ : f θ × X → Y based on loss function L , consider a parameter update rule defined by θ = θ − η |B t | (cid:80)
x
i ,y i ∈B t ∇L ( f θ ( x i ) , y i ) −
[Equation] [Equation]
Memory is updated at the end of each iteration using reservoir sampling Vitter [1985], Chaudhry et al.
[2019] D t +1 M ← RS ( D t M , B t ).
Then the gradient is an unbiased stochastic gradient descent for the following loss function
[Equation]
where β t = 1+2 ∗ NtcurNpast , N tcur = (cid:80) i
=
t i =1 |B i | denotes the number of samples of current task that havebeen seen so far and N past = (cid:80)
j = T j =1 |D j | denotes the number of samples of past tasks.
Note 1 : The objective function R t ( θ ) changes with respect to batch number t due to the changes inβ t .
Note 2 : 1 1+2 ∗ |DT |
Npast ≤ β t ≤ 1 and β t is decreasing with the batch number t .
At the start of a task,β t =
= β max = 1 .
At the end of a task, β t = T = β min = 1 1+2 ∗ |DT | Npast .
Note 3 : Consider a balanced continual learning dataset (e.g. Split-CIFAR100, Split-Mini-ImageNet),i.e.
|D j | =
|D T | , j = 1 , ..
T we have β min =
T − 1 T +1 =
(1 − 2 T +1 )
and lim T →∞ β min = 1 .
Note 4 : Consider general continual learning datasets.
As CL learns more tasks, N past increases,lim N past →∞ β t =
lim N past →∞ 1+
[Equation]
NtcurNpast = 1 .
Proof :
Given the empirical gradient
[Equation]
∇L
[Equation]
f θ
[Equation]
x i ) , y i ) during stochastic optimization, we can derive the gradientexpectation as follows:
[Equation]
The third equality is based on the condition that D t M is updated using reservoir sampling so that allthe data seen so far have a equal probability of being stored in the memory.
In other words, at a giventime t , the memory contains a fraction of
N past N past
+ N tcur samples coming for past tasks and a fraction of
[Table]
N tcurr N past + N t cur samples coming for the current task.
[Table]
B Dataset details
[Table]
C Implementation Details
[Table]
C.1 Continual Learning Implementation
[Table]
The hyperparameter settings are summarized in Table 4.
All models are optimized using vanilla SGD.
For all experiments, we use the learning rate of 0.1 following the same setting as in Aljundi et al.[2019], Shim et al.
[2021], and the Nearest-Class-Mean (NCM) classifier is used for evaluation, as
Mai et al. reported (2021) considerable and consistent performance gains when replacing the Softmaxclassifier with the NCM classifier.
Each mini batch during training consists of 10 new and 10 memorysamples, except for the SCR methods, which employs 100 memory samples and 10 new incomingsamples Mai et al.
By default, the repeated rehearsal parameters for all the results is N = 10and the augmented rehearsal parameters are P = 1 , Q = 14 .
This paper uses Randaugment [Cubuk et al., 2020], which is an auto augmentation method.
Itrandomly selects P number of augmentation operators from a set of 14 operators and applies them tothe images.
The augmentation operator sets include: ’Identity’, ’AutoContrast’, ’Equalize’, ’Rotate’,’Solarize’, ’Color’,’Posterize’, ’Contrast’, ’Brightness’, ’Sharpness’,’ShearX’, ’ShearY’,’TranslateX’,’TranslateY’.
[Table]
[Table]
C.2 RL-based hyperparameter tuning implementation
The memory iteration choices are from 1 to 20 and augmentation choices are (1 , 5) , (1 , 14) , (2 , 14) , (3 , 14) , (4 , 14) .
Action selection probabilities are modeled with softmax weight π w ( a i ) =
[Equation]
Bootstrapped policy gradient are used to adjust action weight.
[Equation]
where ˆ π + w ( a i ) :
= Σ a k ∈X + ai π w ( a k ) and ˆ π − w ( a i ) :=
Σ a k ∈X − ai π w ( a k )
Better/Worse action set The key of BPG idea is to incorporate prior information into the construction of better and worse action set.
To apply BPG into OCL environment, we propose to determine the better/worse action set based on the feedback of current memory batch accuracy A M , which reflects the stability level of CL agent.
We want the stability level is neither too high or too low.
The desirble stability level is defined by A ∗M .
Based on the stability-plasticity analysis (see Fig 3), we find a higher repeated replay iteration leads to higher stability.
Therefore, the better/worse action set is defined as follows:
[Equation]
Reward A challenge in the hyperparameter tuning for OCL setting is that the CL agent may facenew tasks with unseen data distribution.
Therefore, it is often infeasible to assume the existenceof an external validation data containing all the tasks in advance for hyperparameter tuning.
Toachieve online hyperparameter tuning without external validation data, we propose to define thereward based on the accuracy on the memory.
Given a target memory accuracy A ∗M , reward is defined
[Equation]
|
[Equation]
M − A ∗M |
Non-stationary To address the non-stationary nature in the CL envrionment, we reset the weight of
BPG to uniform weight at the start of each task.
The analysis of the selected action can be found in Figure 12 and 13 in section D.7.
Algorithm
M is a memory with fixed size,
B t is the incoming batch data from current task, θ are the parameters of CL network, w is the parameter of RL agent,
K is the number of memory iterations,
P, Q are the augmentation hyperparameters
A ∗M target memory accuracy 1: procedure RAR( M t , B t , θ t , w t ) 2:
K t ∼ π w itert 3:
P t , Q t ∼ π w augt 4: for k = 1 , ..., K t do 5:
B M t,k ∼ MemRetrieval ( M t ) 6:
B aug ← aug ( B M t,k ∪ B t , P t , Q t ) 7:
A M ← MemAcc ( M , θ t,k ) 8:
X + , X − ← ActionSet
( A M , A ∗M ) 9: θ t,k +1 ← SGD ( B aug , θ t,k ) 10: end for 11:
M t +1 ← MemUpdate ( M t , B t ) 12: w itert +1 , w augt +1 ← UpdateRL
( A M , X + , X − ) 13: end procedure
D Supplementary Experimental Results
D.1 Temporal effectiveness of RAR
The previous experiments have analyzed RAR based on the final accuracy it achieves at the end ofcontinual learning.
This section is focused on the performance of RAR during the continual learningprocess.
As shown in Fig 5, ER-RAR (the orange lines) performs better than ER (the blue lines)during the whole training process of continual learning.
However, it should be noted that for the first task, ER-RAR already has some advantages over ER.
The training scenario for the first task is actually an i.i.d.
setting since the data distributions fromthe memory and incoming batch are the same—as they are both sampled from the distribution oftask 1.
Since we are more interested in RAR’s effectiveness in non-i.i.d settings, we design a newexperiment model, RAR w/o T1, which does not employ RAR for the first task and only starts to useit from the second task onward.
The result in Fig 5 shows that without applying RAR for the firsttask (the green lines), the subsequent performance is a bit lower for the first few tasks but quickly
[Figure]
catches up.
More importantly, in terms of the final accuracy, there is not much difference between
ER-RAR and ER-RAR w/o T1.
This result highlights RAR’s effectiveness in a non-i.i.d.
learningsetting with distribution mismatch in experience replay.
D.2 Ablation studies for MIR-RAR, ASER-RAR and SCR-RAR
As shown in Table 1, RAR improves ER and ER variants (MIR, ASER and SCR).
The detailed resultsof the ablation studies of RAR with ER variants are presented in this section.
In particular, Fig 6 and7 shows the comparison of only using repeated rehearsal or augmented rehearsal for MIR and ASERrespectively.
Neither of them alone consistently improves the performance of the baseline.
This resultsuggests the influence of RAR for MIR/ASER is similar to ER.
The performance gain comes fromthe combination of the repeated rehearsal and augmented rehearsal.
Fig 8 shows the results for SCR.
Repeated rehearsal leads to consistent performance gain because SCR already includes augmentation.
In terms of temporal performance during the CL learning process, RAR-MIR, RAR-ASER and
RAR-SCR outperform the baselines during the whole training process, and the improvement is stillevident without employing the RAR technique for the first task.
Fig 9 shows the results of RAR-MIR,
RAR-ASER and RAR-SCR on CIFAR100 datasets with a 2k memory.
D.3 Reweighted ER
To deal with the biased ER loss, one straightforward way is to balance the weight of the memory loss and incoming loss by introducing a reweighting hyperparameter α in the gradient of Eq 2.Specifically,
[Figure]
the gradient for reweighted ER is implemented as below:
[Equation]
The performance of ER and RAR with respect to the different reweighting hyperparameter α ∈[0 .
9] is shown in Fig 10, where α = 0 .
5 denotes vanilla ER loss with the equalweighting of memory loss and incoming loss.
To keep the learning rate comparable to vanilla
ER, the learning rate of ER-rw is twice as that of ER. A key observation is that similar to vanilla ER, reweighted ER (ER-rw) also significantly benefits from repeated and augmented rehearsal, as ER-rw-rar greatly improves over ER-rw (see the red line and blue line in Fig 10) for all four datasets.
D.4 Large-scale online CL
To examine the effectiveness of RAR in large-scale continual learning problem, we apply RAR in
ImageNet-1k.
ImageNet-1k is split into 10 tasks and each task contains 100 classes.
The trainingdataset of 10 tasks contains 1,281,167 images in total.
Considering the task size, we evaluate witha memory size of M= 20k (with the task to memory size ratio λ ≈ 6 . 4 ) and a memory size of
M=100k (with the task to memory size ratio λ ≈ 1 . 28 ).
These choices are similar to Seq-CIFAR100( λ = 1 . 25 ) and CORE50 ( λ = 6 ) with a 2k memory.
We randomly crop the images to size 224x224,and use the ResNet-18 architecture for training on ImageNet-1k with a single epoch.
An incomingbatch size of 32 and a memory batch size of 32 are used.
We employ a learning rate of 0.1 with 0.001of weight decay.
For RAR method, we use 10 memory iterations with RandAugment with P=1 and
Q=14.
The results are shown in the Table 5.
We observe that repeated augmented rehearsal (RAR) is effective in this large-scale online continuallearning problem and improves the vanilla rehearsal from 2.1% to 15.1% with a 20k memory and8.7% to 34.7% with a 100k memory.
The discussion in Buzzega et al.
[2020] suggests that to deal with complex continual learning datasets,it is necessary to employ multiple epochs of training with offline CL to avoid the underfitting problempresent in online learning.
Our results show that RAR can greatly improve online rehearsal for largescale CL problems.
Although several offline CL algorithms have been evaluated on ImageNet-1k, toour knowledge, our result is the first attempt to apply online CL to this problem.
We aim to investigateother online rehearsal-based methods on ImageNet-1k in future work. [Figure] [Table]
D.5 The influence of model size
We investigate the influence of neural network model size by comparing the performance of reduced
ResNet-18 and ResNet-18.
There is not much difference for CIFAR100, Mini-ImageNet and CLRS.
For CORE50, the accuracy of ER decreases with a larger model but the accuracy of ER-RAR increases(see Table 6).
A larger model usually introduces higher overfitting risk, which may explain the greaterperformance gain of RAR in ResNet-18.
[Table]
[Figure]
[Figure]
D.6 Other types of augmentation
[Figure] Besides using the auto augmentation method of RandAugment, we investigate the effectiveness ofusing a pre-defined fixed augmentation strategy for RAR.
Fig 11 shows the results of employingaugmentation with four operators (crop, flip, color jitter and gray).
The main findings on the interplayof repetitive replay and augmentation are similar to the case with RandAugment.
Nevertheless,compared to auto augmentation, a fixed augmentation may be particularly suitable for one dataset,e.g., CIFAR100, and not suitable for others, like CLRS.
D.7 Action Selection in Reinforcement Learning
[Figure]
The selected hyperparameters for four datasets are shown in Figure 12.
Interestingly, the RL-basedmethod assigns a stronger augmentation and lower iteration for a dataset with a higher λ attribute.
As discussed in the ERM analysis, a dataset with a higher λ suggests a higher risk of overfitting.
RL-based method successfully takes this into account to tune the hyperparameters.
In contrast, the
OCL-HT method selects the lowest augmentation and highest iteration for three datasets.
Figure 13 presents the selected hyperparameters at different CL training stages.
Generally, as thecontinual learning continues, RL-based method selects stronger augmentation strength and loweriterations, to balance off the increasing risk of memory overfitting.
D.8 Running time
D.8.1 Running time of RAR
The running time of ER and ER-RAR is shown in Table 7.
Experients are conducted using an Nvidia
GeForce RTX 2080 TI Graphics Card on Mini-ImageNet.
The running time of RAR grows linearlywith respect to the number of replay iterations.
Hence, compared to vanilla experience replay, RARrequires more running time due to the multiple iteration design.
However, RAR is still much morecomputationally efficient compared with offline CL with multiple epochs.
An interesting futureresearch direction is to study how to dynamically adjust the iteration number and augmentationstrength of RAR to balance the trade-off of accuracy and running time.
[Table]
D.8.2 Running time of RL
RL-based hyperparameter optimization is an online method that does not require repeated runningover different hyperparameter choices.
Therefore, RL-based HPO is much more computationallyefficient than offline hyperparameter selection methods.
More specifically, the hyperparameter searchspace in our problem is 100 with 5 augmentation strength levels and 20 memory iteration numbers.
Grid search would need to run the OCL algorithm 100 times while RL only needs to run it once.
Nevertheless, the training of RL agents indeed introduces extra computation.
In practice [Table] we observethe running time of RL-RAR is about two times slower than that of RAR, as shown in the Table
[Table] [Table]
2 ± 0.