Retrospective Adversarial Replay for Continual Learning
Abstract
Continual learning is an emerging research challenge in machine learning that addresses the problem where models quickly fit the most recently trained-on data but suffer from catastrophic forgetting of previous data due to distribution shifts — it does this by maintaining a small historical replay buffer in replay-based methods.
To avoid these problems, this paper proposes a method, “Retrospective Adversarial
Replay (RAR)”, that synthesizes adversarial samples near the forgetting boundary.
RAR perturbs a buffered sample towards its nearest neighbor drawn from the current task in a latent representation space.
By replaying such samples, we are able to refine the boundary between previous and current tasks, hence combating forgetting and reducing bias towards the current task.
To mitigate the severity of a small replay buffer, we develop a novel MixUp-based strategy to increase replay variation by replaying mixed augmentations.
Combined with RAR, this achieves a holistic framework that helps to alleviate catastrophic forgetting.
We show that this excels on broadly-used benchmarks and outperforms other continual learning baselines especially when only a small buffer is available.
We conduct a thorough ablation study over each key component as well as a hyperparameter sensitivity analysis to demonstrate the effectiveness and robustness of RAR.
Introduction
Traditional supervised machine learning methods often rely on the assumption that the data is drawn i.i.d. from a stationary probability distribution.
This assumption does not hold in many practical scenarios where the learner must continuously learn online and adapt to new tasks without revisiting previous tasks (and those tasks’ data).
This has motivated research in Continual Learning (also referred to as Lifelong Learning and Incremental Learning), where a machine learning model learns from a stream of data coming from a succession of different tasks [29, 45, 66].
The primary challenge in continual learning (CL) is to alleviate the “catastrophic forgetting” of the previously learnt tasks after learning new tasks [18, 39, 49].
This is mainly caused by a shift in the distribution of inputs and labels over time.
For example, as a model is updated using new-task gradients, the hidden-layer representations, encoding information about previous tasks, become biased towards these new tasks.
This leads to a model confusing the former with current tasks, thus producing incorrect predictions on older tasks.
A widely studied strategy to alleviate forgetting is experience replay (ER) [10, 49, 50], which repeatedly trains the model on buffered replay data from previous tasks while learning the current task.
However, the buffer in practice is often quite small [63], e.g., in autonomous driving, where the incremental data is large-scale and essentially never-ending.
A model can thus over-fit to such small buffered data.
Although forgetting is mitigated, the goal of retaining broad and accurate knowledge about previous tasks is not achieved, and the problem remains severe.
An ideal strategy to address the above challenge is to focus on the replay of “marginal samples” easier to forget and confuse with the current task’s data, e.g., those near the boundary between the previous tasks’ data and the current task’s data.
However, selecting such samples [2, 55] from a small buffer only brings limited improvement because the buffer (often formed using reservoir sampling [62] and thus is uniform) does not well cover many such marginal samples.
Hence, we surmise that the core challenge is how to generate a small and sufficient set of samples, ones that are easier to forget and hence be confused with current task’s data — given the limited buffer size, it is also important for these samples to be diverse and thus efficient.
In this paper, we propose a novel targeted adversarial synthesis based replay method, Retrospective
Adversarial Replay (RAR), that can generate more informative replay instances with richer variations optimized to capture the forgetting frontier of continual learning, from a limited buffer.
Specifically, given an incoming batch of the current task’s data, RAR first identifies the most-likely-to-be-forgotten buffer samples (Stage
I in Fig. 1) and pairs them with their nearest neighbors within the current task’s data (Stage II in Fig. 1).
For each pair, we then apply a bounded and targeted adversarial perturbation to the buffer sample, which moves the buffer sample’s hidden-layer representation towards that of the paired current task’s sample
(Stage III in Fig. 1).
This three-step procedure, as illustrated in Fig. 1, generates fine-grained adversarial augmentations of the buffered data that capture “forgetting” at the local boundary to the current task’s data.
We then effectively reduce the risk of forgetting by replaying these perturbed samples.
To improve the diversity and variation of the perturbed samples (thus increasing the learnable information from a limited-size buffer), we investigate the role of MixUp [58, 67, 68], a data augmentation technique applied together with RAR — we find that it brings substantial improvements when there are strict buffer size constraints.
For Stage I replay data selections, we can utilize any existing methods [2, 3, 10, 55, 65] resulting in an effective and data-efficient replay solution in CL, hence making RAR generic and complementary to existing memory-retrieval & update methods.
Extensive experiments on RAR and comparisons to SoTA baselines from different CL categories demonstrate the advantages of RAR in mitigating catastrophic forgetting and improving the overall accuracy.
Moreover, we conduct an ablation study showing that the key components such as replay samples selection strategy, sample pairing & adversarial perturbation, MixUp, etc, each bring appreciable improvements.
Additional sensitivity analysis of the key hyperparameters implies that RAR is stable and consistently achieves these improvements.
Connection to human false memories & their correction :
In human psychology, "false memories" [36, 42] refer to a wide variety of human memory errors ranging from the minor misremembering of small details (e.g., distorted recollection) all the way to unmitigated fabrication.
The passage of time after experiencing an event increases the likelihood of these phenomena.
They are referred to as [42] a “natural by-product of a distributed memory system (DMS)”.
Rather than a simple search & recollect, DMS [12] helps us in recombining and reconstructing memories via the integration of episodes from the past and present, but it is not always error-free.
It is also true that when high-confidence false memories are corrected, people attend more to the provided feedback [17].
Despite a tendency for belief perseverance [4], the information that breaks through this perseverance is greatly attended to.
The motivation behind RAR is in line with this theory.
In RAR, we first combine/pair the replay buffer samples from previously seen tasks with the current data.
Then using the pairing, we find existing high-confidence false memories in the form of targeted adversarial perturbations that move the previous data closer to the current data in the model’s latent space.
Finally, we train the model to produce the correct target for the perturbed input during the replay stage.
On the contrary, existing ER-based CL methods lack this adversarial interpolation between past and present data but mainly rely on simple retrieve and replay of past data.
Related Work
Existing approaches in CL can be broadly characterized into two groups based on the model’s architecture.
Methods dealing with dynamic architecture maintain separate parameters for each task [34, 53, 64].
These methods require the knowledge of task descriptors, are expensive to train and often do not scale well when dealing with a long sequence of online tasks.
In this work, we focus on the task-free setting of CL [61].
Fixed architecture approaches use regularization techniques to selectively regularize parameters (either weights or outputs) important for old tasks [1, 8, 27, 32, 35, 52, 66], and require task information in general.
This category also includes memory-based approaches for constrained parameter updates on new samples to minimize interference with previous tasks [9, 37] and for replaying them along with the new task’s data [2, 3, 5, 6, 10, 13, 38, 41, 43, 44, 47, 48, 59, 55, 65].
Other ER methods [2, 51, 56, 60] train generative models on previous tasks which can be challenging owing to the online (single-pass) learning setting.
CL using Adversarial Information : ACL [16], belonging to the dynamic architecture category, uses adversarial learning to train a shared feature encoder and a discriminator using a GAN’s minimax objective [19].
ASER [55] proposes using the kNN-specific Shapley value [24, 54] to select replay samples that are representative of the buffer and adversarially close to the decision boundaries of incoming classes.
GMED [25] uses gradient updates to edit buffered samples individually based on their loss increase after a look-ahead update performed only on the incoming batch.
Compared to these methods, RAR is a fixed architecture task-free CL framework that leverages the local pairwise relationships between the replay and current task’s samples to locally perturb each replay sample in a targeted adversarial manner.
Our targeted perturbation generates samples close to the forgetting frontier that are hence likely confusing for the model but that are also representative of previous data — this results in a good set to further train on.
To elucidate the differences in detail, we also provide a more extensive survey of the relevant prior literature in Appendix B.
Retrospective Adversarial Replay
Formally, the continual learning problem consists of a stream of data distributions ( D 1 , D 2 , . . .
, D T ) that corresponds to T tasks (in this work, the boundaries between tasks are not known).
We get data points uniformly sampled from the data distribution of the current task.
Our target is to train a machine learning model f ( · ; θ ) parameterized by θ that minimizes the losses as we learn each task in the given sequential order without increasing the loss on previously learned samples.
In other words, the objective is
[Equation]
Here, ℓ denotes the loss function and i is an index over the set of tasks.
To retain the knowledge of previous tasks and mitigate catastrophic forgetting, similar to [2, 55, 65], we utilize a limited memory buffer to store samples representing previously seen data.
Let M denote the memory/buffer, which is essentially a set of data points and |M| ≤ m .
Similar to ER (Alg. 2), as we train on task i , we get an incoming batch ( X i , Y i ) ∼ D i , and we combine it with a subset of memory samples ( X ′ M , Y ′ M )
⊂ M to compute the overall loss.
We can then update the memory M by replacing some samples with the ones encountered in the current training step while keeping the size of the buffer unchanged.
In general, ER-based methods typically deal with two problems: (1) Memory Update for populating the buffer and (2) Memory Retrieval for selecting samples from the buffer for replay.
In this work, we focus on synthesizing adversarial memory samples which provide the most conflicting information on the look-ahead probing step.
In other words, the model trained only on the incoming task (one-step look-ahead) misclassifies the synthesized samples (anchored around previously seen classes stored in the memory) as the incoming classes.
Thus, by learning on such adversarially synthesized memory samples in retrospect, we prevent any conflicts in the probing step.
For problem (1), we use reservoir sampling to update the memory (Appendix C), which can give us uniform samples from the streaming data points and is widely adopted in many continual learning methods [2, 10, 55].
For problem (2), one can utilize any existing memory retrieval methods [2, 3, 10, 55, 65], thus making
RAR complimentary and easily integratable with many existing CL methods.
3.1 Retrospective Adversarial Perturbation
Existing methods addressing the memory-retrieval problem optimize the model parameters using ℓ ER in Eq.
2 where ( X ′ M , Y ′ M ) are selected from the buffer M based on their proposed selection strategies.
For the ease of notation, we use y ( x ) to denote the ground-truth label of sample x .
[Equation]
ER [10] uses naive random sampling to select X ′ M , GSS [3] encourages gradient diversity while sampling X ′ M , ASER [55] uses kNN Shapley value to select samples which are representative of
M and close to the samples in X i in the latent space, and MIR [2] selects replay samples which are maximally interfered/forgotten after the look-ahead probing step.
In the look-ahead probing step, the model is updated only on the incoming task samples such that θ ′ =
θ − η ∇ θ ℓ ( X i , y ( X i ); θ ) .
Although the methods mentioned above cover crucial aspects (diversity, representativeness, similarity to incoming batch) for selecting X ′ M , they do not capture the local pairwise interactions between the memory and current-task samples.
Motivated by this, Retrospective Adversarial Replay Loss ( ℓ RAR ) finds a cardinality-constrained pairing ( E ) between memory and incoming task samples such that the loss on memory samples that are perturbed towards the paired current-task samples is maximized:
[Equation]
Here, X M denotes the entire input samples present in M , and g θ ′ l ( x M , x i ) described mathematically in Eq.
4 can be seen as a targeted perturbation of x M that is visually similar to x M but close to the latent embedding of target x i ∈
X i in the l -th layer latent space modeled by θ ′ .
Since the min-objective in Eq.
4 aims to move the perturbed replay sample close to the current-task sample, y ( x M ) ̸
= y ( x i ) for a particular pairing to be considered and to filter out any trivial pairings while optimizing Eq.
[Equation]
where d
[Equation]
Since x M and x i have different labels, such perturbed samples that we get from Eq.
4 are confusing to the continual learner in terms of distinguishability from the current-task sample x
i , while still representing previous tasks based on their visual similarity to x M ∈ M .
Eq.
3 essentially helps in the synthesis of targeted adversarial samples that are close to incoming samples
X i in the latent space and hence the learner will miss making the correct prediction y
( X M ) despite the fact that the adversarial samples are anchored around the original memory samples X M .
Training on such samples can help the model to learn about the boundaries between previous and current tasks and also retain the knowledge of previous tasks [7, 11, 23].
Note that the perturbations are performed based on the look-ahead parameters θ ′ .
In other words, assuming we only learn from the current-task samples
X i and update the parameters to θ ′ , the generated adversarial samples give us many training samples that show conflicting behaviors on θ ′ and get easily forgotten by the model.
In retrospect, we can potentially prevent such forgetting when we are still at θ by using those targeted adversarial samples as training signals, i.e., we minimize the RAR loss (Eq. 3) to update the model θ .
Optimizing Eq.
3 over all feasible pairs of X M and X
i is computationally challenging as we effectively need to compute the adversarial perturbations on all such pairs to select the most confusing subset incurring the maximum RAR loss.
Moreover, each adversarial sample synthesis (Eq. 4) requires multiple gradient steps as we need to run adversarial attack algorithms such as the iterative-FGSM [15, 20] covered in detail in Appendix H.
Thus, to eliminate this bottleneck, we propose two pruning strategies covered in sections 3.2 and 3.3 to reduce the candidate anchor-target pairs in X M ×
X i for adversarial perturbations and approximately optimize Eq.
3.2 Selecting Replay Data from Buffer
To reduce the computational overhead, we only consider a subset of the memory buffer X ′ M ⊂ M instead of considering all memory input samples X M .
Memory-retrieval methods such as ER [10],
GSS [3], ASER [55], and MIR [2] can be used to select a smaller replay set X ′ M .
This again reiterates that RAR is generic and complementary to existing memory-retrieval methods.
3.3 Pairing between Buffer and Current-task Data
To further bring down the computational costs of adversarial perturbations, we perform another pruning over the possible pairs between X ′ M
×
X i based on samples’ hidden representations.
I.e., we first compute the distance d θ ′ l ( x ′ M , x i ) , which is essentially the target value for the adversarial perturbation (Eq. 4) for every pair ( x ′ M , x i )
∈ X ′
M
× X i with different labels y ( x ′ M ) ̸
= y ( x i ) and then screen out the pairs with larger distance values, maintaining only the smaller-distance ones, in accordance with the min-objective of Eq. 4.
To utilize the information from the selected memory samples, we pair each x
′ M ∈
X
′ M with its nearest neighbor with a different class/label in the incoming batch
X i of current task.
Nearest-neighbor-based matched pairs already represent the most confusing samples as the samples in a pair belong to different classes.
Thus, it is easier for the optimization procedure approximating Eq.
4 to further reduce the distances between pairs coming from different classes (shown in Fig. 2b), aiding in the synthesis of even more confusing adversarial samples.
Computational overhead for optimizing Eq.
3 : First, we select memory samples for replay X ′ M
(Sec. 3.2) which has the same costs as the backbone memory-retrieval method.
Next, we compute the distances between latent features of X ′ M and
incoming batch X i with computational complexity of
O ( | X ′ M | ∗ | X i | ) .
This is minimal given the online nature of CL where the training batch size is often set to 10 .
After performing the nearest neighbor matching, we optimize Eq.
4 via iterative-FGSM [15] which introduces some additional computational costs, but we keep it marginal by using only two steps of iterative-FGSM.
We provide further details in Appendix E and H.
Combining our RAR loss with the loss on selected memory samples and incoming batch samples, we have our overall objective (similar to Eq. 2) defined as follows, where β is a hyperparameter to control the trade-offs between the losses on original and adversarially synthesized memory samples.
ℓ all ( X ′ M , X i , θ )
[Equation]
=
[Equation]
1: Input: Tasks ( D 1 , D 2 , . . .
D T ), Model θ , Learning Rate η , Memory M , Memory Size m , Replay
Budget k , Trade-off Parameter β , Perturbation Strength ϵ & number of gradient steps n 2: Output: Model θ 3: for task i ∈
[ T ] do 4: for ( X i , Y i ) ∼ D i do 5: θ ′ ← SGD( ℓ ( X i , Y i ; θ ) , θ, η ) { // Look-ahead update of model parameters} 6:
Select ( X ′ M , Y ′ M ) from M s.t.
| X ′ M |
= k { // using existing memory-retrieval methods} 7:
Let S := ∅ 8: for x ′ M ∈ X
′ M do 9: x ′ i ∈ argmin x i ∈
X i ,y ( x ′ M ) ̸
= y ( x i )
d θ ′ l ( x ′ M , x i ) 10:
S ← S ∪ { ( x ′ M , x ′ i ) } 11: end for 12: ℓ RAR ( S, θ, θ ′ )
← (cid:80) ( x ′ M ,x i ) ∈
S ℓ ( g θ ′ l ( x ′ M , x i ) , y ( x ′ M ); θ ) { // perform the targeted adversarial perturbations (Appendix H) and define the corresponding RAR loss} 13: ℓ all ← ℓ ( X i , Y i ; θ ) + β ℓ RAR ( S, θ, θ ′ )
+ (1 − β ) ℓ ( X ′ M , Y ′ M ; θ ) 14: θ ← SGD( ℓ all , θ, η ) 15:
M ← ReservoirUpdate( X i , Y
i , M , m ) 16: end for 17: end for
Description of RAR framework :
In Alg. 1, we describe our retrospective adversarial replay algorithm for continual learning.
The algorithm essentially consists of three steps: (1) in Lines 5-6, we first perform the look-ahead parameter probing using the incoming samples of the current task to get θ ′ , and select the subset X ′ M using existing memory retrieval methods, such as ER, MIR,
ASER, etc. (2) in Lines 7-11, we prune down the pairings between X ′ M and X i based on distances between hidden representations computed using Eq. 5, and (3) in Line 12, we define the RAR loss by performing adversarial perturbations on the selected pairs S .
Note that since we use the selected pairings S of size k as an approximation of the true maximum-valued pairs, we can just sum over entries in S instead of picking the maximum subset as done in Eq.
In Line 15, we update the memory buffer M based on incoming samples ( X i , Y i ) using reservoir sampling [62] and keep the memory limit m .
3.4 MixUp based Retrospective Adversarial Augmentation
We also propose another method mix-RAR utilizing the mixup technique [67] to bring in more diversity to the pool of replay examples selected from the buffer.
After creating virtual examples (˜ x M , ˜ y M ) using x i , x j ∈ X ′ M and λ ∈ [0 , 1] such that ˜ x M = λx i + (1 − λ ) x j and ˜ y M = λy i + (1 − λ )
y j , we follow the original RAR algorithm to achieve pairing and subsequent targeted adversarial perturbation as shown in Alg 5 in Appendix A.
The proposed MixUp strategy is different from [21, 40] as we apply mixup among the replay samples and then generate RAR perturbed samples anchored around them.
This is consistent with RAR’s objective.
Given a current-task’s sample from class (c), training on an RAR perturbation applied to a mixup between two buffered samples from different past classes (a,b) can potentially help in capturing the forgetting frontier between three pairs of classes (a,c), (b,c) and (a,b), thus minimizing forgetting on both past classes.
Experiments
Datasets : We evaluate RAR on four supervised image classification benchmarks for task-free CL.
Split-MNIST [31] has five disjoint tasks, each having two classes.
(2) Split-CIFAR10 [28] consists 5 disjoint tasks with each task having two classes and 10k training examples.
(3) Split-CIFAR deals with 20 disjoint tasks with each task having 5 classes.
(4) Split-miniImageNet [14] consistsdisjoint tasks with each task having 5 classes and 2.5k training examples.
We cover further statistics in Appendix D. All these datasets deal with the class-incremental scenario [61].
Our implementation code is available at https://github.com/lillykumari8/RAR-CL.
[Table]
Evaluation Metrics :
Following [8, 37], we use two standard metrics used in CL to evaluate RAR’s performance.
Final average accuracy ( A T ) assesses the overall performance of the model whereas
Final average forgetting ( F T ) measures how much the model has forgotten each task once the online learning has completed.
Lower forgetting is better.
Below, a i,j denotes the accuracy evaluated on the test set of task j after the model has experienced all tasks up to i .
[Equation]
Settings and Hyperparameters : We follow the same setup as [2] for deciding model architectures for all four datasets.
For Split-MNIST, we use an MLP classifier with two hidden layers, each with
[Table]
400 units with ReLU activation, followed by a linear classifier layer with 10 units.
For Split CIFAR10, Split CIFAR-100, and Split mini-ImageNet, we use a reduced ResNet-18 classifier [22].
The replay budget k is the same as the mini-batch size (fixed to 10) irrespective of the buffer size m .
For hyperparameters tuning on each dataset, we hold-out 5% of the training samples for each task and use it as a validation set.
We provide additional details about the implementation settings in Appendix F.
4.1 Main Results
[Figure]
its nearest neighbor in the current tasks’ batch evolves across different stages of RAR
We report the average accuracy and forgetting in Table 1 and 2 respectively for experiments on all four datasets under various buffer size constraints.
We use three different memory-retrieval methods, namely ER [10], MIR [2], and ASER [55] and compare RAR and mix-RAR with their respective baselines which do not use the RAR loss (Eq. 3).
For fair comparison, we compare mixRAR with baseline retrieval methods using mixup between replay samples.
We also compare against standard baseline methods: (a) Fine-tuning , which trains the model continuously on the stream of tasks without employing any strategy to avoid forgetting, (b)
Averaged Gradient
Episodic Memory (AGEM)
[9] (c) iid-online , which trains a model on the iid-data sampled from entire dataset D = ∪ T i =
1 D i for a single epoch, (d) iid-offline , which trains a model similar to the iid-online setting but for multiple epochs (three epochs considered in the reported results).
We compare against another memory sample perturbation method called GMED [25] in Table 3 to avoid repetition.
We provide the pseudo-code for ER (Alg. 2), ER-RAR (Alg. 3), MIR-RAR (Alg. 4), and mix-RAR (Alg. 5) in Appendix A to describe how our proposed framework can be unified with existing CL methods.
Also, we leave out the results against GSS [3] as both MIR [2] and ASER [55] outperform it in terms of accuracy and forgetting.
As shown in Table 1, both RAR and mix-RAR using different memory-retrieval methods consistently outperform their respective non-RAR baselines (without and with mixup) across all four datasets.
From Table 2, we observe that the improvements obtained by integrating RAR with respective replay selection methods {ER, MIR & ASER} (both without and with mixup) are significant.
In case of Split-CIFAR10, MIR-mix tends to perform slightly better than
MIR-mix-RAR in terms of forgetting, but based on accuracy improvement that MIR-mix-RAR brings, the relative difference is negligible since forgetting is dependent on the maximum accuracy that a continual learner achieves across different tasks.
In the case of Split-miniImageNet, ER-mix-RAR method achieves the best average accuracy of 28 .
8% , outperforming other MIR & ASER based methods, and has similar observation on the
Split-CIFAR100 dataset.
We hypothesize that for large buffer sizes such as 10 k ( 100 examples per class), the ER-based selection aids in sampling a diverse set of classes from the memory buffer compared to MIR & ASER, thus improving the overall accuracy.
However, since MIR & ASER consider samples suffering interference from the new task’s data, MIR-mix-RAR & ASER-mix-RAR help minimize the forgetting on the old tasks.
The improved performance of ER-RAR, MIR-RAR, and ASER-RAR compared to ER, RAR, and
ASER respectively demonstrates that RAR is able to alleviate catastrophic forgetting on old tasks via generating diverse and confusing samples close to the forgetting frontier.
To ensure fair comparison in terms of computational costs, in Table 5 in Appendix
I
[Table]
we also compare RAR to GMED [25],
ER-3-iters
[Table]
and MIR-3-iters.
3-iters means that the model is trained for multiple iterations on the incoming batch and buffer samples as a new batch comes in.
[Table]
[Figure]
Case study: Visualizing the Three Stages of RAR For the Split-MNIST dataset, we visually analyze the three stages of the RAR framework for a buffer size of 500 using MIR as the memory-retrieval method.
Fig.
2a shows the samples selected for replay using the MIR score (Eq.
8) and their paired samples from the current task’s batch based on distances computed using Eq.
Fig.
2b shows how the distance between a replaying sample and current task’s data changes over the three stages: (1) for each sample selected from the buffer, the blue bar represents its average distance to all samples in the current-task batch; (2) after pairing each buffered replay sample with its nearest neighbor as described in Section 3.3, its distance to the paired sample (orange bar) becomes smaller than the blue bar; (3) after applying adversarial perturbations to each buffered sample as described in Section 3.1, the distance is further reduced.
The decrease of distance indicates that the pairing and adversarial perturbation in
RAR are effective in generating samples closer to the forgetting frontier than the original buffer data selected for replay, meaning the RAR perturbations are similar to X M but very close to the paired samples in X i (current task’s batch) in the l-th layer latent representation space of model θ .
In Fig.
3, we show four examples of RAR perturbed samples generated via Eq.
4 from four buffer samples and their paired samples from the current task.
For most of the buffer samples, a single gradient step of θ replaying them and their RAR perturbed counterparts suffices to correct their wrong predictions (e.g., the current task’s classes), hence effectively alleviating the model’s forgetting on previous tasks.
4.2 Advantage of MixUp in Small-Buffer Cases
On each dataset, as reported in Table 2, all mix-RAR methods perform better compared to RAR methods in terms of reducing forgetting on previous tasks with slight improvement in average accuracy.
The proposed use of MixUp among the replay samples before RAR is able to increase the variation of RAR perturbed samples, which is essential to CL with a small buffer.
To verify its effectiveness, we conduct experiments that store only 20 examples per class.
Figure 4 shows that ER-mix-RAR & MIR-mixRAR consistently outperform other ER & MIR methods respectively.
We can achieve even more significant improvements in forgetting, particularly for more challenging tasks such as Split-CIFAR100.
4.3 Ablation Study to Verify the Effectiveness of RAR
In Table 3, we compare RAR methods to two baseline methods: (a) Random which add a perturbation of the same tolerance as the RAR method to the replay samples before the experience replay stage, (b) GMED [25] which performs smart gradient updates in the input space by treating each replay sample individually (no local pairing considered between buffer and current task’s data).
As can be seen in Table 3, ER-RAR and MIR-RAR consistently outperform other methods confirming RAR’s superior ability to tackle catastrophic forgetting via generating targeted perturbations influenced by the local interference between different task’s data.
4.4 Sensitivity Analysis of Hyperparameters
In Fig.
5, we plot the sensitivity of RAR’s performance with respect to the number of gradient steps n used to generate the retrospective adversarial samples via optimizing Eq.
4 in the input space.
For Split-CIFAR10 and Split-miniImageNet (where we report results using two gradient steps), we observe that for a larger number of steps {5, 10}, the performance stays comparable.
This suggests that nearest-neighbor-based pairing using distance computations (covered in Sec 3.3) helps achieve anchor-target pairs which are already optimized and hence do not require numerous gradient updates to get classified as the paired target class from the current task.
[Figure] [Figure]
Next, we study the sensitivity of RAR corresponding to β which denotes the trade-off coefficient associated with ℓ RAR in the final loss objective Eq. 6 — we use only one coefficient to reduce hyperparameter search costs.
Fig.
6 shows that RAR (using MIR) outperforms vanilla MIR in terms of both accuracy (solid blue line is always above the dashed blue line) and forgetting (solid orange line always stays below the dashed orange line) for various non-zero values of β across the studied datasets.
Conclusion
In this paper, we study how to perturb limited buffer replay data to mitigate catastrophic forgetting in continual learning.
We propose “Retrospective Adversarial Replay (RAR)”, which focuses on previous tasks’ data near the forgetting boundary to the current task in the model’s representation space.
RAR first allocates the most confusing pairs of a buffered sample and a current task’s sample and then applies a bounded targeted adversarial perturbation to the former, which further moves it towards the latter in the model’s representation space and creates a perturbed sample close to the boundary.
Moreover, we study the role of MixUp in increasing the variation of replay augmentations, which significantly improves CL in the small buffer regime.
When integrated with existing replay methods, RAR consistently outperforms previous CL methods across benchmark datasets under different settings
Acknowledgements
This work was supported in part by the CONIX Research Center, one of six centers in JUMP, a
Semiconductor Research Corporation (SRC) program sponsored by DARPA.
We would also like to thank the Melodi lab students: Arnav Das and Gantavya Bhatt for their useful discussions.
A Details of RAR based algorithms
Standard ER
In Algorithm 2, we present the standard experience replay method [10, 49].
Algorithm
1: Input: Tasks
D
, Learning Rate
, Memory
, Replay
Budget k 2: Output: Model θ 3: for task i ∈
[ T ] do 4: for ( X i , Y i ) ∼ D i do 5: ( X ′ M , Y ′ M ) ∼ M { // randomly select k samples from memory} 6: ℓ ER ← ℓ ( X i , Y i ; θ ) + ℓ ( X ′ M , Y ′ M ; θ ) 7: θ ← SGD( ℓ ER , θ, η ) 8:
M ← ReservoirUpdate( X i , Y
i , M , m ) 9: end for 10: end for
ER-RAR
In the ER-RAR method, we simply modify the replay sample selection step (line 6 ) of
Alg. 1.
As shown in line 6 of Algorithm 3, we rely on random sampling to get a subset of the buffer for replay.
1: Input: Tasks ( D 1 , D 2 , . . .
D T ), Model θ , Learning Rate
, Memory M , Memory Size m , Replay
Budget k , Trade-off Parameter β , Perturbation Strength ϵ & number of gradient steps n 2: Output: Model θ 3: for task i ∈
[ T ] do 4: for ( X i , Y i ) ∼ D i do 5: θ ′ ← SGD( ℓ ( X i , Y i ; θ ) , θ, η ) { // Look-ahead update of model parameters} 6: ( X ′ M , Y ′ M ) ∼ M { // randomly select k samples from memory} 7:
Let S := ∅ 8: for x ′ M ∈ X
′ M do 9: x ′ i ∈ argmin x i ∈
X i ,y ( x ′ M ) ̸
= y ( x i )
d θ ′ l ( x ′ M , x i ) .
S ← S ∪ { ( x ′ M , x ′ i ) } .
11: end for 12: ℓ RAR ( S, θ, θ ′ )
← (cid:80) ( x ′ M ,x i ) ∈
S ℓ ( g θ ′ l ( x ′ M , x i ) , y ( x ′ M ); θ ) { // perform the targeted adversarial perturbations (Appendix H) and define the corresponding loss} 13: ℓ all ← ℓ ( X i , Y i ; θ ) + β ℓ RAR ( S, θ, θ ′ )
+ (1 − β ) ℓ ( X ′ M , Y ′ M ; θ ) 14: θ ← SGD( ℓ all , θ, η ) 15:
M ← ReservoirUpdate( X i , Y
i , M , m ) 16: end for 17: end for
Discussion of MIR score and its intuition (find the buffered data suffering from the most amount of forgetting)
The idea of probing to the next-step parameters θ ′ that we perform in RAR is similar to the Maximally Interfered Retrieval (MIR) method for memory/buffer retrieval [2], where we compute forgetting scores on the memory samples to identify a subset of memory samples that are negatively impacted by the current-task samples.
The forgetting score of MIR for every sample in the memory buffer x M is defined as follows:
[Equation]
Then a subset of memory samples with the largest MIR scores i.e. having the greatest loss increase from θ to θ ′ are selected.
We will denote such a subset as X ′ M , i.e.
[Equation]
1: Input: Tasks ( D 1 , D 2 , . . .
D T ), Model θ , Learning Rate η , Memory M , Memory Size m , Replay
Budget k , Subset Size c , Trade-off Parameter β , Perturbation Strength ϵ & number of gradient steps n 2: Output: Model θ 3: for task i ∈
[ T ] do 4: for ( X i , Y i ) ∼ D i do 5: θ ′ ← SGD( ℓ ( X i , Y i ; θ ) , θ, η ) { // Look-ahead update of model parameters} 6: ( X M , Y M ) ∼ M { // randomly select c samples from memory} 7:
Compute MIR ( x M , θ, θ ′ ) for x M ∈ X M and select the subset X ′ M based on Eq.
9 s.t.
| X ′ M | = k 8:
Let S := ∅ .
9: for x ′ M ∈ X ′ M do 10: x ′
i ∈ argmin x i ∈
X i ,y ( x ′ M ) ̸
= y ( x i )
d θ ′ l ( x ′ M , x i ) .
S ← S ∪ { ( x ′ M , x ′ i ) } .
12: end for 13: ℓ RAR ( S, θ, θ ′ )
← (cid:80) ( x ′ M ,x i ) ∈
S ℓ ( g θ ′ l ( x ′ M , x i ) , y ( x ′ M ); θ ) { // perform the targeted adversarial perturbations (Appendix H) and define the corresponding loss} 14: ℓ all ← ℓ ( X i , Y i ; θ ) + β ℓ RAR ( S, θ, θ ′ )
+ (1 − β ) ℓ ( X ′ M , Y ′ M ; θ ) 15: θ ← SGD( ℓ all , θ, η ) 16:
M ← ReservoirUpdate( X i , Y
i , M , m ) 17: end for 18: end for
Intuitively, X ′ M contains samples that are most confusing with the incoming ones, and thus using such samples as the anchor samples would result in more successful adversarial perturbations.
As we need to compute θ ′ for the retrospective adversarial samples, the MIR score does not incur extra computations.
Moreover, we can also utilize the subset X ′ M as additional training signals similar to the MIR approach, which allows us to trade-off between the adversarially perturbed data points versus real data points that are uniformly sampled from previous tasks.
Algorithm
1: Input: Tasks ( D 1 , D 2 , . . .
D T ), Model θ , Learning Rate η , Memory M , Memory Size m , Replay
Budget k , Subset Size c , Trade-off Parameter β , Perturbation Strength ϵ & number of gradient steps n 2: Output: Model θ 3: for task i ∈
[ T ] do 4: for ( X i , Y i ) ∼ D i do 5: θ ′ ← SGD( ℓ ( X i , Y i ; θ ) , θ, η ) { // Look-ahead update of model parameters} 6:
Select ( X ′ M , Y ′ M ) from M s.t.
| X ′ M |
= k { // using existing memory-retrieval methods} 7: ˜ X ′ M , ˜ Y ′ M = mixup( X ′ M , Y ′ M , α ) { // perform mixup on replay samples} 8:
Let S := ∅ 9: for ˜ x M ∈ ˜ X
′ M do 10: x ′
i ∈ argmin x i ∈
X i ,y (˜ x M ) ̸
= y ( x i )
d θ ′ l (˜ x M , x i ) .
S ← S ∪ { (˜ x M , x ′ i ) } .
12: end for 13: ℓ RAR ( S, θ, θ ′ )
← (cid:80) (˜ x M ,x i ) ∈
S ℓ ( g θ ′ l (˜ x M , x i ) , y (˜ x M ); θ ) { // perform the adversarial perturbations (Appendix H) and define the corresponding loss} 14: ℓ all ← ℓ ( X i , Y i ; θ ) + β ℓ RAR ( S, θ, θ ′ )
+ (1 − β ) ℓ ( X ′ M , Y ′ M ; θ ) 15: θ ← SGD( ℓ all , θ, η ) 16:
M ← ReservoirUpdate( X i , Y
i , M , m ) 17: end for 18: end for
MIR-RAR
In Algorithm 4, we change line 6 of Algorithm 1 to use the MIR score to select the replay subset X ′ M .
mix-RAR
In the mix-RAR method detailed in Algorithm 5, after selecting the replay samples
X ′ M (using ER, MIR, ASER, etc.) from the memory M , we simply apply MixUp [67] to perform perturbations/editing on the replay examples.
Mixup generates virtual training samples ˜ x M , ˜ y M by taking convex combinations of pairs of inputs and their labels.
[Equation]
where x i , x j ∈ X
′ M
and
y i , y j ∈ Y ′ M and λ ∈ [0 , 1] is drawn from a Beta distribution λ ∼ Beta( α, α ) , where α is a hyperparameter.
We, then pair up these virtual samples with the current’s task samples based on the distance function defined in Eq.
5 and use Eq.
4 to perturb them such that in the representation space modeled by θ ′ l , the augmentation of the virtual samples are closer to the paired sample from the current task, than the source/anchor samples x i and x j .
B Details of Related Work
We include detailed explanations about prior works which are relevant in the context of task-free continual learning:
• Experience Replay (ER) [10, 49]: ER maintains previously seen examples in a limited size constrained memory buffer for future replay.
It deals with the problem of memory retrieval and can be used in both task-free as well as task-dependent continual learning settings, but in all experiments in this paper, we use it as a task-free CL method.
As a new incoming batch from current task comes in, we draw a subset of examples from the buffer such that the size of the drawn subset is same as the incoming batch size.
In all experiments, we fix this subset size (also referred to as the replay budget) as
. • Gradient Episodic Memory (GEM) [37]: GEM also maintains an episodic memory for each observed task.
As a new incoming batch comes in, GEM projects the gradient of the model parameters such that loss incurred on the episodic memory maintained for each task does not increase while learning a new task.
This approach, thus needs access to the task-ids, and is computationally intensive owing to solving a QP program at each iteration of training.
• Averaged Gradient Episodic Memory (A-GEM)
[9]: Averaged-GEM is an improvement over GEM in terms of relaxing the need to require task-ids.
In stead of constraining the model parameters using the entire stored buffer, it randomly samples a subset of examples from the memory buffer to regularize the model parameters in order to minimize forgetting on previously seen tasks.
• Gradient-based Sample Selection (GSS) [3]: GSS deals with the problem of Memory Update in task-free CL setting.
It aims to maximize samples diversity in the gradient space while selecting examples for memory update.
The performance improvement observed is significant when different tasks data are imbalanced.
In all experiments covered in this paper, we have used reservoir sampling as the memory update method over all compared SoTA methods to ensure fair comparison.
• Maximally Interfered Retrieval (MIR) [2]: MIR is another memory retrieval method which does not need access to task-ids while selecting buffer exemplars for replay.
It proposes to first perform a virtual update (one-step look-ahead update) only on the incoming batch of the current batch.
It then computes the interference on the memory samples in terms of the loss increase observed w.r.t.
the virtual updated model parameters, and selects those buffer samples for which the loss increase is maximum — it selects the maximum interfered/forgotten samples from the buffer.
The implementation code provided by the authors that we have used is located at https://github.com/optimass/Maximally_Interfered_Retrieval.
• Adversarial Continual Learning (ACL) [16]: ACL focuses on learning two kind of features for task-dependent CL: (1) shared task-invariant (2) task specific features.
The architecture grows as a new task comes by adding a task-specific module.
It utilizes adversarial learning to train a shared feature encoder and a discriminator using the classic
minimax optimization objective (as used by GANs).
The role of the discriminator is to correctly classify the encoded features by their task labels/ids while the role of the encoder is to generate task-invariant features that can fool the discriminator.
RAR, on the other hand, focuses on the static architecture task-free CL and utilizes the idea of adversarial perturbations in the input space to perturb the memory replay samples such that they are located close to the forgetting frontier w.r.t.
to the current tasks.
• Adversarial Shapley Value Experience Replay (ASER) [55]: ASER proposes a scoring method based on kNN Shapley Value (SV) to select memory samples which are representative of the stored memory samples and adversarially close to the decision boundaries of new classes.
On the other hand, RAR’s objective is to minimize the worst-case experience replay loss by generating targeted adversarial replay perturbations that are visually similar to the chosen replay samples but the CL model mistakes them for the nearest class (from the incoming batch) based on their distance from the decision boundaries.
When we apply RAR’s objective on ASER chosen replay samples (i.e., compute X ′ M in Sec 3.2 using Adversarial SV), both accuracy and forgetting improve quite significantly (Table 1, 2), suggesting that learning on the RAR’s adversarial perturbations helps the model to learn about the boundaries between past and current tasks.
Also, we use Adversarial SV only for memory retrieval in order to ensure fair comparison across all methods.
The implementation code provided by the authors is at https://github.com/RaptorMai/online-continual-learning.
• Gradient-based Memory Editing for Continual Learning (GMED) [25]: GMED uses gradient updates to edit/perturb the stored replay samples, to create more challenging samples for rehearsal in a task-free setting.
This work is most relevant to RAR.
While their objective aims to perturb samples individually to generate increased losses in the upcoming model updates, our method looks into the local pairwise relationships between the most interfered memory samples and the new task’s samples.
By
utilizing such information, we locally perturb each replay sample in a targeted adversarial manner.
The model then confuses them with the new tasks’ data, creating more confusing examples close to the forgetting frontier as well as representative of previous data.
The implementation code provided by the authors is at https://github.com/INK-USC/GMED.
C Reservoir Sampling
8: end
9: end
We use traditional reservoir sampling method [62] detailed in Alg. 6 to update the limited memory buffer when a new batch’s data comes in.
Reservoir sampling uniformly samples from the incoming task’s stream by assigning a sampling probability of m/t where t is the number of samples seen so far.
It does not need any prior information about the length of the complete data stream.
Prior works based on task-free continual learning [2, 33, 37] have shown promising results when using reservoir sampling as the memory update method.
[26, 37] have proposed modified versions of reservoir sampling which help populate the memory with diverse samples in a balanced way when dealing with heterogeneous and long-tailed data streams.
Since, in this work, our focus is primarily on class-balanced datasets, we use Reservoir Sampling as the memory update method and plan to study other memory update methods which can be balanced in terms of the present classes in future work.
[Table]
D Dataset Details
Based on the taxonomy described in [61], the datasets that we studied fall under the class-incremental (CI) category.
The dataset details are as follows:
Split-MNIST is a variant of the MNIST dataset (http://yann.lecun.com/exdb/mnist/) of handwritten digits [31] split into five disjoints tasks based on the labels.
Each task consists of two classes, and we use only 1k examples from each task for training and report results on the complete test set.
The goal is to classify all ten classes at the end of the last task.
Split-CIFAR10 is a variant of the CIFAR-10 dataset [28] (https://www.cs.toronto.edu/ kriz/cifar.html) split into 5 disjoint tasks based on labels.
Each task consists of two classes and 10k training examples.
Split-CIFAR100 is a variant of the CIFAR-100 dataset (https://www.cs.toronto.edu/ kriz/cifar.html) comprises of 20 disjoint tasks with each task dealing with 5 classes.
This is 100-way classification problem since we do not use any task information during training/testing.
Split-miniImageNet splits the mini-ImageNet dataset [14] (https://lyy.mpiinf.mpg.de/mtl/download/) into 20 disjoint tasks based on labels.
Each task comprises ofclasses and 2.5k training examples.
In Table 4, we present the statistics of all four datasets.
Training samples used for each task is 95% of the mentioned numbers as 5% is held-out as validation set.
For fine-tuning the hyper-parameters associated with the RAR framework, we use the held-out validation set.
In case of Split-MNIST dataset, following the setting of previous methods such as [2, 25], we use only 1000 training examples per task.
E Complexity Analysis
We compare the replay-based methods in terms of forward and backward computations used, besides the forward features computed on the incoming batch X i and backward pass used for the final parameters updates:
ER : 1 extra forward pass on the replay samples i.e. f ( X ′ M ; θ ) 2.
ER-RAR : 1 backward pass for the virtual model update to θ ′ , n forward and backward passes to generate the targeted augmentations of X ′ M using θ ′ where n denotes the number of iterative steps used to generate the RAR data, 1 forward pass on the replay examples and the generated RAR data using θ . 3.
MIR : 1 backward pass for the virtual model update to θ ′ , 2 forward passes on the larger memory set, i.e. f ( X M ; θ ) , f ( X M ; θ ′ ) , 1 forward pass on the smaller set of replay samples i.e., f ( X ′ M ; θ ) .
MIR-RAR : 1 backward pass for the virtual model update to θ ′ , 2 forward passes on the larger memory set,
i
e. f ( X M ; θ ) , f ( X M ; θ ′ ) , n forward and backward passes to generate the targeted augmentations of X ′ M using θ ′ where n denotes the number of iterative steps used to generate the RAR data, 1 forward pass on the smaller set of original replay samples and the RAR augmented data using θ .
Thus, comparing ER-RAR w.r.t ER, ER-RAR needs n extra forwards passes and ( n + 1) extra backward passes on top of ER.
Similarly, MIR-RAR adds n additional forward and backwards passes on top of MIR.
Setting n = 2 for all datasets (except Split-MNIST with 1000 training samples per task)
meaning using only two gradient steps to optimize the RAR augmentation objective defined in
Eq. 4 in the input space, we limit the additional computational costs introduced by RAR.
Computational costs reduction based on Sec 3.2 and 3.3 : Instead of using all | X M | examples in the memory buffer M , replay data selection reduces the potential candidates to be considered for adversarial data generation to | X ′ M | .
For example, given a buffer size of 500 and replay budget of k = 10 , the potential buffer samples considered for generating RAR adversarial augmentations is only 10 instead of 500 .
Having reduced the size of the replay set to | X ′ M | , one would still need to consider every potential pairing between X ′ M and X i (current task samples) to find the most confusing subset of cardinality c which incurs maximum RAR loss defined in Eq.
Here, c is essentially the replay budget.
Generating targeted adversarial samples for each pair can be computationally prohibitive.
Our nearest neighbor-based pairing brings down the number of adversarial samples to be generated (by optimizing
Eq. 3) to c from | X ′ M | × | X i | .
In the above example for a current batch size of 10 , we only need to generate c = 10 targeted adversarial samples to approximately optimize Eq.
3 instead of generating 10 × 10 adversarial samples and then choosing c samples with the largest RAR loss.
F Implementation Details
Following the current SoTA methods [2, 10, 25], we use a MLP classifier with two hidden layers as the backbone architecture for Split-MNIST dataset.
For remaining datasets, we use the reduced
ResNet-18 classifier [22] with the output layer having as many units as the number of classes present in the entire dataset.
In all experiments based on experience replay methods, we use SGD optimizer with a learning rate of 0 .
1 for all datasets.
For GMED [25], ASER [55] based methods, we present the results using the set of hyperparameters reported in the papers.
Both mini-batch size and replay budget are set to 10 following above mentioned methods to make the comparisons fair.
For MIR based methods, we set the subset size c to 50 for all datasets following [2].
c denotes the size of the larger set of buffer examples on which the forgetting/interference score is computed.
Hyperparameters Tuning : We tune the hyperparameters associated with the RAR framework { n , β , ϵ } on the held-out validation set across all datasets.
For all datasets, we analyze (as shown in Figure 5) the sensitivity of the performance of RAR in terms of the number of gradient steps n used to perform the constrained targeted optimization in the input space as shown in Eq. 4.
Specifically, we study values of n ∈ { 2 , 5 , 10 } .
Based on the trade-off between the performance and the computational costs associated with the forward and backward passes employed to generate the targeted adversarial augmentation, we set n = 2 for all datasets except for Split-MNIST where we set n = 5 based on the average forgetting score as shown in Fig.
5a.
Since the MNIST dataset is relatively easier to train on, more gradient steps are needed to generate difficult samples which are indiscernible for the model in the learnt representation space after the look-ahead update on the new task’s data.
For the weight assigned to the ℓ RAR computed using Eq.
3 in the final loss objective (denoted by β ), we search over { 0 . 05 , 0 .
5 } and fine-tune them separately for different datasets across various buffer sizes that we study.
For Split-MNIST, β equal to 0 .
3 , 0 . 3 , and 0 .lead to the best reported performance across buffer size 200 , 500 , and 1000 respectively.
For SplitCIFAR10, these values are 0 .
5 , 0 . 1 , and 0 .
075 for buffer sizes 200 , 500 , and 1000 respectively.
For
Split-CIFAR100 and Split-miniImageNet, we get the best setting as β = 0 .
We search for the best perturbation tolerance ϵ hyperparameter in { 0 . 15 , 0 . 3 } for SplitMNIST, { 0 . 0314 , 0 . 07 } for Split-CIFAR10 and Split-CIFAR100, and { 0 . 01568 , 0 . 0314 } for SplitminiImageNet.
The best setting ϵ values are 0 .
3 for Split-MNIST and 0 .
0314 for remaining three datasets.
Prior works such as [46] show that the deeper/higher layers of neural networks are disproportionately responsible for catastrophic forgetting in line with the observation that the shallower/lower layers learn general features of and higher layers learn more task-specific features.
Therefore, we use the penultimate layer (layer just before the final classification/softmax layer) as layer l in Eq.
4 and in the representation space of θ ′ l , we minimize the augmentation objective such that the penultimate feature embedding of the RAR augmented sample and its paired current task sample are as close as possible, but visually still similar to the original buffer sample.
Compute Resources : We use a single NVIDIA GeForce GTX 1080 Ti to train our CL models.
The
PyTorch and CUDA toolkit versions used are 1.6.0 and 10.1 respectively.
G Limitations & Societal Impacts
Limitations :
In this work, we have utilized Reservoir Sampling across all compared methods to update the memory/buffer as a new batch comes in.
Using other recently proposed methods [3, 55] to tackle this problem could potentially help in maintaining a diverse & representative memory/buffer, particularly in case of class-imbalanced incremental datasets.
We also plan to explore updating the memory/buffer with perturbed samples (in stead of original samples) and see if re-learning them at later stages of training could help ameliorate forgetting.
Similar to other CL algorithms, our work is validated on small datasets with single pass training on streams generated in a synthetic fashion.
Therefore, the final average accuracy achieved on all set of classes is significantly small compared to traditional (i.i.d. assumption based) supervised training on the entire dataset.
Negative Societal Impacts : The potential improvements brought in terms of mitigating forgetting while learning on a continual stream of data might have some indirect negative impact.
For example, as spam detection systems are updated to prevent spam, bots can learn to continually evolve to defeat them.
Similar issues could arise in other surveillance systems in place.
The replay of RAR augmentations might introduce some bias if malicious attackers can manipulate the data stream in the continual learning setting.
H Background on Targeted Adversarial Augmentations (or Attacks) & Related Algorithms
Given a model θ , associated layer l of the model, source input-output pair ( x s , y s ) and a target input-output pair ( x t , y t ) , the goal of targeted adversarial attack/perturbation is to generate an input z which looks visually similar to the source input x s , but in the latent space of model’s l -th layer, the feature embedding of z is close to the feature embedding of the target input x t .
The objective can be mathematically summarized as:
[Equation]
Here, if we use the model’s last layer denoted by L , then often times L θ l ( z, x t ) can be replaced with the cross-entropy loss w.r.t.
to the target label y t of the paired sample x t i.e., ℓ ( z, y t ; θ ) .
In case we use layer l such that l < L , meaning layers lower/shallower than the last layer, one can use the defined distance function in Eq.
5 i.e., d θ l ( z, x t ) :
=
||
θ l ( z ) − θ l ( x t )
|| 2 as the minimization
objective L θ l ( z, x t ) in the above constrained optimization problem.
The Fast Gradient Sign Method (FGSM) proposed in [20] can be used to solve Eq.
11 when dealing with infinity norm constraints in the input space.
Specifically:
[Equation]
where ∇ x s L θ l ( x s , x t ) is the gradient of the loss function in Eq.
11 w.r.t.
x s .
Iterative-FGSM [30] which iteratively applies FGSM n times with a smaller step size α (proportional to ϵ/n ) can also be used.
Here n denotes the number of iterations used to iteratively optimize Eq. 11.
To stabilize the gradient update directions and avoid poor local maxima, [15] proposed MomentumIterative FGSM (MI-FGSM) which integrates momentum with a decay factor µ and accumulates the gradients of previous iterations while updating the targeted augmentation at each step.
At step t = 0 , we initialize x 0 s = x s .
Then, at step t , the accumulated gradient and targeted augmentation are derived as follows (the superscript t denotes the iteration step, while the subscript t denotes the target example):
[Equation]
At the end of each step, we project back into the norm ball
|| x t +1 s − x s || ∞ ≤ ϵ
and we also clip the values back to the input range observed corresponding to the original pixel values when using image data.
In this work, we use MI-FGSM with the decay factor µ as 1.0, the step-size α same as ϵ since we employ only two iterations to generate the RAR augmented data using buffer replay samples ( X ′ M , Y ′ M ) as the source/anchor sample.
The paired sample from the incoming batch of current task ( X i , Y i ) can be seen as the target sample.
I Additional Experiments
[Table]
To ensure fair comparison in terms of computational costs (extra forward and backward passes as described in Sec. E), in Table 5, we compare RAR to GMED [25], ER-3-iters, and MIR-3-iters.
3-iters means that the model is trained for three iterations on the incoming batch and buffer samples as a new batch comes in.
GMED also performs multiple iterations of gradient updates to edit/perturb the buffer samples.
Across all compared methods, RAR-based methods perform significantly better in terms of forgetting (lower is better) as well as final accuracy.
Comparison to knowledge distillation based methods : On Split-CIFAR10 dataset for two different buffer sizes, we augment RAR with DER [6] which utilizes knowledge distillation in the logits space.
DER which imposes regularization in the logits space can be seen as a complimnetary method to RAR and we observe that combining them together leads to further improvements in Table 6.
ER-DER is essentially DER that uses random sampling to select the replay samples and was proposed in the DER paper.
In MIR-DER, we use MIR scores to select the replay batch for DER.
In Table 6, aug means that we apply standard augmentations such as random cropping and rotations on the replay samples.
[Table]
Comparison to other augmentation techniques : In Table 3, we compare RAR to another perturbation method random which adds random noise of the same perturbation magnitude as RAR
[Table]
and show that RAR leads to significant improvement in reducing forgetting as well as increasing the average accuracy on different CL benchmark datasets.
In Table
[Table]
we compare to other standard augmentations techniques such as random cropping and rotations applied to the replay samples.
Utilizing these standard augmentation strategies along with RAR leads to impressive improvements in terms of the forgetting metric as shown in Table
[Table]
When using standard augmentations along with RAR, ER-RAR-aug and MIR-RAR-aug achieve similar performance irrespective of the memory retrieval/selection method used.
This indicates that
RAR, when combined with standard augmentations such as random cropping and rotations, is more
[Table]
robust to replay data selection and can generate high-quality replay samples (diverse, confusing, and representative of previously seen tasks).
However, such augmentation strategies rely on strong human priors, and manually selecting such augmentation operators requires domain expertise and prior knowledge of the dataset of interest [70].
They may perform poorly in the scenarios when human knowledge is weak for the targeted domain.
For example, for medical image analysis, such simple transformations are shown to be insufficient in capturing many of the subtle variations present in the data [69].
Moreover, they are static and pre-defined before the training so they cannot capture the training dynamics during the continual learning process for the purpose of reducing the forgetting.
RAR, on the other hand, automatically generates augmentations adaptive to the forgetting dynamics of the model and thus focuses on reducing the local interference near the forgetting frontier between different tasks.
Virtual update for RAR samples generation : When training sequentially on a set of tasks, catastrophic forgetting of previous tasks occurs as the network weights become biased to meet the objectives of the new task.
Performing a single-step virtual update of the model by exposing it only to the current task’s data helps us explore which buffer samples are most likely to be forgotten.
We use θ ′ , the virtually updated model to generate the adversarial samples using Eq.
4 as such perturbed samples are likely to be forgotten in future updates and confused with the current task’s classes leading to increased loss on them.
In retrospect, using the generated perturbed sample (visually close to the original sample x M in the input space and hence in-distribution) with its correct label y ( x M ) and replaying it while we are still at θ helps in alleviating forgetting.
As we combine the losses, we may also consider the procedure as first doing an update on the incoming task to get θ ′ and then performing an adversarial training step based on θ ′ using the RAR augmented samples.
One can argue that the step size used to perform this virtual update ( η virtual ) could be different from the step size ( η ) used to update the continual learner such that the adversarial samples generated using θ ′ are not too easy or too hard to classify by the current model θ .
We perform experiments on the
Split-CIFAR10 dataset with a buffer size of 500 and an SGD optimizer with a learning rate ( η ) of 0.1.
We find that using the same learning rate as the SGD optimizer to perform the virtual update results in the best performance on MIR-RAR as shown in Table 8.
[Table]
Ratio between incoming mini-batch size and replay budget : Following prior works [2, 25, 55], we select the same number of replay samples from the buffer as the mini-batch size of the incoming new task.
This is set to 10 irrespective of the buffer size.
Here, we perform additional experiments where we vary the ratio between the incoming mini-batch size and replay budget.
As expected, when the combined batch is dominated primarily by the incoming task samples (mini-batch size / replay budget = 15 / 5 ), we observe a significant drop on the overall performance with respect to both the average accuracy and forgetting as shown in Table 9.
However, when replaying more samples from the buffer (mini-batch size / replay budget = 5 / 15 ) in each training iteration, which over-compensates for forgetting, we do not observe any improvement in reducing the forgetting.
This suggests that to achieve a good plasticity-stability trade-off, the ratio between incoming mini-batch size and replay budget need to be close to one.
[Table]
J Visualizations of Learnt Decision Regions
[Figure]
describes the color used to denote different classes.
Remaining legends denote the classes corresponding to the chosen triplet samples.
[Figure] [Figure]
Inspired from recent works in decision boundaries visualization [57], we try to visualize the decision regions learnt by the continual learner along the data manifold.
To plot the decision regions, we use the plotting technique from [57] (https://github.com/somepago/dbViz).
Using a sampled triplet of points ( we construct
Specifically, for our use-case, once the training on all tasks ( 5 in case of Split-CIFAR10) is finished, we use the trained model to make predictions on the trained data.
We, then, select three training set instances ( x 1 , x 2 , x 3 ) on which the model makes the correct predictions in following ways: (1) all samples are from the same class, (2) two samples share the same class label while the third belongs to a different class, and (3) all three samples are from different classes.
We analyze the decision regions learnt by the ResNet-18 model using ER [10], MIR [2], MIR-RAR, and MIR-mix-RAR methods on the Split-CIFAR10 dataset.
Task to classes mapping is as follows:
Task 1 : {AIRPL, AUTO}, Task 2 : {BIRD, CAT}, Task 3 : {DEER, DOG}, Task 4 : {FROG, HORSE},
Task 5 : {SHIP, TRUCK} .
For case (1), we select all samples from class Auto and plot the regions in Fig 7.
It is expected that in case of worse forgetting, the decision regions would be dominated by the last task classes even when we use the first task’s samples to model the visualization plane.
In Fig.
7, the plane is mostly dominated by green (and blue) color in case of ER.
In case of MIR-mix-RAR, we see the plane mostly covering the class used to construct the visualization plane, even when the class selected ( Auto ) belongs to the first task the model encountered.
Similarly, in Fig.
8 (case (2)) & 9 (case (3)), as we go from left to right, we see the decision regions being more structured around the classes covered in the triplet set and less dominated by the color of the classes from the last task — this demonstrates that forgetting is minimized when using RAR and mix-RAR.