G ENERATIVE
N EGATIVE
R EPLAY
FOR
C ONTINUAL
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning continually is a key aspect of intelligence and a necessary ability to solve many real-world problems.
One of the most effective strategies to control catastrophic forgetting, the Achilles’ heel of continual learning, is storing part of the old data and replay them interleaved with new experiences (also known as the replay approach).
Generative replay, that is using generative models to provide replay patterns on demand, is particularly intriguing, however, it was shown to be effective mainly under simpliﬁed assumptions, such as simple scenarios and low-dimensional benchmarks.
In this paper, we show that, while the generated data are usually not able to improve the classiﬁcation accuracy for the old classes, they can be effective as negative examples (or antagonists) to learn the new classes, especially when the learning experiences are small and contain examples of just one or few classes.
The proposed approach is validated on complex class-incremental and data-incremental continual learning scenarios (CORe50 and ImageNet-1000) composed of high-dimensional data and a large number of training experiences: a setup where existing generative replay approaches usually fail.
I NTRODUCTION
The majority of neural networks training approaches assume that is feasible to build a set of independent and identically distributed (i.i.d.)
samples to train the model.
This assumption is in contrast with biological learning since intelligent beings observe the world as an ordered sequence of highly correlated data.
When state-of-the-art deep neural networks are trained continually, and the whole data cannot be accessed at once, the model suffers from the catastrophic forgetting problem (McCloskey & Cohen, 1989), and the knowledge about old data (old experiences) tend to be overwritten by new examples.
Storing part of past data and replaying them interleaved with new data proved to be an effective approach to mitigate forgetting (see Hayes et al.
(2021) for a comprehensive survey).
However, in some applications, the storage overhead together with privacy issues make replay techniques unfeasible.
Therefore, generative replay has been recently explored, where a generative model is trained to produce data from past experiences (see Lesort et al.
(2018) and Shin et al.
Besides solving the replay memory issue, generative replay can theoretically be capable of generating more general and novel examples not included in past experiences, thus potentially overcoming replay methods.
Unfortunately, generative replay introduces much complexity due to the need for an interleaved incremental training of both a classiﬁer and a generator.
Moreover, generative models are usually complex and unstable to train, especially in incremental scenarios.
Several researchers have shown that generative replay fails in complex CL scenarios with high-dimensional data (see Aljundi et al.
; Lesort et al.
(2018) and van de Ven et al.
mainly due to the inaccuracies in the data generation that progressively grows across the experiences if a single generator is incrementally updated (see related works in section 5 for more details).
The photocopy example helps to understand why.
Let us consider a high-quality photocopy machine: when a picture is initially copied the output looks very similar to the original, but if the process is repeated several times by using as input the output of the previous step, some artifacts will soon appear and, after many iterations, the result will be highly compromised.
Hence, even if some state-of-the-art models have been proved to be effective in generating also high dimensional data (Huang et al.
Karras et al.
, the continual training of such generators remains a challenging problem.
[Figure]
Although generative models are a hot research topics and we can expect improved methods in the future, as of today we must deal with imperfect generated data and try to exploit them at best when a classiﬁer is incrementally trained.
The proposed approach, denoted as Generative Negative Replay , does not attempt to improve the knowledge of old classes using the generated data because it assumes that the data quality is not enough for this purpose.
Nevertheless, it makes use of generated (latent) data as negative examples to better learn the classes of current experience, especially when the number of classes per experience is small and we incur in the “learning in isolation” problem.
We experimentally demonstrate, on complex benchmarks such as CORe50 and ImageNet-1000, where (positive) generative replay fails, that negative replay is effective to contrast the learning in isolation problem, allowing to train a classiﬁer incrementally across a high number of experiences (see Figure 1).
We also investigate the impact of data quality on negative replay with an ablation study (section 4.4) where negative examples are sampled from original past patterns (upper bound) and randomly generated.
P ROBLEM F ORMULATION
A continual learning (CL) problem consists of a number N E of experiences, each containing a subset of data that is only accessible during the corresponding experience:
[Equation]
each of them is composed of several data points and the corresponding labels:
[Equation]
where x ki and y ki are the data points and the associated labels contained in the k -th experience and
N k is the number of samples in the k -th experience.
Let D =
( X , Y ) be the entire dataset, then X = (cid:83) N E i =
1 X i and Y =
(cid:83) N E i =
1 Y i .
We can deﬁne three different scenarios for supervised continual learning (Maltoni & Lomonaco, 2019; van de Ven & Tolias, 2018): New Instances (NI), New Classes (NC), and New Instances and
Classes (NIC).
In NI (also known as domain incremental) all the classes are introduced in the ﬁrst experience, and only new examples of the same classes are included in the following experiences.
In
NC (also known as class incremental) each experience contains only examples from classes never seen before.
NIC is the combination of NI and NC, so each experience can be composed of new examples of already seen classes and/or examples from new classes.
A formal deﬁnition of the above scenarios can be found in Appendix A.
Given the above deﬁnitions, our goal is to ﬁt a function f , parametrized by Θ , to the sequence of experiences.
A naive approach is ﬁnding the best parameters Θ ∗ that minimizes:
[Equation]
where L ( · ) is a loss function (e.g. cross entropy loss).
As ﬁrst pointed out by McCloskey & Cohen (1989), this simple approach is prone to catastrophic forgetting, thus the model f Θ is not able to learn the experiences { e 1 , e 2 , ..., e N E } sequentially.
2. C ONTINUAL LEARNING WITH GENERATIVE REPLAY
Generative replay requires to train simultaneously and incrementally a classiﬁer and a generative model (Shin et al., 2017;
Wu et al., 2018; Thandiackal et al., 2021)
The generative model g , parametrized by Ω provides surrogate data similar to the past experiences’ data.
In the case of a conditional generative model (in which we can control the class of the generated data), the optimal parameters of the classiﬁer can be derived using a replay memory as follows:
[Equation]
where M xi and M yi are the datapoints and labels contained in the replay memory during the training on experience i when the replay memory is populated as:
[Equation]
where R is the number of generated replay patterns (size of memory), z is a latent random input vector to the generative model, c is a label sampled from the set of labels encountered in the past experiences, and “ ← ” indicates the insertion of an element in the memory.
The same generated data fed to the classiﬁer can be used to control forgetting in the generative model as well.
Instead of a generic generative model, suppose we have a conditional generative model composed of an encoder q γ parametrized by γ and a decoder p ξ parametrized by ξ , such that g Ω = p ξ ◦ q γ , Ω = ( γ, ξ ) .
The optimal parameters of the generative model can be obtained requiring that the generated data are similar (L2 loss) to the original ones:
[Equation]
where q γ ( X i ) is forced to follow a target distribution, typically N (0 , 1) .
G ENERATIVE NEGATIVE REPLAY
As discussed before, generative replay is an appealing strategy for continual learning, but, to exploit it in complex scenarios with many experiences, we need to overcome the data degradation issue.
Since this problem is not easily addressable on the generator side, we propose to circumvent it by changing the way the classiﬁer makes use of generated data.
Let us suppose the classiﬁer f Θ can be divided into a feature extractor f φ , parametrized by φ and a classiﬁcation head c ψ parametrized by ψ , so that f Θ = c ψ ◦ f φ , Θ = ( φ, ψ ) .
The parameters ψ of the classiﬁcation head can be divided into C groups, where C is the number of classes.
The groups, denoted as ( ψ 1 , ψ 2 , ..., ψ C ) represents the parameters associated to the connections between the features extracted by f φ and the output neuron of the corresponding class.
For simplicity, let us assume that the feature extraction weights φ are frozen (after an initial pretraining) and, across the experiences, we only learn the classiﬁcation head weights ψ .
As explained in Section section 3.3, this assumption is not necessary and our experiments were carried out by learning both φ and ψ .
3. L EARNING CLASSES IN ISOLATION
Learning in isolation is one of the main causes of catastrophic forgetting, especially in the NC or NIC scenarios where only a limited number of classes are present in a single experience, and the parameters of the classiﬁcation head are learned without negative examples that counteract the “greediness” of the optimization.
As an example, let us consider an NC scenario where only one class is present in each experience.
Suppose that c is the only class in the experience k , then the best way to optimize the model is to change the parameters ψ c to maximize the output of the output neuron c for every input and change the rest of ψ j , j (cid:54)
=
c to minimize the output for remaining classes.
This still holds if in the experience are present only a few classes, since the model is only optimized to discriminate between the present classes and has no interest in maintaining the past acquired knowledge.
3. P OSITIVE AND NEGATIVE REPLAY
Replay can be used to counteract the learning in isolation problem, however, when the replay data comes from a generative model, the data quality degradation has a negative impact on the classiﬁer training.
The aforementioned problem is typical of the standard generative replay approach (hereafter denoted as generative positive replay ), where replay data is used by the classiﬁer in the same manner of the current experience’s data, and therefore the classiﬁcation head’s weights associated to the replay classes are optimized based on the replay data.
On the contrary, in the proposed generative negative replay approach the replay patterns are used to counteract the detrimental effects of the training in isolation but they are not used to modify the parameters ψ associated with the replay classes.
The key idea (validated experimentally) is that the generated patterns are valid antagonists to mitigate the learning in isolation problem, but their quality is not enough to improve the knowledge of classes originally learned on real data.
It is well known that one class learning approaches are in general less effective than discriminative learning because the presence of both positive and negative examples allows to better characterize the classiﬁcation boundaries (Hempstalk & Frank, 2008).
Therefore, the proposed approach exploits generated data to constrain the classiﬁcation boundary and to avoid that the real data in the current experience pull it too much in their direction.
3. T RAINING A CLASSIFIER WITH GENERATIVE NEGATIVE REPLAY
The idea of generative negative replay is quite general can be used in conjunction with different continual learning classiﬁcation approaches and scenarios (NI, NC, NIC).
To avoid replay data (i.e. negative examples) alter the knowledge of the already learned classes, the gradient accumulation can be selectively blocked during the backward pass.
The general idea is illustrated in Figure 2.
While the original examples ( X i ) normally ﬂows forward and backward throughout the model, the replay examples ( M xi ) are passed forward, but, before the backward pass, the loss tensor is masked at the class level by resetting the gradient components corresponding to the classes in M yi .
The negative replay implementation illustrated in Figure 2 is discussed in more details in Appendix C where preliminary tests on Core50 NC are also included.
Hereafter, we provide an alternative implementation embedded in AR1 algorithm (Maltoni & Lomonaco, 2019), whose update mechanism for the classiﬁcation head weights allows very simple and efﬁcient integration of negative replay.
AR1 is a ﬂexible continual learning approach that can achieve state-of-the-art accuracy on complex CL benchmarks.
In Appendix E, AR1 is shown to outperform several recent CL algorithms of the difﬁcult ImageNet-1000 benchmark proposed by Masana et al.
AR1 uses different mechanisms to learn the classiﬁcation head and the feature extractor weights.
The feature extraction weights φ are protected against forgetting: i) through the Synaptic Intelligence regularization technique (Zenke et al., 2017) or ii) using a replay memory with a small learning rate (denoted as AR1free in Pellegrini et al.
The classiﬁcation head weights ψ are managed by
CWR.
CWR is a simple method aimed at addressing the score bias problem produced by imbalance learning during continual learning (Belouadah et al., 2020).
CWR (Maltoni & Lomonaco, 2019) maintains a copy of the weights of the classiﬁcation head of the previous experience ( ψ (cid:48) ) and at the start of each experience the classiﬁcation head is reset and only weights of classes of the current experience are loaded from ψ (cid:48) .
At the end of the experience, a weight consolidation phase takes place, where the weights ψ learned in the current experience are consolidated with the weights ψ (cid:48) .
This is the point where positive and negative replay behaves differently.
In particular, during the consolidation phase, for each parameter group ψ c associated to a class c belonging to the current experience ( c ∈ Y k ∪ M yk ), the mean of all the parameter group µ ( ψ c ) is calculated, and subtracted to all the parameters in the group, in order to force zero mean: ψ c = [Figure] ψ c − µ ( ψ c ) .
This prevents class bias problems due to the different magnitudes of the weights.
Then, there are three possibilities, based on c :
c is a new class never seen before ( c ∈ Y k
∧ c / ∈
(cid:83) k − 1 i =1 Y i ): in this case ψ c is maintained as is.
c is a class seen before ( c ∈ Y k ∧ c ∈ (cid:83) k − 1 i
=1 Y i ): the consolidation step is applied, so
[Equation]
where w past c is a parameter that balances the contribution of the past w.r.t.
the present, calculated as follows:
[Equation]
where
past
c is the number
of data
points
of class c encountered in past experiences, while current c is the number of data points of class c present in the current experience.
c is not in the current experience but is a replay example (
c / ∈
Y k
∧ c ∈ M yk ): • in case of positive replay apply consolidation (step 2).
• in case of negative replay ψ c is substituted with ψ (cid:48) c
(no contribution to the parameters ψ c from replay examples).
The pseudo-code of the above weigh consolidation algorithms is reported in Appendix B.
It is worth noting, that in the proposed embedding of negative replay in AR1, the replay pattern can alter the feature extraction weights since CWR weight consolidation only “protects” the classiﬁcation head
However, in our experiments, we found that a more complex embedding of negative replay in AR where we block the gradient propagation for negative patterns throughout the feature extraction layers performs very similarly, and therefore we opted for simplicity.
E XPERIMENTS AND RESULTS
In this section, we describe the experimental setup used to validate the proposed negative replay.
We focus on difﬁcult continual learning scenarios, where data is high-dimensional, non-i.i.d.
and the number of experiences is very large.
Negative replay is compared with alternative strategies (e.g. positive replay) and the role of quality of generated data is investigated by also using, as negative replay patterns, real and random data.
4. E XPERIMENTAL SETUP
Datasets We performed our experiments on the CORe50 dataset (Lomonaco & Maltoni, 2017) and
ImageNet-1000 dataset (Deng et al., 2009).
CORe50 dataset was speciﬁcally collected for continual learning (NI, NC, and NIC scenarios) and is composed of small video sessions (about 300 frames) of 50 objects taken from the point of view of a person that handles them in the hand.
Every class has 11 video sessions (a total of about 3,300 images) with different backgrounds and illuminations.
Eight video sessions for each class are used for training, and 3 for testing.
Images have size 128 ×
Classiﬁer architecture AR1 algorithm was used with Synaptic Intelligence (SI) regularization when trained without replay, and without protection on the feature extraction weights (AR1free) in case of positive and negative replay.
In the experiments with CORe50 dataset we follow Maltoni & Lomonaco (2019) and Lomonaco et al.
by employing a MobileNetV1 network (Howard et al., 2017).
As in Pellegrini et al.
(2020) and van de Ven et al.
, we opted for latent replay, that is replaying latent activations instead of input data.
As described in Pellegrini et al.
(2020), the choice of the latent replay layer is related to a tradeoff between accuracy and efﬁciency.
For CORe experiments, as in Pellegrini et al.
(2020), we used the conv5 4 layer as latent replay layer, and the classiﬁer was pretrained on ImageNet-1000.
We also substituted all the batch normalization layers of the network with batch renormalization (Ioffe, 2017).
For ImageNet-1000 we use a ResNet-18 (He et al., 2016) architecture.
Following the benchmark proposed by Masana et al.
(2020) the model was not pretrained. To maintain compatibility with the experiments on CORe50, even on ImageNetwe use latent replay, setting the replay layer on the fourth residual block of the network (after conv4 x using He et al.
(2016) nomenclature).
Generative model architecture For the choice of a generative model we initially focused on two state-of-the-art approaches whose implementations are open source (van de Ven et al., 2020; Shin et al., 2017; Ayub & Wagner, 2021).
However, since they were designed to work in simpler settings (with a lower data dimensionality and a smaller number of experiences), we were not able to port and scale them to our complex setups.
Therefore, we implemented a generative model by trying to combine the most promising techniques and ideas from different sources and control its overall memory/computation complexity.
In particular, taking inspiration from van de Ven et al.
we use a Variational Autoencoder (VAE) Kingma & Welling (2014) model, but unlike van de Ven et al.
we opted for a conditional VAE (cVAE) conﬁguration Sohn et al.
Moreover, we partially blend the generator (encoder) with the classiﬁer model: both the networks share the same feature extractor f φ .
Finally, instead of generating raw data, we generate activations at an intermediate “latent” level as suggested by van de Ven et al.
A detailed discussion on the architecture of the generator is provided in Appendix D, including a pseudo-code that highlights the details of the interleaved training of the generator and the classiﬁer.
4. E XPERIMENTS ON THE NC SCENARIO
The ﬁrst round of experiments has been performed on the NC scenario using CORe50 and ImageNet1000.
For CORe50 the benchmark is composed of 9 experiences: the ﬁrst one contains 10 classes while the following contains ﬁve classes each.
We used a replay memory of 1,500 patterns, and (for generative replay) we inserted in each minibatch, of size 128, 14 replay patterns, andpatterns from the current experience.
We train both the classiﬁer and the generator for 4 epochs for each experience.
Hyper-parameters of the classiﬁer and generator are reported in Appendix F and
Appendix G respectively.
For ImageNet-1000 the benchmark follows the one proposed by Masana et al.
(2020): the dataset is divided into 25 experiences of 40 classes each.
We used a replay memory of 20,000 patterns, and (for generative replay) we inserted in each minibatch, of size 128, 36 replay patterns, and 92 patterns from the current experience.
We did not expect negative replay to perform well in this setup, because each experience already contains 40 classes and, therefore, the learning-in-isolation problem is here marginal.
Nevertheless, we were interested in understanding if, in this setup, negative replay hurts the learning process or still provides minor beneﬁts.
The results are shown in Figure 3 and Table 1.
In CORe50 the baseline with no replay (using the
AR1 algorithm) reaches a ﬁnal accuracy of about 60% while using replay raises the accuracy to more than 70% (Positive Replay Original Data - PR-OD).
These were expected to be the lower and upper bounds of this experiment, respectively.
However, because of the data degradation problem, performing positive replay with generated data (Positive Replay Generated Data - PR-GD) performed signiﬁcantly worse than the case with no replay.
Using replay in a negative manner with generated data, as proposed in this work (NR-GD), only slightly decreases the ﬁnal accuracy with respect to the upper bound PR-OD. [Figure] [Figure] cy
[Figure]
[Figure] [Figure] [Figure] cy
[Figure]
[Figure]
[Figure] [Figure] [Table] For ImageNet-1000, due to the complexity of the experiment and the fact that the network is fully trained only during the ﬁrst experience (blocked after conv4 x in the following experiences) the ﬁnal accuracy are quite similar for all the methods (except PR-GD that performed far worse).
However, in the ﬁrst 10 experiences some differences can be appreciated: see the insect view in Figure 3-right.
The impact of the generated data quality on negative replay is more evident in Table 1: using negative replay with generated data (in this case highly degraded) improve the average accuracy (calculated as the mean of the accuracy after each experience) of more than 24 points and the ﬁnal accuracy of more than 10 points w.r.t.
using replay data in a positive manner.
Furthermore, even if in this scenario the advantage of negative generative replay is little with respect to the no replay case, we note that negative replay is not hurting the training process even in scenarios where learning in isolation is not an issue.
4. E XPERIMENTS ON THE NIC BENCHMARK
CORe50 NIC-391 protocol is composed of 391 learning experiences, each containing examples of a single class (300 frames of a short video).
This scenario is particularly challenging and prone to learn-in isolation issues, hence we may expect the role of replay to be more important here.
In this scenario, we used a replay memory of only 300 patterns.
The minibatch size is 128, and when generative replay is employed, we generate 64 patterns for every mini-batch (plus 64 from the current experience).
Hyper-parameters of the classiﬁer and generator are reported in Appendix F and
Appendix G respectively.
The results are shown in Figure 4 and they are quite in line with the previous experiment, but here the accuracy gaps grow and the beneﬁt of replay is more evident.
The proposed negative replay with generated data (NR-GD) performs quite well, about 10 points better than with no replay and just less than 5 points worse than positive replay with real data, the upper bound.
Using generated data in a positive manner (PR-GD) is here even worse than in the NC case, because the data degradation [Figure] is ampliﬁed with so many iterations: PR-GD is losing 30 points w.r.t.
not using replay at all, and performs about 40 points worse than using the same replay data with the proposed generative negative replay approach.
4. A BLATION STUDY
The effect of generated data quality on negative replay is investigated by performing two further experiments: NR-OD uses original data (max.
quality) for negative replay, while NR-RD uses randomly generated replay data, obtained by uniform random sampling in the latent replay layer and assigning to each data point a random class label.
Since in our experiments we replay hidden features, in order to produce reasonable replay data we ﬁrst calculated the range of latent activations on a sample dataset, and then we set our random generator to produce values in the range: 0 (since we use ReLU activation functions) - 90 th percentile of the real activation values.
We used CORe
NC and CORe50 NIC in these experiments.
The results are reported in Table 2.
Surprisingly, even with random replay data (that we assume to be the worst degradation possible), negative replay is still able to perform better than no replay.
Furthermore, the difference between original and generated data is minimal, thus proving that negative replay is tolerant in terms of data quality.
Note that in both the experiment using random data with negative replay performs way better than using generated data in a classical (positive) manner (PR-GD in previous ﬁgures).
Comparisons in all the benchmarks of all the experiments (positive and negative replay with original, generated, and random data) are reported in Appendix H.
[Table]
R ELATED WORKS
The use of negative examples to learn more discriminative class boundaries can be traced back to one-class support vector machines (SVM) (Chen et al., 2001), where the data points belonging to the other classes in the training set are used as negative examples. Malisiewicz et al.
(2011) proposed using an ensemble of one-class SVMs instead of a single multi-class classiﬁer. This approach operates in a scenario which is similar to the experiments on the CORe50 NIC benchmark, whose experiences contains only one class and all the replay data points (possibly belonging to many past encountered classes) are used as negative examples. The use of negative examples can also be seen as a kind of contrastive learning (Khosla et al., 2020), where negative examples are used to cluster embeddings of data points of the same class while moving away embeddings of data from different classes. Masking parts of a neural network has been experimented before in continual learning.
Wortsman et al. (2020) masked the weights of a randomly initialized neural network in order to ﬁnd a sub-network that yields good performance for a particular task.
The loss masking proposed for standalone negative replay introduced in section 3.3 and Appendix C (without using any continual learning strategy) is similar to the masking method proposed by Masana et al.
In that work, each feature can be used normally, masked (not used), or used only during forward (no modiﬁcation of the related parameters during network update).
Generative replay for continual learning was ﬁrst introduced by Shin et al.
(2017) who proposed Deep Generative Replay (DGR), using a generative adversarial network (GAN) (Goodfellow et al., 2014).
Many works on generative replay (Wu et al., 2018; Ostapenko et al., 2019) use GANs as generative models, but GANs are usually slow and complex to train, even in non-incremental scenarios.
Ayub & Wagner (2020) proposed to use autoencoders, however that approach requires maintaining a generative model for every experience, making it not scalable to long incremental sequences.
Kemker & Kanan (2018) and van de Ven et al.
(2020) proposed continual learning framework inspired to biological brain functionalities and memories.
In particular, (van de Ven et al., 2020) showed signiﬁcant results in continual learning scenarios with dozens of experiences. However, this approach was not tested on high-dimensional data and in much bigger scenarios.
C ONCLUSION
In this paper, we addressed the problem of continual learning with generative replay, focusing on the obstacles of generative replay in complex scenarios.
Our experience conﬁrms that incrementally training a generator over a long number of experiences with high dimensional data is a very challenging problem and remains an open issue.
Therefore, instead of trying to design a better generative model, we focused on classiﬁer training.
We found that even inaccurate replay data can be useful to contrast the learning in isolation problem, especially in scenarios where only a limited number of classes is present in each experience.
We called this approach negative replay since the replay data is used as negative examples when the model is trained with data from the current experience.
We validated negative replay using complex continual learning scenarios, with high dimensional data and hundreds of incremental experiences.
The results show that using negative replay largely improves classiﬁcation performances w.r.t.
using the generated data in a traditional fashion.
We also investigated the impact of generated data quality, by considering the two extremes of using original data and random data for negative replay, and, surprisingly, we found that negative replay is effective even using random replay data.
Since negative replay can be easily applied to other continual learning strategies (besides AR1), we believe that many other CL approaches may beneﬁt from our proposal, especially when complex scenarios are employed.
Moreover, negative replay could be used in the pre-training phase of large models, possibly making them more robust to noise or degraded data.
Finally, is worth noting that dealing with imprecise replay data can be viewed as a biological feature since human’s memory is far from being accurate, but is thought to be essential to consolidate learning (van de Ven et al., 2020), therefore investigating the role of negative replay-like mechanisms in biological learning could be an interesting research direction for computer scientists and neuroscientists.
R EPRODUCIBILITY S TATEMENT
The source code of the project, alongside conﬁguration ﬁles for the reproducibility of experiments, is included in the additional materials.
A public version of the code, based on the Avalanche framework (Lomonaco et al., 2021) will be released upon publication.
The datasets used are publicly available and can be downloaded from the respective ofﬁcial websites 1,2 .
Both the datasets are preprocessed normalizing them using statistics derived from ImageNet-1000.
Images from ImageNet-1000 were randomly cropped and resized to the ﬁnal dimension of 224 × 224 , then horizontally ﬂipped with a probability of 50% .
A NI, NC, AND NIC DEFINITIONS
Given the deﬁnitions of section 2, the NI (New Instances), NC (New Classes), and NIC (New
Instances and Classes) continual learning scenarios can be deﬁned based on the labels Y k contained in the experiences ( k ∈ { 1 , ..., N E } with N E the total number of experiences) as follows:
New instances (NI) also known as domain-incremental learning, where all the labels are known from the ﬁrst experience, and the successive experiences, only new instances of the same classes are included.
Formally, we could deﬁne the NI scenario as:
[Equation]
meaning that every possible label of the entire dataset must be present in the ﬁrst experience.
New classes (NC) also known as class-incremental learning, where each experience includes data of classes not present in any other experience.
Formally, we can deﬁne the NC scenario as:
[Equation]
New instances and classes (NIC) where a new experience can contain already seen classes, new classes, or a mix of the two.
This is the most natural scenario since in the real world an agent may sense both known and unknown objects.
Formally the NIC scenario can be deﬁned as:
[Equation]
B P SEUDO - CODE OF THE WEIGHT CONSOLIDATION PHASE
Require: ψ, ψ (cid:48) , Y e , M ye 1: for each class c ∈ Y e ∪ M ye do 2: ψ c = ψ c − µ ( ψ c ) 3: if c ∈ Y e ∧ c ∈ (cid:83) e − 1 i =1 Y
i then 4: ψ c = ψ (cid:48) c · w pastc + ψ c w pastc + 5: end if 6: if
c / ∈ Y e ∧ c ∈ M ye then 7: if positive replay then 8: ψ c = ψ (cid:48) c · w pastc + ψ c w pastc
+ 9: end if 10: if negative replay then 11: ψ c = ψ (cid:48)
c 12: end if 13: end if 14: end for 15: ψ (cid:48) =
ψ
C S TANDALONE NEGATIVE REPLAY IMPLEMENTATION
To validate the proposed negative replay approach, we implemented a version of negative replay that does not depend on any speciﬁc continual learning strategy.
As discussed in the main text, the main idea behind negative replay is to avoid replay data altering the knowledge of the already learned classes.
In other words, the replay data cannot change the weights of the already learned classes.
This behavior can be obtained by selectively blocking the gradient accumulation during the backward pass.
The original examples ( X i ) ﬂow normally through the network, accumulating the gradient in all the feature extractor weights ( ψ ).
On the other hand, the replay data points ( M xi ) participate in the loss calculation, but their backward contribution is limited to the weights associated with the output neurons of the original data classes.
The loss tensor associated to replay data is thus masked at class level, resetting the gradient contribution to zero for all the classes in M yi .
The procedure is graphically explained in Figure 5.
Note that the proposed standalone negative replay implementation does not provide any speciﬁc mechanism to counteract catastrophic forgetting problem, and since it applies negative replay on top of the naive CL strategy we denote the resulting strategy as naive negative replay .
In line with the experiments included in the main text, we compared naive negative replay (NR-GD) against naive with no replay (lower bound), naive with positive generative replay (PR-GD), and naive with positive replay with original data (PR-OD, upper bound).
We used the CORe50 NC benchmark to perform this experiment.
The results are shown in Figure 6.
As expected, the overall accuracy with naive is lower than with
AR1, but the relative ranking of the different replay approaches in maintained. In particular, negative replay performs better w.r.t. no replay (naive) or using the generated data in a positive manner (naive PR-GD).
In this scenario the gap between negative replay with generated data and positive replay with real data is larger than when using AR1, because replay is the only methods to contrast forgetting and the replay data quality is more relevant. [Table]
[Figure]
[Table] D D ETAILS OF THE GENERATIVE MODEL IMPLEMENTATION We designed our generative model using different insights from previous works in the ﬁelds, bringing together different ideas and proposals. We extensively tested the generative model alone to ﬁnd the better combination of building blocks that yield the best performance.
Our design choices have been also inﬂuenced by the computation complexity since our aim is to develop a (near) real-time system [Figure]
This is a particularly hard constraint since many incremental generative replay methods are based on generative adversarial networks (GANs) (Goodfellow et al., 2014), which notably have long training phases and often suffer from instabilities due to the adversarial nature of the training procedure.
As discussed in the main text, we took inspiration from some state-of-the-art methods, trying to combine promising techniques and ideas from different sources.
Taking inspiration from van de
Ven et al.
we use a Variational Autoencoder (VAE) Kingma & Welling (2014) model, but unlike van de Ven et al.
we opted for a conditional VAE (cVAE) conﬁguration Sohn et al.
So, while in van de Ven et al.
(2020) a mixture of Gaussian is used to sample latent vectors and soft labels are provided to the classiﬁer itself, in our approach the latent vector is sampled from the normal distribution N (0 , 1) and conditioned to the desired class.
This results in a faster and less complicated sampling of a replay pattern.
Moreover, as in van de Ven et al.
we partially blend the encoder part of the generative model with the classiﬁer model: in fact both the network share the same feature extractor f φ .
For the classiﬁer, this branch is connected with the classiﬁcation head c ψ , while, for the generator, it is connected with some other layers that transform the feature into a latent vector z .
The bifurcation is located in the latent replay layer.
The resulting on-the-loop training of the generative model is consistent with brain structures and neuroscience’s ﬁndings (van de Ven et al., 2020).
Since we use a cVAE, the objective for the generative model can be expressed as:
[Equation]
where ( x ki , y ki ) are the data point and the label of the i -th pattern of the k -th experience, and the D KL term represents the Kullback-Leibler divergence between the latent space distribution and the target distribution p ( z ) =
N (0 , 1) .
The two terms of Equation 11 determine two losses:
[Equation]
[Equation] We also add another loss term, denoted as classiﬁcation loss , which is similar to the classiﬁer loss adopted in the AC-GAN model (Odena et al., 2017).
The rationale is to guide the generative model to produce data that are not only visually similar to the original ones (L2 loss) but that is also classiﬁed [Figure] by the current classiﬁer in the same way.
Hence, we use f Θ as “auxiliary” classiﬁer, adding the following term to the generator’s loss:
[Equation]
which represents a typical negative log-likelihood classiﬁcation loss.
Note that the parameters Θ of the classiﬁer are not trained in this phase, since only the generative model is updated.
Overall, the generative model is trained using the following loss function:
[Equation]
where β is a hyper-parameter inspired to the β -VAE framework (Higgins et al., 2017), and η is a hyper-parameters that weights the importance of the classiﬁcation loss.
A visual representation of generative model training is shown in Figure 7.
To keep notation light, in the equations above the replay memory is not used, but it is trivial to include patterns from the replay memory, since there is no distinction in the generative model training procedure between current and replay data.
Note that the utilization of raw images is not mandatory for the method, and any intermediate (or latent representation) can be used, making our proposal compatible with latent replay methods (Pellegrini et al., 2020; van de Ven et al., 2020).
In fact, in the case of latent replay, the data points x ki in the above equations can be simply substituted with f φ (cid:48)
( x ki ) , where f φ (cid:48) is the set of feature extraction layers before the latent replay layer.
The blending of a part of the generative model into the classiﬁer poses some difﬁculties in the training, especially regarding the balancing of the two models and how to train each of them without destructive inference on the other.
After some initial experiments, we opted for blocking model parameters when the other model is trained.
Detailed pseudo-code for the proposed negative generative replay strategy is provided in Algorithm 2.
1: f Θ ← R AND I NIT or P RE T RAINED 2: g Ω ← R AND I NIT or P RE T RAINED 3: M x ← ∅ , M y ← ∅ 4: R = memory size 5: for each k from 1 tp N E do 6:
if k > 1 then 7:
S AMPLE { z 1 , ..., z R } ∼ N (0 , 1) 8:
S AMPLE { c 1 , ..., c R } ∼ (cid:83)
k − 1 t =1 Y t 9:
B LOCK generator parameters ( γ, ξ ) 10:
P OPULATE M xk = p ξ ( z j | c j )) , j = { 1 , ..., R } 11:
P OPULATE M yk = { c 1 , ..., c R } 12: end if 13: # classifier training 14: ψ (cid:48)
=
ψ 15:
B LOCK generator parameters ( γ, ξ ) 16:
U NLOCK classiﬁer parameters ( φ, ψ ) 17: φ ∗ , ψ ∗ =
O PTIMIZE ( f Θ , X k ∪ M x k , Y k ∪ M yk ) using Equation18:
W EIGHT C ONSOLIDATION ( ψ, ψ (cid:48) , Y k , M yk ) (see Algorithm 1) 19: # generator training 20:
B LOCK classiﬁer parameters ( φ, ψ ) 21:
U NLOCK generator parameters ( γ, ξ ) 22: γ ∗ , ξ ∗ =
O PTIMIZE ( g Ω , X k ∪ M xk , Y k ∪ M yk ) using Equation23: end for
E V ALIDATION OF AR1 ON I MAGE N ET -
To validate the chosen AR1 algorithm we performed a test on a competitive benchmark on ImageNet1000, following the NC benchmark proposed by Masana et al.
(2020), which is composed ofexperiences, each of them containing 40 classes.
The benchmark is particularly challenging due to a large number of classes (1,000), the incremental nature of the task (with 25 experiences), and the data dimensionality of 224 × 224 (as with ImageNet protocol).
With this experiment we want to assess the performance of AR1 in a complex continual learning scenario, validating the choice of AR1 as the baseline algorithm on which the tests on negative replay are conducted.
In this experiment, we tested AR1 against both regularization-based methods (Dhar et al., 2019;
Kirkpatrick et al., 2017; Li & Hoiem, 2016) and replay-based approaches (Belouadah &
Popescu, 2019; Castro et al., 2018; Chaudhry et al., 2018; Hou et al., 2019; Rebufﬁ et al., 2017; Wu et al., 2019).
We use the same classiﬁer (ResNet-18 (He et al., 2016)) and the same memory size for all the tested methods (20,000 patterns, 20 per class); for the regularization-based approaches, the replay is added as an additional mechanism.
For AR1, we trained the model with an SGD optimizer.
For the ﬁrst experience, we used an aggressive learning rate of 0 . 1 with momentum 0 .
9 and weight decay of 10 − 4 .
We multiply the initial learning rate by 0 .
1 every 15 epochs.
We trained the model for a total of 45 epochs, using a batch size of 128 .
For all the subsequent experiences we used SGD with a learning rate of 5 · 10 − 3 for the feature extractor’s parameters φ and 5 · 10 − 2 for the classiﬁer’s parameters ψ .
We trained the model forepochs for each experience, employing a learning rate scheduler that decreases the learning rate as the number of experiences progresses.
This was done to protect old knowledge against new knowledge when the former is more abundant than the latter.
As in the ﬁrst experience, the batch size was set to 128 , composed of 92 patterns from the current experience and 36 randomly sampled (without replacement) from the replay memory.
The results are shown in Table 3.
Replay-based methods exhibit the best performance, with iCaRL and
BiC exceeding a ﬁnal accuracy of 30%.
AR1 outperforms all the baselines (33.1%), demonstrating the validity of this approach also in difﬁcult continual learning benchmarks.
However, considering [Table] that top-1 ImageNet accuracy for a ResNet-18 when trained on the entire dataset is 69.76% 3 , even for the best methods the accuracy gap in the continual learning setup is very large.
This suggests that continual learning, especially in complex scenarios with a large number of classes and high dimensional data
[Table]
is far to be solved
[Table]
and further research should be devoted to this ﬁeld
[Table]
[Table] [Table] [Table]
[Table]
Common hyper-parameters are the same for each experience, 1st experience hyper-parameters are used in the ﬁrst experience
Due to the complexity of the ImageNet-1000 scenario
[Table]
we found it useful to use a learning rate scheduler that decreases the learning rate as the number of experiences progresses.
The scheduler can be formalized as:
[Equation]
where i indicates the index of the current experience.
[Table]
[Table]
The amount of replay data included in each minibatch has a direct impact on the performance of the continual learning strategy adopted. We observed that the optimal value changes with the quality of the replay data, and that a large amount of degraded replay data in each minibatch may decrease disruptively the performance of the model. We compared different original/replay proportions, ﬁnding that when using real replay data, the model is not much sensitive to the amount of replay data in the minibatch and different proportions work well: we empirically noticed a peak of performance around a 50-50 split.
Using generated (degraded) or random data is quite different. We noticed that if the data used is highly degraded the maximum gain in performance is when 10-30% replay data are added. Exceeding 30% usually leads to a degradation of performance, and if the amount of replay data is still higher (depending on the replay data quality) the accuracy of the model can be lower than not using replay data.
G G ENERATIVE MODEL HYPER - PARAMETERS G.
[Table] [Table]
Equation
[Table]
G
[Figure]
[Table]
Common
[Table]
Following experiences replay patterns in the minibatch
H A DDITIONAL PLOTS
[Figure]
[Figure] [Figure] [Figure]