L OOKING B ACK ON L EARNED E XPERIENCES F OR C LASS / TASK I NCREMENTAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Classical deep neural networks are limited in their ability to learn from emerging streams of training data.
When trained sequentially on new or evolving tasks, their performance degrades sharply, making them inappropriate in real-world use cases.
Existing methods tackle it by either storing old data samples or only updating a parameter set of deep neural networks, which, however, demands a large memory budget or spoils the ﬂexibility of models to learn the incremented task distribution.
In this paper, we shed light on an on-call transfer set to provide past experiences whenever a new task arises in the data stream.
In particular, we propose a DataFree Incremental Learning (DF-IL) not only to replay past experiences the model has learned but also to perform this in a data- manner.
Towards this end, we introduced a memory recovery paradigm in which we query the network to synthesize past exemplars whenever a new task emerges.
Thus, our method needs no extra memory for data buffering or network growing, besides calls the proposed memory recovery paradigm to provide past exemplars, named a transfer set in order to mitigate catastrophically forgetting the former tasks in the Incremental Learning (IL) setup.
Moreover, in contrast with recently proposed methods, the suggested paradigm does not desire a parallel architecture since it only relies on the learner network.
Compared to the state-of-the-art data techniques without buffering past data samples, DF-IL demonstrates signiﬁcantly better performance on the wellknown datasets (CIFAR-10, Tiny-ImageNet) whether a task oracle is available in test time (Task-IL) or not (Class-IL) 1 .
I NTRODUCTION
Conventional Deep Neural Networks (DNNs) are typically trained in an ofﬂine batch setting where all data are available at once.
However, in real-world settings, models may incrementally encounter new classes when trained online.
In such scenarios, deep learning models suffer from catastrophic forgetting McCloskey & Cohen (1989), meaning they forget the previously obtained knowledge when adapting to the new information from the incoming observations.
This is mainly because, models overwrite the decisive parameters for earlier tasks while learning the new one.
Incremental Learning (IL) methods aim at training a single DNN from an unlimited stream of data, mitigating catastrophic forgetting conditioned on the limited computational overhead and memory budget.
In the IL literature, the typical setting is a model to learn many tasks sequentially where each task contains one or more disjoint classiﬁcation problems.
The majority of the existing works assume that task identities are provided at the test time so that one can select the relevant part of the network for each example.
This setup has been named task-IL, where a more general circumstance is that task labels are available only during training, has been named class-IL.
Recently, various approaches have been put forward to solve catastrophic forgetting.
Some works investigate on parameter isolation approaches using either expanding network architecture (Aljundi et al., 2017; Xu & Zhu, 2018; Yoon et al., 2017; Rosenfeld & Tsotsos, 2018) or considering a ﬁxed-sized set of model parameters (Masse et al., 2018), pruning (Mallya & Lazebnik, 2018) to learn a new task.
Thus, they are only applicable to the task incremental scenario.
Besides, several studies suggest a regularization term added to the loss function (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017; Nguyen et al., 2017) to preserve the knowledge of previous classes without a need for knowing the task label during inference.
Despite the increasing number of these methodologies, they can not reach a reasonable performance.
Another followed terminology is experience replay-based methods which store actual data samples from the past (Buzzega et al., 2020; Riemer et al., 2018; Chaudhry et al., 2018) to alleviate catastrophic forgetting problem, where more of the recent state-of-the-art (SOTA) works fall into this category.
Despite the success of this approach, they suffer from a main limitation that is the ﬁxed memory budget when increasing the number of classes, leading to severe performance degradation.
To overcome this constraint, several works propose to generate past data samples (Zhai et al., 2019), as pseudo-data examples, conditioned on the model underlying distribution.
Although their method are considered a data-free approach by discarding the need of storing real data, they achieve better performance than previous data-free works.
However, one main drawback that these approaches encounter is a need for a highcapacity auxiliary network to generate data which increase the complexity of the system in terms of training-difﬁculty and computation overhead.
In this paper, we propose a novel Data-Free Incremental Learning (DF-IL) paradigm, which allows the model to remember previous knowledge by itself, thereby eliminating the need for external memory.
Also, in the proposed method, there is no need for a parallel network to generate past exemplars.
Besides, we ask the learner network to remember what it has learned before learning new classes.
Inspired by (Fredrikson et al., 2015;
He et al., 2019; Nayak et al., 2019), we propose a solution to reply the previous experiences by recovering the learner network memory.
The main idea is having only the learned network to not only remember what it has learned but also stay learned while encountering newly received knowledge, much like how humans continually learn and remember them by themselves with no external help.
The most likely method to learn in this way is the regularization-based method.
However, they still could not compromise between humanlike learning and performance.
Besides, our method can achieve a better result by a signiﬁcant margin with no additional requirement.
In summary, the main contributions of this paper are: (1)To the best of our knowledge, we introduce, for the ﬁrst time, the idea of DF-IL to make experience replay without a need to a parallel auxiliary network, or any other additional requirement, (2)Since we only rely on the learner network, there is no need for buffering past data samples, extracted prior knowledge, and a parallel high-capacity network, resulting in signiﬁcantly reduced computational cost and memory footprint, and (3)Our method compromises between decreasing memory budget and maintaining performance comparing to the current SOTA task on both class and task incremental learning methods.
R ELATED W ORK
Several studies have been carried out to tackle catastrophic forgetting which falls into four categories as follows.
Model Growing: This approach is the ﬁrst investigations into incremental learning in which a new set of network parameters is dedicated for each task in a dynamic architecture networks (Aljundi et al., 2017; Xu & Zhu, 2018; Yoon et al., 2017; Rosenfeld & Tsotsos, 2018).
Therefore, they require the task label to be known at test time to trigger the correct state of the network, which can not be employed in the class-IL setting where task labels are not provided.
Moreover, maintaining class incremental performance can be met at the cost of limited scalability.
Memory Replay: These methods allow access to a ﬁxed memory buffer to hold actual samples (Rebufﬁ et al., 2017; Isele & Cosgun, 2018; Chaudhry et al., 2019; Rolnick et al., 2018).
When learning new tasks, samples from the former tasks are either reused as model inputs for rehearsal or to constrain optimization (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018) of the recent task loss to prevent preceding task interference.
Although this approach is pioneering, its main limitation is a ﬁxed budget, which caused the bias problem (Hou et al., 2019; Wu et al., 2019) by the imbalanced number of old and new class training samples, which destroys the performance (Belouadah &
Popescu, 2019).
Recent works use distillation loss (Javed & Shafait, 2018) or unlabeled data (Lee et al., 2019) to solve this problem, thereby improving learning performance.
However, in the real world, keeping past samples may not be allowed because of privacy concerns.
In the meantime, some works explore a pseudo-memory replay (Robins, 1995;
Atkinson et al.) approach in which a generative model is adopted to synthesize past samples when facing a new task.
Although these methods omit the need for buffering past samples, they mostly employ an auxiliary network to generate samples, increasing training complexity.
An exceptional approach is introduced by (Choi et al., 2021) in which only the main network is needed to generate past samples. Although it is the most similar work to ours, they suffer from more training phases because they need to retrain the network from scratch to synthesize new task samples whenever a new task arrives.
Regularization-based: These works penalize the model for updating important parameters for earlier tasks by adding a regularization term to the loss function.
The importance score of each parameter can be estimated through the Fisher Information Matrix (Kirkpatrick et al., 2017), gradient extent (Aljundi et al., 2018), and uncertainty estimation using Bayesian Neural Network (Nguyen et al., 2017).
Although the vast amount of works falls into this family, soft penalties might not be sufﬁcient to restrict the optimization process to stay in the feasible region of previous tasks, particularly with long sequences (Farquhar & Gal, 2018), resulting in increased forgetting of earlier tasks.
Parameter Isolation: These methods rely on estimating binary masks that allocate a ﬁxed number of parameters for each task where the network architecture remains static.
Such masks can be estimated either by random assignment (Masse et al., 2018), pruning (Mallya & Lazebnik, 2018), or gradient descent (Mallya et al., 2018; Serra et al., 2018).
However, this family is limited to the task incremental setting and better agreed for learning a long sequence of tasks when model capacity is not a concern, and optimal performance is the priority.
P ROBLEM SETTING
Without loss of generality, a classical IL scenario can be assumed as a set T = { T 1 , . .
, T N } of N different tasks.
Each task T i takes the form of a classiﬁcation problem, T i = { x j , y j }
i × l j =1 , where y j ∈ { 1 , . . .
, l } .
When a new task arrives we ask the network to learn to classify the new images, being simultaneously able to solve the former tasks.
Our method only relies on a single DNN network, named the learner network L , in all steps with no external memory and no parallel network.
We formalize the setting conditioned on the absence of a task oracle at test time as follows:
TaskIL and Class-IL.
In the Task-IL setting, the learner network L is composed of a feature extractor followed by a multiple heads where each head is composed of a linear classiﬁer with a softmax activation which computes task-speciﬁc classiﬁcation probabilities.
On the contrary, in the Class-IL setting, the learner network has a single head while is shared over all the tasks that make the Class-IL more challenging.
In this paper, we consider both settings and report the results regarding them.
P ROPOSED METHOD
Availability of the actual data samples can enable solving for straightforward retraining, thus learning incrementally can overcome catastrophic forgetting problem.
However, preserving earlier data is not straightforward in real-world scenarios due to computational overhead and privacy concerns.
In this paper, the main idea is to recall (instead of preserving) past exemplars, known as network experiences, without using either a parallel network or memory buffer.
To this end, we propose DFIL in which, whenever a new task arises, we trigger a memory recovery paradigm R ( L ) to obtain a transfer set S of past experiences as surrogates for the earlier tasks observations.
Then, we form a combined training dataset D from the recovered transfer set S and the current incremented class(es)
I data.
Thus, the learner network L , which is supposed to learn the task sequences T incrementally, is retrained on D in each time step.
In the suggested memory recovery paradigm, we ﬁrst model the learner network output L regarding the case when a real sample from the past is fed to the model.
These outputs are provided by generating initial output vectors {V} and reﬁning them by applying the
F ( V , γ ) constraint, as a supervision, to perform adjustment due to underlying distribution obtained from a dynamic recommender γ , resulting in reaching {V} ∗ .
Fig.
1 shows a clear overview of the proposed method.
In the rest of this section, we ﬁrst describe the proposed DF-IL and then outline the suggested memory recovery paradigm in detail.
4. DATA - FREE I NCREMENTAL L EARNING
In this work, the data-free incremental learning term expresses that there is no need for a memory buffer to learn incrementally, while the DNN model complexity is ﬁxed. Besides, we synthesize a [Table] L
[Table]
[Figure]
[Figure] F ( {V} , γ ) using a dynamic recommender vector γ to adjust due to underlying distribution. Then, a random noise input is initialized to be optimized conditioned on the reﬁned output {V} ∗ . This procedure is performed for each learned class several times to form the transfer set S .
Finally, the learner network is retrained on the combined dataset D , including the transfer set S and new incremented task data I using a two-part cost function composing a classiﬁcation (CE) and knowledge distillation (KD) terms, respectively.
transfer set S as past experiences of the network in a data-free manner.
To put this idea into practice, we run the following steps, which are listed in Alg. 1.
Whenever DF-IL receives a new incremented task data
I = { ( x i , y i ) , 1 ≤ i ≤ L, y i ∈ [1 , . . .
, l ] } , where L is the total number of samples of l classes belonging to the new incremented task T t , it initiates the memory recovery procedure
R ( L ) (Alg. 2) to create the transfer set S =
{ (ˆ x j , ˆ y j ) , 1 ≤ j ≤ K, ˆ y j ∈ [1 , . . .
, k ] } , where K is the total number of synthesized samples of k classes that have been learned by the learner network L .
Next, the samples from the transfer set S is fed to the learner network, and the resulting network’s logits
O =
[ o 1 ( x ) , . . .
, o K ( x )] for all samples are stored.
Then, we form a combined dataset D per each task T i , i ∈ [1 , . . .
, N ] in the task sequence T .
Finally, the learner network L parameters are updated to minimize a cost function such a way that each data sample from the new incremented class I will be classiﬁed correctly, as classiﬁcation loss, and for the samples in S , current network’s logits will be reproduced as close as those have been stored in the previous step, as knowledge distillation loss by KD term.
We employ the softmax cross entropy as the classiﬁcation loss, which is computed using Equ.
[Equation]
where δ y
=
i is the indicator function and p i ( x ) is the output probability (i.e, softmax of logits) of the i th class in l new incremented class(es).
For the distillation purpose, we adopt knowledge distillation from network output (logits), known as dark knowledge Hinton et al.
; Wang & Yoon (2020); Buzzega et al.
and our objective is to minimize the Euclidean distance between the stored logits in the transfer set O =
[ o 1 ( x ) , . . .
, o K ( x )] and those generated by the learner network
ˆ O =
[ˆ o 1 ( x ) , . . .
, ˆ o K ( x )] as follows:
[Equation]
Having a look at both cost functions, we deﬁne the total loss function by a linear combination as below:
[Equation]
where λ is predeﬁned parameter to control the degree of distillation.
[Figure]
The memory recovery paradigm is devised to synthesize the previously learned experiences (knowledge) by L .
In particular, we synthesis past samples in a way that the learner network strongly believes them to be actual samples that belong to categories in the underlying data distribution.
Therefore, they are the network acquaintance that might not be natural-looking data.
It is worth noting that we only use the learned parameters of the learner model L since they can be interpreted as the memory of the model in which the essence of training has been saved and encoded.
In the suggested paradigm, we ﬁrst model the network output space using a two-stepped approach and then generate synthesized samples by back-propagating the modeled output through the network, where each is described in the rest.
The whole procedure is shown in Alg. 2.
Modeling the Network Output Space
In the ﬁrst phase, we model the output space of the learner network.
Suppose y i,j ∼ L ( x, θ L ) ( T i ,C j ) is the actual model output when j th class of i th task observation was fed to the network.
To reproduce such output in the absence of actual previous one, ( V ∗ ) i,j ∼ y i,j , we consider a two-stepped approach as is explained bellow.
In (Nayak et al., 2019), it is investigated that sampling the output vector from a Dirichlet distribution in which ingredients are in the range [0 , 1] and whose sum is one is a straightforward way for modelling the output space of the learner network.
Despite interesting results, this method lacks preserving extra class similarity, hence generating outlier vectors that do not follow the networks underlying distribution.
Thus, false memory happens when we query the memory for a speciﬁc target.
In this case, the result of memory recovery is a mixture of some targets that do not really exist in the past experience, resulting in confusing the learner in the retraining phase.
In our novel memory recovery process, we make supervision as the second step, to detect such outliers to boost the retrieved memory in terms of generating a single-target-based transfer set.
To do this, we make supervision by applying a constraint on the generated vector F ( V , γ ) , in which an arbitrary generated vector from the previous step V is a good candidate only if it has a distance lower than a predeﬁned threshold η to a dynamic recommender γ .
This recommendation vector supposed to be declared at a low-cost and represent the class similarity as well.
Considering these concerns, we imply a dynamic confusion matrix CM that is constructed incrementally when training each task.
An arbitrary confusion matrix is a table that is typically used to describe the performance of a classiﬁcation model.
Intuitively, when a model wrongly classiﬁes some samples to class c , it means class c is highly correlated with the target class.
On the other side, when a model always correctly classiﬁes class c from the target class, it means class c is highly distinguishable from the target class.
This is the exact expected result from a DNN model output.
Mathematically, we check if the generated vector V i for arbitrary class i , has a distance smaller than a predeﬁned threshold η to the i th row of the constructed dynamic confusion matrix CM i , so as to obtain the checked output vector V ∗ i .
Back propagate Through the Network In the second phase, conditioned on the network’s generated outputs {V ∗ } , we synthesize a transfer set S .
For an arbitrary generated vector V ∗ , we start with a random noisy image x sampled from a uniform distribution in the range [0 , 1] and update it till the cross-entropy (CE) loss between the generated model output V∗
and the model output predicted by the learner network L ( x, θ T , τ ) is minimized.
[Equation] [Equation]
where τ is the temperature value used in the softmax layer (Hinton et al., 2015) for the distillation purpose.
This procedure is renewed for each of the k classes has been learned K/k times, where K is the whole number of samples to be formed the transfer set S .
The proposed paradigm is a remedy for memory bottlenecks so that the model alone can alleviate catastrophic forgetting, resulting in remarkable performance among the data-free works on both task-IL and class-IL, as will be seen in the next section.
Algorithm
Input : Task sequence set T , Transfer set size
K , Learner model parameters θ L
Require: Confusion Matrix CM from L ,
Incremented class(es)
I // memory recovery paradigm
S ← R ( θ L , K, CM ) L : number of samples in
I // store network outputs and Ground
Truth for S with pre-update parameters for ( x, y ) in S do
O ← L ( x ; θ L ) // form combined training set
D ← (cid:83)
[Equation] [Equation]
// define a two-parted loss function (cid:96) Total = Σ (
[Equation]
[Equation] [Equation]
+
[Equation]
] // optimize the network for the total loss for ( x, y ) in range ( K + L ) do sample a batch b 1 form
D , where ( x, y,
O ) ∈ S sample a batch b 2 form D , where ( x, y ) ∈
I update θ L by taking a SGD step on b 1 + bloss (cid:96) Total ( θ L )
Algorithm 2:
Input : Learner model parameters θ L ,
Transfer set size K , Confusion
Matrix CM
Output : Transfer set S Require: Dirichlet distribution parameters α, β , Thresh.
η ,
Temperature for distillation τ k : number of classes from L p : K/k // Number of samples per class
S ← {} Constraint
F ( X , γ ) ← ( (cid:107)X − γ (cid:107) 2 2 < η ) for i = 1 : k do γ ← CM i for j = 1 : p do do
Sample V ij ←
Dir (cid:0) d, β × α i (cid:1)
V ∗ ij ← V ij while Constraint F ( V ij , γ ) passed
Initialize x ij to random noise ¯ x ij ← optimize x ij with (cid:96) CE (cid:0) V ∗ ij , L (cid:0) x ij , θ L , τ (cid:1)(cid:1)
S ← S ∪ ¯ x ij
E XPERIMENTS 5. E XPERIMENTAL S ETUP
We evaluate our method on two commonly-used datasets for incremental image classiﬁcation tasks:
CIFAR-10 (Krizhevsky et al., 2009) and Tiny-ImageNet (Le & Yang, 2015).
To form incremented tasks, we randomly choose 2 , 20 classes for each task resulting in a sequence of 5 , 20 sequential tasks for CIFAR-10 and Tiny-ImageNet, respectively, which is a typical conﬁguration in literature.
Following iCaRL (Rebufﬁ et al., 2017), a ResNet18 (He et al., 2016) is used in the learner network without pre-training.
To perform a fair comparison with other IL methods, we train the networks using the Stochastic Gradient Descent (SGD) optimizer with a learning rate of 0 .
1 with other parameters set to their default values.
We consider the number of epochs per task concerning the dataset complexity; thus, we set it to 50 for Sequential CIFAR-10 and 100 for Sequential Tiny-ImageNet, respectively.
In the training phase, we select a batch of data from the incremented task and a minibatch of data from the transfer set S , where depending on the hardware restriction, we set both to 32 .
We show all the classes in the same ﬁxed order across different runs, and results are averaged across ten runs, each involving a different initialization.
For visual simplicity, we only report mean values.
Following the convention of the ML community, hyperparams are selected by performing a grid-search on a validation set, obtained by sampling 10 % of the training set.
In the Memory Recovery Paradigm, we consider the temperature value τ to 20 for the distillation purpose.
In the Dirichlet distribution, we set β in [1 , 0 . 1] for each dataset, where half the transfer set S is synthesized by 1 and
[Figure] [Figure]
the rest with 0 .
1 as in (Nayak et al., 2019).
The η value in the constraint step is empirically valued at 0 .
To optimize the random noisy image, we employ the Adam optimizer with a learning rate of 0 .
01 , while the maximum number of iterations is set to 1500 .
For all experiments, we consider the average accuracy across tasks as the evaluation criteria.
5. A BLATION S TUDIES
Synthesized Transfer Set: We follow a two-step strategy to generate the network output in our proposed model memory recovery process.
Firstly, we follow the approach adopted in (Nayak et al., 2019) in which a Dirichlet distribution Dir( d, β × α ) is utilized to generate the vectors in sotmax space, where we need to set two parameters in this distribution: α , and β ( d is the dimension of the expected vector).
α is the concentration parameter to control the probability mass of a sample from the distribution.
Decreasing its value caused a vector that is hugely concentrated in only a few ingredients and vice versa.
α is interpreted as a class similarity vector obtained from the network’s ﬁnal layer and can be scaled with the β parameter.
This method encounters false memory since some generated vectors are a mix-up of several targets.
To tackle this problem, we suggest applying a constraint to reﬁne these vectors.
An examples of false memory and a correct one for the target class dog are shown in Fig.
Moreover, we present several samples of retrieved images belonging to several classes, when querying the learner network L using our novel memory recovery paradigm, as shown in Fig.
We can see how the network retains its learned knowledge in its memory which is a speciﬁc pattern representing the target classes.
Effect of transfer set size: We examine the impact of transfer set size on the performance of the incrementally learning classes.
To this end, we set up the proposed DF-IL on CIFAR-10 dataset with different sizes of transfer sets, including [50 , 500 , 5000 , 6000 , 7000] , wherein in the one before the latest case, the retrieved data from the past is equal to the incremented class.
Fig.
3 (a) shows the performance.
Obviously, increasing the number of synthesized samples in the transfer set has a signiﬁcant impact on the performance.
It is worth noting that an optimal size for the transfer set depends on the task complexity in terms of the number of classes and variations of the actual images.
Thus, increasing the set size higher than a reasonable one might increase the risk of overﬁtting as we can see this case when the transfer set size is equal to 7000 in the
Data-Free vs Data-Allowed Incremental Learning:
To further evaluate the proposed method, we also investigate the Data-Allowed Incremental Learning (DA-IL), where a few samples from the past are stored to be integrated with the transfer set S when retraining the learner network L to learn a new incremented class I .
In CIFAR-10 experiment, we store only 50 samples for each class.
We report the result in Table 1.
Obviously, the ablated model DA-IL outperforms the DF-IL at the cost of requiring an external memory to save past examples, which means our method is not only a standalone approach but also an extension to boost the performance of memory-based methods by providing a more balanced dataset at each time step.
[Table]
Hyper-parameters: λ and η : λ controls how much the transfer set S should be attended as Eq. 3, and η is related to how much the generated model output should be close to the recommendation vector.
To select the best one for the two mentioned parameter, we perform grid search, where a brief of examination are reported in Fig.
3 (c) and (d) , where a take λ and η as 0 . 3 and 0 . 45 , respectively.
DF-IL in memory-based works: Our MRP is an alternative to the buffer-based methods to omit the need for a memory buffer and decrease the risk of overﬁtting due to the more balanced ﬁne-tuning at the same time.
To validate this assertion, we embed our memory recovery paradigm into a prominent method iCaRL(Rebufﬁ et al., 2017).
Performance result are shown in Fig.
3 (b) .
From the ﬁgure, we can see adopting our DF-IL can compromise between performance and memory footprint.
5. C OMPARED M ETHODS
To have a fair comparison, we ﬁrst should determine that our work falls into which categories are mentioned in Sec 2.
Secondly, discuss two main aspects that should be considered when it comes to comparing methods: Performance, complexity.
Obviously, our method is neither model-growing
PNN (Rusu et al., 2016) nor parameter-isolation since our learner network has a ﬁxed architecture while the whole parameters are share among all tasks.
In terms of replaying past samples, DF-IL may be counted as a member of memory-replay-based group MER (Riemer et al., 2018), iCaRL (Rebufﬁ et al., 2017), DER (Buzzega et al., 2020), GEM (Lopez-Paz & Ranzato, 2017), a-GEM (Chaudhry et al., 2018), GSS (Aljundi et al., 2019), TPCIL (Tao et al., 2020), Mnemonics (Liu et al., 2020), BiC (Wu et al., 2019) that more recent SOTAs fall into this category.
However, keeping past examples in a ﬁxed-size buffer is not applicable when growing the number of tasks and has a memory overhead resulting in increasing complexity of the system.
Keeping these issues in mind, we compare our method with prominent works in this category while allocating the same buffer size and omit the buffer for our approach.
Our work somehow can be regarded as a data-free work, so it falls into the regularization-based group oEWC (Schwarz et al., 2018), SI (Zenke et al., 2017), ALASSO (Park et al., 2019), UCB (Ebrahimi et al., 2019).
Although this category of works performs incremental learning without any equipment, they can’t gain outstanding performance, while DFIL achieves remarkable results.
The more comparable group of works is pseudo-memory-rehearsal
DGM (Ostapenko et al., 2019), DGR (Shin et al., 2017), MeRGAN (Wu et al., 2018) which try to synthesis past knowledge.
Thanks to the generative adversarial networks, this group of papers meet a comparable result against memory-rehearsal works without buffering strategy.
However, DF-IL has the same terminology while reducing the complexity by eliminating the need for an auxiliary network along with improving performance.
5. R ESULTS
Table 2 shows the performance of DF-IL in comparison to the other mentioned methods on the two commonly used considered datasets.
Our proposed method DF-IL has achieved the SOTA performance in almost all settings.
PNN (Rusu et al., 2016), one of the dynamic architecture works, produces the most substantial results in the task-IL setting, speciﬁcally 95 .
13 % and 67 .
84% in CIFAR10 and tiny-ImageNet, respectively.
However, it suffers from an exponential memory increasing issue, making it impossible in class-IL and challenging datasets with more classes.
Besides, our
[Table]
-IL setting at the cost of keeping the network architecture ﬁxed with margins 2 .
01 % on CIFAR-10 and 0 .
42 % on Tiny-ImageNet than the top rank.
Compared with regularization-based methods SI (Zenke et al., 2017), oEWC (Schwarz et al., 2018), and ALASSO (Park et al., 2019)
the gap does not seem to be bridged, indicating that the old parameter set’s regularization is not sufﬁcient to mitigate forgetting since the wights’ importance, calculated in an earlier task, might be unreliable in later ones.
Even though DF-IL works without allocating memory buffer, our method can achieve excellent results in both class-IL and task-IL on both datasets.
In particular, our proposed method obtains 75 .
12 % in the class-IL, Task-IL settings on the CIFAR-10, and 733 .
42 % in the class-IL, Task-IL settings on the Tiny-ImageNet, respectively, making the relative improvement over the recent SOTA TPCIL (Tao et al., 2020) by 16 .
03 % in the class-IL settings on the CIFAR-10 and Tiny-ImageNet, and 19 .
01 % in the task-IL in the more challenging Tiny-ImageNet respectively.
We further investigate the impact of buffering samples against synthesizing samples in the ablation study.
Compared to pseudo-rehearsal-based methods, which have recently gained much attention due to the power of simulating past experiences rather than storing them, the proposed method shows a signiﬁcant outperformance in both accuracy and processing time since there is no auxiliary network needed.
In particular, our method achieves a relative improvement of 3 .
61 %, 6 . 1 % over DGM (Ostapenko et al., 2019).
C ONCLUSION
We have proposed a novel strategy for IL to address the memory issue, which is crucial when the number of classes becomes large.
In particular, we perform IL in both class-IL and task-IL settings in a data-free manner.
This strategy is implemented through a memory recovery paradigm with no additional equipment.
It only relies on the single DNN, known as the learner, to retrieve the network’s past knowledge as a transfer set to look back on learned experiences.
Our method has outstanding results on two datasets CIFAR-10 and Tiny-ImageNet, compared with recent prominent works.