Effective Decision Boundary Learning for Class Incremental Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Rehearsal approaches in class incremental learning (CIL) suffer from decisionboundary overfitting to new classes, which is mainly caused by two factors: insufficiency of old classes data for knowledge distillation and imbalanced data learningbetween the learned and new classes because of the limited storage memory.
In thiswork, we present a simple but effective approach to tackle these two factors.
First,we employ re -sampling strategy and M ixup K nowledge D istillation (Re-MKD)to improve performances of KD, which would greatly alleviate the overfittingproblem.
Specifically, we combine mixup and re-sampling strategy to synthesizeadequate data used in KD training that are more consistent with the latent distribution between the learned and new classes.
Second, we propose a novel I ncremental
I nfluence B alance (IIB) method for CIL to tackle the classification on imbalanceddata by extending the influence balance method into the CIL setting, which reweights samples by their influences to create a proper decision boundary.
Withthese two improvements, we present the E ffective D ecision B oundary L earningalgorithm (EDBL) which improves the performance of KD and deals with theimbalanced data learning simultaneously.
Experiments show that the proposed
EDBL achieves state-of-the-art performances on several CIL benchmarks.
Introduction
Applications of deep neural networks (DNNs) in a real world require the ability of the system tolearn new classes incrementally [1, 2, 3].
However, DNNs typically suffer from drastic performancedegradation on the previously learned tasks after learning new classes when the past data areunavailable, which is well documented as catastrophic forgetting [4, 5, 6].
Recently, many methodsin CIL have been proposed to try to deal with this problem.
Owing to their superior performances,
R ehearsal-based approaches utilizing KD [7] ( RKD ) [8, 9, 10, 11, 12] have been widely applied in
CIL.
However, recent works revealed that the RKD methods suffer from decision boundary overfittingto new classes, which is referred to as the (task-recency) bias towards new classes [3, 8, 10, 13].
As shown in Fig.
1, basic RKD approaches typically have two training objectives: transferring theexisting knowledge from the old model by KD and learning new classes via minimizing the KDloss [7] and the cross entropy (CE) classification loss.
However, RKD approaches usually suffer fromthe decision boundary overfitting problem caused by insufficient KD and imbalanced data betweenthe old and new classes.
Problem of Insufficient KD.
In CIL setting, the stored exemplars of old classes are limited becauseof the small memory budget.
It is well known that the high capacity of DNNs is sufficient to memorizethe entire training data [14, 15], so RKD methods suffer from insufficient data for KD training.
RKDmethods commonly use the stored exemplars of old classes and data of new classes to compute the KDloss to preserve the existing knowledge. However, data of new classes are out of distribution (OOD)
[Figure]
with the original training dataset of the old model and Beyer et al.
[16] demonstrated that OOD datawork in KD to some extent while their performance suffers from great degradation, compared withusing the original training data.
The combination of the scantness of exemplars for the old classesand OOD between the learned classes and the new classes (data of old classes and new classes aretypically OOD in the CIL setting) lead to the new model overfitting to new classes.
This phenomenonis also demonstrated in our experiments (c.f.
Sec. 5.5.2)
and we refer to it as insufficient KD.
Problem of Imbalanced Learning.
Due to the limited memory budget for storing exemplars ofthe old classes in RKD approaches, there exists a serious problem of imbalanced data for learning(few samples for the learned/old tasks are available while we have a large number of samples forthe current/new task), which would lead to decision boundary overfitting to the dominant (new)classes [17, 18, 19].
In order to tackle the insufficient KD, we employ MKD and re-sampling strategy.
We over-samplethe exemplars of the old classes and mix the samples from the old classes and the new classes tosynthesize mixed data for KD training.
The interpolated data by an old class and a new class are moreconsistent with the distribution of the original training data of old classes than the data of new classes,which can improve KD in CIL and form a smoother decision boundary (c.f. Fig.
1, Re-MKD).
In order to deal with the imbalanced data learning problem, we attempt to attenuate the influence ofsamples that cause the overfitting.
To this end, we extend the method of influence balance (IB) [19]to CIL setting and propose the incremental influence-balanced (IIB) method for CIL.
Specifically, wefirst derive a metric that measures how much each sample influences the biased decision boundaries.
Then, we decompose the metric into two parts which are referred to as the classification weightingfactor and the KD weighting factor, respectively.
Finally, we design the incremental influencebalanced (IIB) loss function for CIL, which adaptively assigns different weights to samples accordingto their influence on decision boundaries (c.f. Fig.
1, IIB Method).
Based on the previous proposal, we divide our method into two phases after re-sampling Mixup-baseddata augmentation.
Phase 1 is MKD training, which improves the efficacy of KD to form a betterdecision boundary through generating sufficient and more related data with the old classes used in
KD training.
Phase 2 balances training by utilizing the derived IIB loss, attenuating the influenceof the samples that cause overfitting of the decision boundary.
Through the two training phases, welearn a decision boundary with better generalization capability.
In summary, the main contributions of this work are as follows: (i)
We combine MKD with resampling strategy to relieve the insufficient KD problem in RKD methods through synthesizingdata that are more related with the original training data of old classes than data of new classes.
(ii)
We extend the IB method into the CIL setting and propose a novel IIB loss function to tackle theimbalanced learning effectively in CIL.
(iii)
We propose the EDBL algorithm via combining Re-MKDand IIB by deriving the incremental influence factor for mixed data.
Experiments demonstrate thatthe proposed EDBL method achieves state-of-the-art performances on several CIL benchmarks.
Related Work2.1 Bias-Correction Approaches
CIL belongs to continual learning and the main challenge is catastrophic forgetting [4].
Previousapproaches to tackle catastrophic forgetting can be divided into three categories [3]:regularization [20,
21], parameter isolation [22, 23] and replay approaches [24, 25].
RKD belongs to replay approachesand achieves the state-of-the-art performances.
RKD methods [12, 9, 8, 10, 26] commonly train thenew model by preserving the existing knowledge and learning classes incrementally via minimizingthe KD loss and the CE loss.
We introduce the problem formulation and summarize the popularstrategies of RKD training in Appendix A.
RKD methods suffer from the task-recency bias and the ideas in previous works to alleviate thisproblem in CIL are very similar to the approaches in the long-tail learning.
For example, iCaRL [12]employs representation learning strategy, and EEIL [9] utilizes classical image transformation e.g.random cropping, mirroring etc. to make data augmentation and fine-tune the newly trained modelwith a balanced dataset.
BiC [8] trains a bias-correction classification layer with a balanced dataset,which is similar to the approaches of decouple learning and LUCIR [10] belongs to cost-sensitivelearning.
Different from the previous works, our method employs re-sampling and MKD strategy torelieve insufficient KD problem and propose IIB method to deal with the imbalance data learning.
Both improvements are complementary to the previous bias-correction approaches.
2.2 Classes Imbalance
Classes Imbalance is a significant challenge for machine learning [27, 28].
In long-tail learning, thetrained models usually bias towards the dominant classes.
Previous efforts to tackle the overfittingproblem can be roughly divided into three groups: (i) data re-balance including re-sampling [29, 30],data augmentation [31, 32], etc.; (ii) cost-sensitive learning including re-weighting strategy [33, 19],etc. ; (iii) module improvement including representation learning [34, 35], decoupled training [36]and ensemble learning [37, 38], etc..
In this work, we combine mixup with re-sampling to make dataaugmentation and utilize the re-weighting strategy to relieve overfitting.
But, we mainly use mixeddata in KD training to relieve insufficient KD problem in CIL
and we extend the re-weighting methodin [19] (referred to as IB method in this work) into CIL and combine it with the re-sampling&mixupbased data augmentation to tackle imbalanced data learning.
2.3 Mixup and Knowledge Distillation
Mixup is first proposed in [39], which generates an interpolated sample (ˆ x, ˆ y ) by ( x i , y i ) , ( x j , y j )according to Eq. 1:
[Equation]
where x i , x
j are images and y i , y j are labels, respectively, λ is randomly drawn by the Beta function.
In [39], the CE loss ( L ce ) for mixed data is computed as follows:
[Equation]
After that, some other label mixing methods are proposed such as cutmix [40], manifold mixup [41],, Remix [31], etc.. Mixup-based data augmentation can greatly improve the generalization of DNNsin image classification learning while there are some works utilize label mixing methods in variousscenarios such as the long-tail learning [31], continual learning [42, 43].
Recently, some worksvalidate that label mixing methods improve the performance of KD
e.g. [16, 44], which compute the
KD loss with the mixed sample (ˆ x, ˆ y ) as follows:
[Equation]
where T, S are the teacher model and the student model, t is the temperature, m is the number of thelearned classes of T and σ is the softmax .
These works utilize Mixup to improve KD performancesin model compressing setting where the training data are independent-identically-distributed (IID)with the original training dataset of the teacher model.
However, they cannot verify the effectivenessof their methods in the CIL setting, where the training data are imbalanced and new classes are OODwith the old classes.
In this work, we employ re-sampling in Mixup and validate effectiveness ofmixup between old classes (IID dataset) and new classes (OOD dataset) for KD in the CIL setting.
Preliminaries
Influence Function.
The influence function [45, 46] in robust statistics was proposed to find theinfluential instance of a sample to a model.
Recent works have used influence function in DNNs, e.g., [47].
Given a empirical risk R (Θ) = 1 N (cid:80)
Ni =
1 L ( x, Θ) , of which the optimal parameter isΘ ∗ = argmin Θ R (Θ) , where N is the number of training data, Θ = ( θ, W ) are the parameters ofthe network and θ, W are the parameters of the feature extractor and the linear classifier (the lastfull connected layer, FC), respectively.
According to the results in [47], the influence function of thepoint ( x, y ) is given by:
[Equation]
where ∇ Θ is the gradient operator and H L ≜ (cid:80)
Ni =1 ∇ 2Θ L ( x i , Θ) is the Hessian, which is positivedefinite based on the assumption that L is strictly convex in a local convex basin in the vicinity of Θ ∗ .
Influence Balance Method.
Park et al.
[19] firstly apply influence function to a learning scheme,where they design the influence-balanced (IB) loss by utilizing the influence function during trainingin long-tailed classification.
They treat the influential instance of a sample to a model of a point ( x, y )as the parameters of the model change when the distribution of the training data at the point ( x, y ) isslightly modified and referred to it as the IB weighting factor.
IB method uses L 1 norm of Eq.
4 toqualify the influence of the point ( x, y ) and further ignores the inverse Hessian reasonably.
The IBweighting factor is computed as follows:
[Equation]
IB method focuses on the change in the FC layer of DNNs and the IB weighting factor can besimplified by:
[Equation]
where f is the network and h is the hidden feature vector of ( x, y ) .
Finally, the IB loss, which is usedin balancing training to attenuate the influence of samples that cause an overfitted decision boundaryis given by:
[Equation]
where in IB method, L is CE loss function, λ k
= γn − 1 k / (cid:80) Ki =1 n − 1 i is the class-wise re-weightingterm, k is the label of ( x, y ) , n i is the number of samples in the k -th class and γ is the hyper-parameter.
IB method has two training phases: (i) Normal classification training via minimizing the CE loss;(ii) Balancing training, where IB method fine-tunes the model via minimizing L IB by Eq.
The Proposed Method
4.1 Re-sampling MKD
Re-sampling Mixup.
Beyer et al.
[16] demonstrates that KD training with OOD data suffers fromgreat degradation and they validate empirically that the data, which are related or overlapped with theoriginal training data (consistent with the latent distribution of original training data) can performas good as the original training data in KD training.
So the mixed data generated by mixup usingsamples from old classes and new classes, which are usually more related with old classes thanthe data of new classes (basic RKD methods directly use data of new classes in KD training) mayimprove KD performances in CIL.
Given that the data between the old and new classes are seriously imbalanced in RKD methods, weconsider the ratio between the old classes from the past tasks and the new classes of the current task.
We generate three kinds of mixed data in a batch: mixup among old classes, mixup between oldclasses and new classes and mixup among new classes.
Owing to the limited data per old class in
RKD methods, we mixup data more frequently for the first two types.
Specially, we mixup N 2 timesfor the first two types and 1 time mixup for the last type of mixed data, where N is the proportion ofthe data between per tail and head class, and we make sure the number of samples from old classes ina batch isn’t less than a fixed number (32 in this work).
Finally, re-sampling-based Mixup can generate much data through label mixing between old classesand new classes.
These generated data increases the diversity and the number of data used in KD,which improves the performance of KD in CIL.
Our experiments demonstrate that re-sampling-based
Mixup outperforms the original Mixup in [39] (referred to as Vanilla-Mixup in this work) significantly(c.f.
Sec. 5.5.1).
MKD Training.
After data augmentation by re-sampling-Mixup, we train the new model throughminimizing two losses: the CE loss ( L ce ) and the KD loss ( L kd ) with the synthesized data in the similar way as in the basic RKD method.
We compute L ce and L kd according to Eq.
2 and 3,respectively
and we replace the teacher and student model with the old model ( f t − 1 θ,W ) and the newmodel ( f tθ,W ) in Eq. 3, respectively.
In this paper, we denote the strategy of Re-sampling Mixup and
MKD training (Re-sampling MKD) as Re-MKD while we denote the strategy of vanilla Mixup and
MKD training as vanilla-MKD.
4.2 IIB Method
4.2.1 IIB Weighting Factor
We extend the IB method into CIL and propose a novel IIB loss function to attenuate the influence of the samples to relieve the overfitting.
The experience risk at task t in RKD is R (Θ) =1 N (cid:80)
Ni =
1 L ( x, Θ) , where L ≜ L ce ( y i , f t Θ
( x i ))
+ L kd ( f t − 1 Θ ( x i ) , f t Θ ( x i )) and Θ ∗ =
argmin Θ R (Θ)is the optimal parameter after initial training.
We first utilize Eq.
4 to compute the influence of thesample ( x, y ) for CIL.
Then, like in IB method, we ignore the inverse of the Hessian, which is givenby H L ≜ (cid:80) Ni =1 ∇ 2Θ L ( x i , Θ) =
(cid:80) Ni =1 ∇ 2Θ L ce ( x i , Θ) + ∇ 2Θ L kd ( x i , Θ) because it is commonlymultiplied by all the training samples and just the relative influences of the training samples areneeded and we use L 1 norm to qualify the influence to obtain the incremental influence-balanced(IIB) weighting factor.
Therefore, according to Eq. 5 in the IB method, the IIB weighting factor ofthe point ( x, y ) for CIL can be represented by:
[Equation]
At task t , let h =
[ h 1 , . . .
, h L ] T be a hidden feature vector (an input to the FC layer), and f ( x, Θ) =[ f 1 , . . .
, f m , . .
, f m + n ] T be the output denoted by f k ≜ σ ( w Tk h ) , where σ is the softmax function,m, n are the number of learned classes and the new classes, respectively.
The weight matrix of g tW isdenoted by W = [ w 1 , . . . , w m , . . .
, w m + n ] T ∈ R ( m + n ) × f .
Then, ∇ Θ L ce ( x, Θ) and ∇ Θ L kd ( x, Θ)is computed as below, respectively.
[Equation]
[Equation]
Finally, we apply Eq.
9 and 10 to Eq.
Thus, the IIB weighting factor can be computed as:
[Equation]
where [ V ] ji denotes the slice [ v i , . .
. , v j ]( j ≥ i ) of the vector V =
[ v 1 , . . .
, v N ] .
4.2.2 Decomposition of IIB Weighting Factor
The Stability-Plasticity Dilemma.
The stability-plasticity dilemma [48] is a well-known constraintfor artificial and biological neural systems.
The basic idea is that a learning system requires plasticityfor the integration of new knowledge, but also stability in order to prevent the forgetting of previousknowledge.
Too much plasticity will result in previously encoded data being constantly forgotten,whereas too much stability will impede the efficient coding of this data at the level of the synapses.
In RKD, the KD loss is used to preserve the existing knowledge by encouraging the new model tomimic the output of the old model and the CE loss is used to learn to recognize new classes.
Eq.11 considers the influence of the CE loss and the KD loss on the decision boundary, but a samplemay actually perform differently on the classification and KD training.
For example, we use dataof motorbike to train the new model to recognize motorbike incrementally while preserving theknowledge of knowing bike.
Because of the similarity between motorbike and bike, samples ofmotorbike may improve the performance of KD and preserve the previous knowledge well.
Thatis because the data which are related or overlapped with the original training data perform well in
KD [16].
But in the classification training, the new model may bias to motorbike.
Decomposing IIB Factor.
Considering the stability-plasticity dilemma, we decompose the IIBweighting factor into two factors: the classification weighting factor for learning new tasks andthe KD weighting factor for preserving the existing knowledge, which measure how much eachsample influences on forming new boundaries by the classification training and the previous decisionboundaries of the old model preserving training (KD training), respectively.
Definition 4.1.
In CIL, the classification weighting factor and the KD weighting factor of a sample( x, y ) are defined as the magnitude of the gradient vector of the CE loss and the KD loss on that point,respectively:
[Equation]
Then, we use a hyper-parameter α to make trade off on these two factors.
Therefore, IIB weightingfactor is computed via:
[Equation]
where IIB ( x, Θ) ≤ ∥∇ Θ L ce ( x, Θ) ∥ 1 + ∥∇ Θ L kd ( x, Θ) ∥ 1 , and α ≤ 1 .
4.2.3 IIB Loss
During the balancing training, we attempt to fine-tune the decision boundary to create a moregeneralized one, so we multiply the CE loss function by the inverse of IIB weighting factor, resultingin IIB loss as follows:
[Equation]
where λ k is the class-wise re-weighting term (c.f.
Sec.
3 and Eq. 7).
4.3 EDBL Algorithm
We propose the EDBL algorithm to combine Re-MKD and IIB method to improve knowledgetransferring and deal with imbalanced data classification synthetically to generate more generalizeddecision boundaries in CIL.
After MKD training, considering an mixed sample (ˆ x, ˆ y ) , we apply Eq. 1to Eq.
13 to compute the IIB weighting factor as follows:
[Equation]
Then, we use this IIB weighting factor to compute L IIB according to Eq. 14.
Inspired by [16], which validates empirically that a large number of epochs improves the performanceof KD, so we add the KD loss to the IIB loss.
Then the overall loss L overall for balancing training isas follows:
[Equation]
Overall, EDBL has two training phases: MKD training and balancing training.
During the twotraining phases, L kd is continually used for distillation to encourage the decision boundary in thenew model to mimic the one in the old model, and L IIB re-weights the interpolated samples by theirinfluences on a decision boundary in the balancing training phase.
The pseudo-code of EDBL ispresented in Algorithm 1 in Appendix B.
Experiments
5.1 Datasets
We conduct experiments on CIFAR-10, CIFAR-100 [49] and Tiny-Imagenet [50] to validate ourapproach.
We pad four pixels for images in CIFAR-10 and CIFAR-100 and then random crop theminto 32 × 32 pixels.
We use horizontal flip in all image pre-processings [12, 8].
5.2 Baselines and Protocols
We compare EDBL with some SOTA methods in CIL such as Mnemonics [51], PODNet-CNN [52]and SS-IL [26], etc..
We adopt two protocols to conduct experiments: (1) Base-half: we start froma model trained on half of all classes in the dataset, and the remaining half of classes come in 5and 10 phases and stores 20 samples for each old class [10].
(2) Base-0: we follow the protocolin iCaRL [12] to conduct experiments, where all of classes come in 2, 5, or 10 phases and thememory budget is a fixed value.
We report the results of two inference strategies: CNN output andthe nearest-mean-of-exemplar strategy (NME) [12] to evaluate our methods (denoted as EDBL-CNNand EDBL-NME, respectively).
We follow the exemplar management of iCaRL to select exemplars for each old class.
We use twocommon metrics: average accuracy and average incremental accuracy [12] to measure performances.
[Figure]
5.3 Implementation Details and Hyper-parameters Tuning
The λ in Eq.
1 is randomly sampled from the Beta function B = (1 , 1) for all the experiments.
To make comparison fairly, we use the same networks for our methods and baselines.
We follow[12, 8, 10] and [43] to use ResNet32 (d=64) for experiments on CIFAR-10/100 and ResNet18(d=512) for Tiny-Imagenet in this work.
When training networks, we follow the standard practicesfor fine-tuning existing networks.
We use stochastic gradient descent (SGD).
As for the traininghyper-parameters such as batch size, weight decay, momentum, learning rate, epochs etc., we setthese parameters semi-heuristically and tune a few of parameters on CIFAR-100 and Tiny-Imagenetexperiments with 5 incremental learning phases, then we use the same setting of these parametersfor other experiments on CIFAR-10/100 and Tiny-Imagenet, respectively.
We mainly tune λ k in Eq.
14 and α in Eq.
15 by grid searching.
All the hyper-parameters are given in Appendix C.
, we [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
5.4 Results5.4.1 Results on Base-half
Table 1 presents the comparisons of our methods with the baselines on Base-half experiments.
From Table 1, we can find that EDBL-NME outperforms allthe compared methods on all the experiments significantly.
EDBL-CNN also surpasses all the comparedmethods on all the experiments except the experimenton CIFAR-100 with 10 phases.
The comparison results of average accuracy at each incremental learningphase are given in Fig.
Fig.
2 demonstrates that ourmethods (EDBL-CNN and EDBL-NME) surpass allthe baselines at nearly each incremental learning phase.
5.4.2 Results on Base-0
[Table]
We further conducted experiments on CIFAR-10/100following Base-0 protocol to evaluate our method.
Thememory budgets on CIFAR-10 and CIFAR-100 arefixed to 200 and 2000 [12], respectively.
CIFAR-10 issplit into 2 and 5 phases while CIFAR-100 is split into 2,5 and 10 phases.
Our methods employ Mixup strategywhich may help to relieve the overfitting problem while
Remix [31] adapted Mixup to generate more effectiveinterpolated data to tackle the long-tail learning.
Tocompare our method with Remix, we directly employ
Remix to train the new model to learn new classesincrementally with the optimal hyper-parameters given in [31].
We report the results of Remix-CNNand Remix-NME.
From Table 2, we can find that EDBL-CNN and EDBL-NME outperform allbaselines on nearly all experiments except the experiment with 5 phases and 2 phases on CIFAR-10,
[Figure]
[Table]
respectively.
Both of EDBL-CNN and EDBL-NME outperform Remix-CNN and Remix-NME on allthe experiments by large margins about [1.1%, 7%].
We also give the results of average accuracy ateach incremental learning phase and the results show that our methods surpass all the baselines atnearly each incremental learning phase, similarly (c.f. Appendix D).
5.5 Ablation Study
5.5.1 Analysis on Re-MKD and IIB
The base work in [9] is a vanilla basic RKD which trains the new model via minimizing the KDloss and the CE loss without any other strategies.
We use it as the baseline in ablation study.
Thememory size in the ablation study is fixed to 200 and 2000 for CIFAR-10/100, respectively and theexperiments are conducted in the same way as in the Base-0 experiments.
Table 3 demonstrates theeffectiveness of Re-MKD and IIB components in the proposed EDBL algorithm, where IIB-KD is thetraining strategy that uses the IIB loss and the KD loss to fine-tune the new model in the balancingtraining phase.
In summary, we can observe that: (1) The Re-MKD component can improve the performance of thebaseline at each incremental batch by large margins, about [1% , 16%] and Re-MKD outperforms
Vanilla-MKD (vanilla-Mixup-based MKD) significantly.
This validates that the generated data byre-sampling-based Mixup are more related with old classes to improve the performance of KD.
This also demonstrates that the Re-MKD would relieve the problem of insufficient KD; (2) The IIBcomponent can improve the performance of the baseline at each incremental batch remarkably, andsurpasses baseline by up to [5 . 5% , 7 . 5%] on CIFAR-10 and CIFAR-100 at the last phase.
Besides,
IIB-KD further improves the performance based on the improvement by IIB, which exceeds baselineby 19% and 13% on CIFAR-10 and CIFAR-100, respectively at the last phase.
All of these reflect the effectiveness of IIB loss for CIL; (3) EDBL achieves the best performances, which implies EDBLcan learn more generalized decision boundaries effectively.
Further, we conduct visualization experiments with 2 incremental learning phases on a randomly selected subset with 4 classes from CIFAR-10 and the memory is fixed to 200.
Asshown in Fig.
3, the features extracted by the new model trained by EDBL has the less mutual intrusion between the learned classes (the first batch classes) and the new classes (thesecond batch classes), which reflects indirectly that the decision boundaries learned by
EDBL are with more generalizability than the ones generated by the Basic RKD method.
[Table]
5.5.2 Study on Memory Budget
We further study the sensitivity of EDBL on the size ofmemory budget by conducting experiments on CIFAR100 with 5 phases following Base-0 protocol.
The average accuracy on the last phase of the experimentsare demonstrated in Table 4 and the table shows that
EDBL-CNN obtains the best performances.
CLDR [42]utilizes manifold mixup to make a regularization to alleviate catastrophic forgetting while our methodoutperforms CLDR by large margins about [8%,15%].
5.5.3 Sensitivity of α
[Table]
In IIB method, we introduce the hyperparameter α to trade off the classificationweighting factor and the KD weightingfactor to compute IIB loss.
Here, wechoose the strategy, which is RKD + IIBKD in table 3 and α = { 1 , 1 e − 3 , 1 e −5 , 0 } to conduct experiments with 5 incremental phases on CIFAR-10/100 and the memory is fixed to 200, 2000, respectively.
From Table 5, we can observe that IIB method is sensitive to the setting of α and setting a small valuefor α improves the performance significantly while α = 0 is not the optimal value, where α = 0denotes that IIB degenerates into the IB method.
This implies that stability-plasticity dilemma reallyexists in CIL and we should carefully treat this issue to achieve a better performance.
Conclusion
In this paper, we analyzed the causes of the decision boundary overfitting problem in CIL by twofactors: insufficient KD and imbalanced data learning.
We proposed the EDBL algorithm to addressthese problems.
First, in order to deal with insufficient data for KD, we presented the re-sampling
MKD strategy, which generates much data by Mixup and re-sampling to be used in KD training,which are more consistent with the latent distribution of old classes.
Second, we extended IB methodinto CIL through deriving a novel IIB loss, which re-weights samples by their influence on decisionboundary to alleviate the problem of learning with imbalanced data.
On top of this, we propose
EDBL algorithm to combine re-sampling MKD and IIB method which can improve the knowledgetransferring and deal with the imbalanced data classification simultaneously.
Experiments show that
EDBL achieves state-of-the-art performances on several benchmarks and ablation study validatesthat both of Re-MKD and IIB are effective in CIL.
Limitation and Broader Impacts.
One limitation is that more GPU memory and training time areneeded because:(i) re-sampling Mixup generates much mixed data during training, (ii)EDBL hastwo training phases, so EDBL nearly requires twice as much training time as basic RKD method.
Fortunately, both of re-sampling MKD or IIB are effective in CIL, so we can employ them in realapplications flexibly.
Moreover, EDBL significantly relieves catastrophic forgetting, which promotesthe practical use of DNNs.
The negative impacts may occur in some malicious or misuse scenarios.
The appropriate proposes of employing EDBL (re-sampling MKD/IIB) are supposed to be ensuredwith attentions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)?
[Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]