3EF: C LASS -I NCREMENTAL L EARNING VIA E FFICIENT E NERGY -B ASED E XPANSION AND F USION
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks suffer from catastrophic forgetting when sequentially learning tasks phase-by-phase, making them inapplicable in dynamically renewal systems.
Class-incremental learning (CIL) aims to enable neural networks to learn different categories at multi-stages.
Recently, dynamic-structure-based methods achieves remarkable performance.
However, these methods train all modules in a coupled manner and do not consider possible conflicts among modules, resulting in increasing training costs and spoilage of eventual predictions.
In this work, we propose a unifying energy-based theory and framework called E fficient E nergy-Based
E xpansion and F usion (3EF) to analyze and achieve the goal of CIL.
We demonstrate the possibility of training independent modules in a decoupled manner while achieving bi-directional compatibility among modules through two additionally allocated prototypes, and then integrating them into a unifying classifier with minimal cost.
Furthermore, 3EF extends the exemplar-set to a more challenging setting, where exemplars are randomly selected and imbalanced, where 3EF maintains its performance when prior methods fail dramatically.
Extensive experiments on three widely used benchmarks: CIFAR-100, ImageNet-100, and ImageNetdemonstrate that 3EF achieves state-of-the-art performance in both the ordinary and challenging CIL settings.
I NTRODUCTION
The ability to learn new knowledge continuously is necessary in this ever-changing world, which is seen as an important symbol of human intelligence.
Applicable AI systems are expected to learn novel concepts in a stream form while preserving knowledge for prior ones.
However, deep-neuralnetwork-based systems, which, though, have achieved tremendous success under the closed-world assumption (Minker, 1982), encounter the notorious problem known as catastrophic forgetting (French, 1999; Golab & ¨Ozsu, 2003) that they abruptly forget the prior learned knowledge when they are directly fine-tuned on new tasks.
To address this challenge, the class-incremental learning (CIL) field aims to design learning paradigms that enable deep neural networks to learn novel categories in multi-stages while maintaining discrimination abilities for prior ones (Rebuffi et al., 2017; Masana et al., 2020b; Delange et al., 2021).
So far, many approaches have been proposed to achieve the goal of CIL.
Generally speaking, typical
CIL methods can be categorized into two groups: regularization-based methods and dynamicstructure-based methods.
Regularization-based methods (Kirkpatrick et al., 2017; Aljundi et al., 2018; Li & Hoiem, 2017; Rebuffi et al., 2017) add constraints ( e.g ., parameter drift penalty) when updating, thus forcing the model to maintain crucial information for old categories.
However, this kind of method typically falls into the stability-plasticity dilemma, without enough capacity to handle all categories simultaneously.
Dynamic-structure-based methods (Yan et al., 2021;
Li et al., 2021) expand new modules at each learning stage to enhance the model’s capacity and learn the task-specific knowledge through the new module, achieving remarkable performance.
Whereas, these methods face several critical challenges.
First, with an increasing number of modules coupled together, the model becomes rather bloated, resulting in increasing training costs.
Second, they directly retain all learned modules without considering conflicts among modules, thus corrupting the joint feature representation and misleading the ultimate predictions.
For example, old modules may mislead final prediction for new classes, since they are directly retained and have limited knowledge about new tasks.
These defects make them unsuitable for long-term incremental tasks.
[Figure]
Coupled training brings prohibitive training costs, so is it possible to train each module independently in a decoupled manner?
That is, all the old modules are not involved in the new training process.
Hence, the training cost is always equivalent to that of tuning a single module and will not increase in incremental stages.
To solve the conflicts among different modules, we propose to achieve bidirectional compatibility , namely backward compatibility and forward compatibility.
Here, backward compatibility is committed to making the discrimination ability of old modules unaffected by new ones.
On the contrary, forward compatibility aims to make old modules sensitively capture input distribution shift and automatically reduce their impact on the ultimate predictions when new categories emerge.
By achieving bi-directional compatibility, given a sample from a specific task, the module responsible for the task will dominate the ultimate predictions.
Motivated by this, we propose a novel yet unifying perspective from the energy-based model (EBM) (LeCun et al., 2006), dubbed as E fficient E nergy-Based E xpansion and F usion (3EF).
Fig.
1 displays our 3EF training framework, which is made of two phases (Model Expansion and
Model Fusion).
At the expansion phase, we create a new module and train it isolatedly without the involvement of the other modules.
Then, we propose an efficient fusion strategy to combine all the modules and form a unifying classifier for all classes with minimal cost.
To achieve bi-directional compatibility, we introduce two additional prototypes: forward prototype p f and backward prototype p b when tuning a single module.
The backward prototype measures the confidence of the current module for old classes.
By setting p b as the prototype for all old classes, it forces the new module to learn a task boundary between current classes and prior ones and effectively reduces the risk of overfitting caused by the limited access to old samples.
The forward prototype aims to measure uncertainty in the open world.
Specifically, it helps to transform the open world problem into a closed world form and introduces an adversarial training process.
Combining the forward prototype and the theory of EBMs, we can model the input distribution synchronously while learning the discrimination ability, which makes the model representation robust and sensitive to the input distribution shift.
We theoretically prove that the overall training phase is based on a unifying energy modeling process.
Besides, each module is inherently an energy-based model measuring the energy for given samples, where energy represents their open world uncertainty.
When evaluating on the test set, since all categories in the test set are known, we additionally apply energy alignment to align the input samples to sub-spaces with lower energies, making the model more suitable for in-distribution discrimination and boosting performance.
Vast experiments on three widely-used benchmarks show that our method achieves state-of-the-art performance.
Besides, prevalent CIL methods all require a well-selected balanced exemplar-set for rehearsal, which might be impractical due to data privacy issues (Delange et al., 2021;
Ji et al., 2014) and computation cost when choosing exemplars.
Our method pushes it to a harder setting.
With only randomly sampled data from prior tasks, 3EF maintains its effectiveness while the performance of other methods declines drastically.
R ELATED W ORK
Incremental learning.
Most recent studies on incremental learning are either task-based or classbased.
The crucial difference between them is whether task-id is known at the evaluation phase (Van de
Ven & Tolias, 2019).
Our work is a class-based method, which is typically called class-incremental learning (CIL).
Prevalent CIL methods can be categorized into two classes: regularization-based, and dynamic-structure-based.
Regularization-based methods impose constraints when learning tasks.
Kirkpatrick et al.
(2017); Aljundi et al.
(2018) penalize the parameter drift.
Li & Hoiem (2017);
Rebuffi et al.
; Wu et al.
; Zhao et al.
(2020) utilize knowledge distillation (Hinton et al., 2015) to constrain the model’s output.
Douillard et al.
(2020) propose a novel spatial knowledge distillation.
Zhou et al.
(2022) propose the concept of forward compatible
(Gheorghioiu et al., 2003; Shen et al., 2020) and squeeze the space for known categories, thereby reserving feature space for future categories.
Dynamic-structure-based methods create new modules to enhance the capacity for learning new tasks.
Yan et al.
; Li et al.
(2021) combine all modules together to form a unifying classifier, but it leads to an increasing training cost.
Douillard et al.
(2021) applies transformer (Dosovitskiy et al., 2020; Touvron et al., 2021) to CIL and dynamically expands task tokens when learning new tasks.
Wang et al.
(2022) proposes to dynamically expand and compress the model based on gradient boosting (Mason et al., 1999), thereby adaptively learning new tasks.
Besides, prevalent CIL approaches usually require a well-selected class-balanced exemplar-set for rehearsal (Rebuffi et al., 2017), which has an evident impact on their performance (Masana et al., 2020a) as we verify experimentally.
(Liu et al., 2021) proposes a novel dynamic memory management strategy that is optimized for the incremental phases.
3EF not only achieves state-of-theart performance but shows strong robustness to the choice of exemplar-set.
Energy-based learning.
EBMs define probability distributions with density proportional to exp( − E ) , where E is the energy function (LeCun et al., 2006).
So far, the theory and implementation of EBMs have been well studied.
Xie et al.
(2016) show that the generative random field model can be derived from the discriminative ConvNet.
Xie et al.
well study the cooperative training of two generative models for image modeling and synthesis.
Additionally, Xie et al.
(2022) study cooperative learning of two generative flow models.
Nijkamp et al.
(2019) propose to treat the non-convergent short-run MCMC as a learned generator model or a flow model and show that the model is capable of generating realistic samples.
Xiao et al.
(2021) propose to learn a VAE to initialize the finitestep MCMC for efficient amortized sampling of the EBM.
Xiao et al.
(2021) propose a symbiotic composition of a VAE and an EBM that can generate high-quality images while achieving fast traversal of the data manifold.
Zhao et al.
(2021) propose a multistage coarse-to-fine expanding and sampling strategy, which starts with learning a coarse-level EBM from images at low resolution and then gradually transits to learn a finer-level EBM from images at higher resolution.
Besides, EBMs have been successfully applied in many fields such as data generation (Zhai et al., 2016;
Zhao et al., 2016; Deng et al., 2020; Du & Mordatch, 2019), out-of-distribution detection (OOD) (Hendrycks & Gimpel, 2016; Bai et al., 2021;
Liu et al., 2020; Lee et al., 2020; Lin et al., 2021), and density estimation (Silverman, 2018; Zhao et al., 2016), etc .
Wang et al.
(2021) model the open world uncertainty as an extra dimension in the classifier, achieving better calibration in OOD datasets.
Grathwohl et al.
(2019) propose to model the joint distribution P ( x , y ) for the classification problem.
Bian et al.
(2022) apply energy-based learning for cooperative games and derive new player valuation methods.
There have been several attempts to apply EBMs to incremental learning.
Li et al.
(2020) propose a novel energy-based classification loss and network structure for continual learning.
Joseph et al.
(2022) build an energy-based latent aligner that recovers the corrupted latent representation.
M ETHOD
In this section, we give a description of 3EF and how we apply energy-based model to CIL to efficiently learn a unifying classifier while achieving bi-directional compatibility.
In Sec. 3.1, we first introduce some basic knowledge of CIL.
In Sec. 3.2, we present the definition of energy and optimization objective at the expansion phase.
Then, to avoid the intractability of normalizing constant, we prove a gradient equivalent objective and explain why it helps to achieve the bidirectional ability.
After that, we propose an efficient yet effective fusion strategy in Sec. 3.3.
Finally, we apply energy alignment to further boost the performance in Sec. 3.4.
Additionally, we extend 3EF to Stable-3EF by allocating multiple forward prototypes and backward prototypes experimentally and theoretically.
Experiments show that it helps to stabilize the training and boost performance.
The detailed illustration and proof is provided in Appendix B.
3. P RELIMINARIES
CIL aims to learn a unifying classifier from a sequence of data divided into several incremental sessions with different class groups.
At the t th incremental session, the model receives the training dataset D t = { ( x t i , y t i ) } n t =1 , where x t i ∈
X t is an input sample and y t i ∈
Y t is the corresponding label, which is not accessible at latter sessions.
Only a small amount of exemplars of previous categories are retained in a size-limited exemplar-set V t ⊆ ∪ t − 1 i =1 D i .
The model is expected to train on D t ∪ V t and be evaluated on the test set of all known categories.
At the following discussions, we focus on the details of the t th incremental session without loss of generality.
Particularly, we denote the label spaces of all known classes and novel classes as Y o = ∪ t − 1 i =1 Y i and Y n =
Y t , respectively.
|Y n | = K and |Y o | =
M , representing the number of new categories and that of old ones.
3. E NERGY - BASESD M ODEL E XPANSION
Let h θ : X −→ ∆ K +1 be the newly created module, where X = ∪ ti =1 X
i
and ∆ K +1 is a K + 1standard simplex.
Therefore, we can further decompose h θ as S ◦ F ◦ Φ , where Φ : X −→ R d is the non-linear feature extractor, F : R d −→ R K +2 is a linear classifier transforming the feature into the K + 2 - dimensional logits, and S denotes the non-linear activation function softmax which constrains the final output on the K + 1 - standard simplex.
h θ ( x )[ k ] represents the k + 1 th element of the final output.
First, given an input-label pair ( x , y ) ∈
X × ( Y o ∪ Y n ) , we define the energy E θ ( x , y ) as
[Equation]
where σ : Y n −→ 1 , 2 , . .
, K is a bijection function mapping a given label to its corresponding class index.
The energy E ( x , y ) measures the uncertainty of predicting x ’s label as y .
Hence, the definition of the energy is compatible with traditional classification definitions, since we typically use h θ ( x )[ σ ( y )] , which is the negative exponent of the energy, to indicate the confidence of predicting x ’s label as y .
Moreover, we use h θ ( x )[0] to represent the overall confidence of x ’s label belonging to Y o and do not expect the new module to distinguish between the old categories.
Hence, E ( x , y ) for any y ∈ Y o is represented as − log ( h θ ( x )[0] /M ) .
The denominator M makes the energy larger and indicates that the current module has a larger uncertainty about old categories due to the limited supervision for old categories from V t .
Since P θ ( y | x ) =
[Equation]
and P θ ( x
[Equation]
=
[Equation]
) , the conditional probability density and marginal probability density can be formulated as
[Equation]
Note that P θ ( x ) =
[Equation]
, then we get that the energy function in P θ ( x ) is formulated as
[Equation]
With energy functions defined above, we give proof of the derivation of our optimization objective when training a new module and demonstrate how it works to achieve bi-directional compatibility.
Instead of simply learning a discriminator P θ ( y | x ) ,
which usually causes overconfident predicts even when receiving samples from unseen distributions, we estimate the joint distribution P θ ( x , y )
[Equation]
The estimation of the joint distribution not only encourages the model to learn to distinguish all known categories but also helps model the input distribution, thus making the model sensitive to the input distribution drift.
Therefore, when unseen categories emerge, it helps those modules alleviate overconfident predictions and reduce their impact on the ultimate predictions.
However, due to the intractability of the normalizing constant (cid:80) x ′ (cid:80) y ′ exp( − E θ ( x ′ , y ′ )) , we optimize the gradient equivalent objective of Eq. 4.
Theorem 3.1 (Marginal Distribution Maximum Likelihood Estimation) .
Defining E
θ ( x ) = − log h θ ( x )[ K + 1] and its corresponding marginal distribution as P ′ θ ( x ) , the optimization of E P real ( x ) [ − log P θ ( x )] is equivalent to that of E P real ( x ) (cid:104) − log (cid:80)
Kk =
h θ ( x )[ k ] (cid:105) +
[Figure]
λ
[Equation]
x ) [ − log h θ ( x )[ K + 1]] when gradient descend is applied, where λ ¯ θ is the ratio of the normalizing constants determined by E ′ θ ( x ) and E θ ( x ) , and ¯ θ means that parameters of θ is frozen ( i.e ., instances sampled from P ′ ¯ θ ( x ) are detached).
Theorem 3.2 (Conditional Distribution Maximum Likelihood Estimation) .
With preliminaries from Thm.
3.1, the optimization of E P real ( x ,y )
[ − log P θ ( y | x )] is equivalent to that of E
P
[Equation]
[ K + 1]] when gradient descend is applied, where µ ¯
[Equation]
=
[Equation]
Due to the space limit, detailed proofs for Thm.
3.1 and Thm.
3.2 are deferred to Appendix A.
Combining Thm. 3.1 and Thm. 3.2, the ultimate optimization objective can be formulated as
[Equation]
which is upper bounded by
[Equation]
We take Eq.
7 as the eventual training objective.
Here, we explain the roles of different components.
E
P real ( x ,y )
[ − log h θ ( x )[ σ ′ ( y )]] prompts the new module to accurately discriminate all categories from current task as well as build explicit decision boundaries between current task and prior ones.
By setting p b as the special prototype for all old categories, we can better exploit the shared structure of all old categories and reduce the risk of over-fitting, which typically results from inadequate training samples on old categories.
Given a sample from prior tasks, the new module perceives this task boundary and reduces the confidence of its own task, so that the old module dominates the ultimate prediction.
Hence, we achieve better backward compatibility for old categories than naively tuning the new module on all categories.
E P real ( x ) [ − log h θ ( x )[ K + 1]] encourages the module to reserve a certain degree of confidence for virtual class p f , thus measuring the out-ofdistribution uncertainty for given samples and mitigating overconfident predictions.
As shown in
Fig.
2, E P ′ ¯ θ ( x ) [ − log h θ ( x )[ K + 1]] introduces an adversarial learning process, where we iteratively generate instances believed to have lower energies from P ′ θ ( x ) and then update the energy manifold to increase the energy of generated samples and decrease that of real samples.
This process effectively enhances the modeling of the known input distribution, making in-distribution samples have low energy and out-of-distribution data have high energy.
Therefore, for a sample from unseen distributions, the module will produce predictions with large uncertainty ( h θ ( x )[ K + 1] ) and low confidence due to the fact that the confidence must be lower than 1 − h θ ( x )
[ K + 1] .
Then, modules created in the future to handle these unknown distributions will dominate the final prediction.
Hence, we achieve the forward compatibility for the future unseen categories.
3. E NERGY - BASED MODEL F USION
[Figure]
After training the new module, we aim to fuse it with the prior ones to form a unifying classifier for all seen categories.
Assuming that we have trained a unifying model h
θ o for all the old tasks and σ o maps the label to the output index of h
θ o , a vanilla approach to combining the h θ o
and h θ is to redefine the energy function as
[Equation]
Then we have
[Equation]
However, this might cause task bias.
Different modules may produce predictions with different entropies, the combined model has a tendency to modules with larger entropies.
As shown in Fig.
Simply combing the modules as Eq.
9 leads to misclassification due to the larger entropy in module2.
Considering that p b measures the confidence for old categories, we redefine E { θ o ,θ } as
[Equation]
Then we have
[Equation] [Equation]
[Equation]
We finetune α, β to minimize the negative log-likelihood on a tiny sub-dataset (exemplar-set), thus mitigating the task bias, namly
[Equation] [Equation]
3. E NERGY - BASED MODEL ALIGNMENT
Equipped with an energy-based nature, 3EF allows for a pleasant by-product: test-time algnment.
During the evaluation phase, we make energy alignment to the input.
Specifically, we apply
SGLD (Welling & Teh, 2011) to decrease energy of given samples.
That is,
[Equation]
(13) where τ is the step size and ω is the random noise sampled from standard normal distribution.
Because all samples are from known classes at test time, aligning given samples to a manifold with lower energy can make the model more suitable for testing in a closed world.
As shown in Fig.
2, through energy alignment, some useful characteristics shared by training samples are transferred into the test sample to reduce its energy, making it more explicit and producing more convincing predictions.
3. SUMMARY OF 3EF
To conclude, 3EF provides a unifying energy-based framework for CIL.
In order to achieve decoupled training and reduce training overhead, we propose a two-stage training approach: expansion and fusion.
At the expansion phase, we utilize two additionally allocated special prototypes to achieve the bi-directional compatibility.
Then, we propose a novel fusion strategy to mitigate the task-bias and form a unifying classifier.
At the evaluation phase, by advantage of the inherently built energy-based model, we additionally apply energy alignment to further boost the performance.
3. 3EF-D ISTILL
Note that even 3EF has kept the training cost nearly the same for all the incremental sessions, which is a great advantage over classical dynamic-architecture-based methods.
However, 3EF still suffer from the increasing number of model parameters, which is a great disadvantage for almost all dynamic-structure-based methods and might violate the memory usage limitation in class-incremental learning, and therefore it is necessary to compress the increasingly expanded modules.
Note that
FOSTER (Wang et al., 2022) proposed the feature compression phase to compress the expanded dual branch model into a single skeleton, and we show that it is also possible to compress the expanded 3EF model into a single skeleton with the same strategy in FOSTER.
Specifically, after each training stage, we apply the balanced knowledge distillation to compress the model, that is
[Equation]
14) where the ⊗ means the tensor product, w is the weighted vector obtained as effective number (Cui et al., 2019).
We will show that even though the distillation strategy in FOSTER is not specially designed for 3EF, 3EF-Distill still achieves competitive performance.
E MPIRICAL S TUDIES 4. E XPERIMENTAL S ETTINGS
Datasets.
We validate our methods on widely used benchmarks of class-incremental learning CIFAR
(Krizhevsky et al., 2009) and ImageNet100/1000 (Deng et al., 2009).
CIFAR-100 : CIFARconsists of 50,000 training images with 500 images per class, and 10,000 test images withimages per class.
ImageNet-1000 : ImageNet-1000 is a large scale dataset composed of about 1. million images for training and 50,000 for validation with 500 images per class.
ImageNet-100 :
ImageNet-100 is composed of 100 classes randomly chosen from the original ImageNet-1000 dataset.
CIAFR-100 Protocol.
For benchmark CIFAR-100, we evaluate two widely recognized protocols:
CIFAR-100 B0 :
All the 100 classes are averagely divided into 5, 10, and 20 groups, respectively, i.e ., we should train all the 100 classes gradually with 20, 10, 5 classes per incremental session.
In addition, models are allowed to save an exemplar-set to store no more than 2000 exemplars throughout all sessions.
CIFAR-100 B50 : We first train half of 100 classes at the base learning stage.
Then, the rest 50 classes are averagely divided into 5, 10, and 25 groups, respectively, i.e ., we should train the rest 50 classes gradually with 10, 5, and 2 classes per incremental session.
Slightly different from the first protocol, models are allowed to store no more than 20 exemplars for each class.
Therefore, after training all the 100 classes, there are also no more than 2,000 exemplars.
ImageNet Protocol .
For benchmarking ImageNet-100, we evaluate the performance on two different incremental tasks.
In the first task, we split the 100 classes averagely into 10 sequential incremental sessions, and up to 2,000 exemplars are allowed to be stored in the exemplar-set.
In the second task, models are first trained on 50 base classes and then sequentially trained on the 10 classes at the following incremental sessions ( i.e ., 5 incremental sessions totally).
The same as the protocol CIFAR100 B50, models are allowed to store no more than 20 exemplars for each class.
For benchmark
ImageNet-1000, we train all the 1000 classes with 100 classes per step (10 steps in total) with an exemplar-set storing no more than 20,000 exemplars.
Compared methods.
We compare 3EF to strong regularization-based methods: iCaRL (Rebuffi et al., 2017), BiC (Wu et al., 2019), WA (Zhao et al., 2020), and PodNet (Douillard et al., 2020).
Beside, we compare to the dynamic-structure-based method RPSNet (Rajasegaran et al., 2019),
DER (Yan et al., 2021), Dytox (Douillard et al., 2021), FOSTER (Wang et al., 2022).
Apart from the above methods, we also compare with RMM (Liu et al., 2021), which adjusts the memory partition strategy for new and old data.
Among the compared methods, Dytox applies stronger neural architecture (Convit (d’Ascoli et al., 2021)) and additional data augmentation; FOSTER (Wang et al., 2022) additionally uses the AutoAugmentation (Cubuk et al., 2019) to enhance the sample efficiency and improve classification accuracy.
RMM (Liu et al., 2021) achieves a better memory management strategy.
With the same training memory, RMM chooses some new samples to train and discards the rest to allow restoring more old exemplars.
Note that all these are orthogonal to the method itself, and therefore we combine the augmentation and memory management strategy with 3EF for fair comparison with these methods, which we regard as 3EF-Distill++.
We show that 3EF-Distill achieves state-of-the-art performance.
4. R ESULTS AND A NALYSIS
Comparison with SOTAs.
CIFAR-100 :
Table 2 summarizes the experimental results on CIFARbenchmark.
Significant performance improvements can be seen under both B50 and B0 protocols.
Particularly, we achieve 1.16, 2.00, and 1.35 increase of average accuracy under the protocol B with 5, 10, and 20 incremental sessions.
As shown in Fig.
4, 3EF consistently surpass other methods at each incremental sessions.
Besides, we achieve more evident performance improvements under
B50 protocols.
Specifically, we improve average accuracy by 3.12, 3.62, 5.45 under protocol B
[Table] [Table]
with 5, 10, 25 incremental sessions. This shows that the effectiveness of 3EF in both short-term and long-term incremental learning tasks.
The detailed performance in Fig.
7 demonstrates that 3EF consistently outperforms other methods after each incremental session.
ImageNet-100/1000 :
Tabel 1 summarizes the experimental results on both ImageNet-100 and ImageNet-1000 benchmarks.
It is observed that both the top-1 and top-5 accuracy of 3EF still surpass other methods in most protocols, indicating the robustness of 3EF.
The detailed performance on ImageNet-100 are shown in Fig.
E.1 We achieves 1.96 and 0.53 increase of top-1 last accuracy under ImageNet-100 B0 and
B50 protocols with incremental step 10.
3EF improves the top-1 and top-5 last accuracy by 0.51 and 0.72 on ImageNet-1000 with 10 incremental sessions, demonstrating that the efficacy of 3EF under large-scale incremental tasks.
Comparison under imbalanced exemplar-set.
Fig.
4 displays the average accuracy changes of different methods after each incremental session on CIFAR-100 B50 with 5 steps.
Fig. 5(c) illustrates the performance when exemplars are randomly sampled from all the available old instances.
Though the exemplar-set is statistically balanced, prior methods encounters a performance drop and the gap between 3EF and prior methods is enlarged.
Fig.
5(b) and Fig.
5(a) illustrate the performance changes under extreme class imbalance and sample from half classes missing, respectively.
Although the performance of prior methods declines dramatically, 3EF maintains its effectiveness under these two imbalanced protocols.
3EF achieves more than 10% performance gain under these challenging settings.
In addition, since some classes have no exemplars stored in the exemplar-set to calculate the class center, iCaRL based on NCM-classifier (Mensink et al., 2013) fails in the base training phase.
4. A BLATION S TUDIES
Ablations of key components in 3EF.
To verify the effectiveness of the components in 3EF, we conduct ablation studies on CIFAR-100 B50 with 5 incremental sessions.
As shown in Table 3, the average and last accuracy gradually increase as we add more components.
It is notable that after the fusion strategy, the unifying model get a large increase.
Besides, by learning energy manifolds with the forward prototype p f and energy alignment, we can further improve the performance.
    
[Figure]
   
 $ FF X U DF \     
[Figure]

   
 [Figure]
  
[Figure]
X U DF \   
[Figure]
[Figure]

[Figure]
[Figure]
 
[Figure]

[Figure]
[Figure]
[Table] ✓
[Table] ✓ ✓
[Table]
✓ ✓
[Table]
✓ ✓ ✓
[Table]
✓ ✓ ✓
[Table]
Sensitive study of hyper-parameters.
There are three hyper-parameters in 3EF when modeling the energy manifold: hidden layer where energy modeling, the trade-off coefficient λ , and the number of forward prototypes F in Stable-3EF.
As shown in Fig.
6(a) we conduct experiments on three different hidden layers for modeling the energy manifold, and the experimental results show 3EF the robustness to the choice of different hidden layer.
We also change the trade-off coefficients λ from { 0 , 1 e − 1 , 1 e − 2 , 1 e − 3 } and the number of forward prototypes F from { 1 , 5 , 20 , 50 } , and the average accuracies on CIFAR-100 B50 with 5 incremental sessions are shown in Fig.
6(b).
We can see that with the increase of F and λ , the accuracy has an upward trend.
 
 

 1XPEHURIFODVVHV      $ FF X U DF \     ()+()+ ()+ (a) Influence of hidden layers     )XWXUH3URWRW\SHV1XPEHU F  H H H 7 U D G H  R II  3 D U D P H W H U     
[Figure] C ONCLUSION .
In this work we presented 3EF for achieving efficient class-incremental learning.
Under this framework, we efficiently train a specific module for the current task while achieving bidirectional compatibility and fuse it with the prior model under minimal effort.
3EF is equipped with a theoretical analysis showing that its training process is inherently the modeling of an energy-based model.
This energy-based nature enables test-time alignment to further improve the performance.
Appendix of 3EF C ONTENTS
A Proof for Thm. 3.1 and Thm. 3.
B Proofs for Stable-3EF
C More Experimental Settings
C.1 Exemplar selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Implementation details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D
More
Discussions
D.1 Contributions of 3EF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Difference and connections with prior methods . .
D.3 Why 3EF is robust to the imbalanced exemplar-set . . . . . . . . . . . . . . . . . .
D.4 Why 3EF is efficient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E More
experimental results
E.1 Detailed performance
on standard CIFAR-100 B
protocols. .
E.
Performance of 3EF-Distill with backbones in various sizes
E.3 Detailed
performance on standard ImageNet-100 protocols
E.
More results on imbalanced protocols
A P ROOF FOR T HM . 3.1 AND T HM . 3.
Here we provide the detailed proofs for Thm. 3.1 and Thm. 3.2.
Theorem 3.1 (Marginal Distribution Maximum Likelihood Estimation) .
Defining E
θ ( x ) = − log h θ ( x )[ K + 1] and its corresponding marginal distribution as P ′ θ ( x ) , the optimization of E P real ( x ) [ − log P θ ( x )] is equivalent to that of E P real ( x ) (cid:104) − log (cid:80)
Kk =
h θ ( x )[ k ] (cid:105) + λ ¯ θ E P
′ ¯ θ ( x ) [ − log h θ ( x )[ K + 1]] when gradient descend is applied, where λ ¯ θ is the ratio of the normalizing constants determined by E ′ θ ( x ) and E θ ( x ) , and ¯ θ means that parameters of θ is frozen ( i.e ., instances sampled from P ′ ¯ θ ( x ) are detached).
Proof.
Since P θ ( x ) =
[Equation]
, we have
[Equation]
Take the gradient of the right part in Eq. 15, and we have
[Equation]
where ¯ θ represents that θ is frozen ( i.e ., gradients are not computed),
Z ′ ¯ θ = (cid:80) x ′ exp( − E ′ θ ( x ′ )) is the normalizing constant for energy E ′
θ ( x ) , and Z θ is the normalizing constant for energy E θ ( x ) .
Hence, the objective E P real ( x ) [ − log P θ ( x )] is equivalent to
E
[Equation]
where λ ¯ θ =
Z ′ ¯
θ
Z ¯ θ .
Theorem 3.2 (Conditional Distribution Maximum Likelihood Estimation) .
With preliminaries from Thm.
3.1, the optimization of E P real ( x ,y )
[ − log P θ ( y | x )] is equivalent to that of E
P real ( x ,y
[Equation]
[Equation]
[Equation]
y
[Equation]
[Equation]
µ
[Equation]
E P [Equation]
x ) [ − log h θ ( x )[ K + 1]] when gradient descend is applied, where µ
[Equation] Proof.
Considering the definition of P θ ( y | x ) in Eq.
2, we have
[Equation]
The gradient of E P
[Equation]
( x ,
[Equation]
[Equation]
[Equation]
(cid:105) with respect to θ is equal to that of
E P real ( x ,y ) ∩ y ∈Y o
[ − log h θ ( x )[0]] .
Therefore, define σ ′ ( y ) =
[Equation] , y ∈ Y n 0 , y ∈ Y o , and the first two components in Eq.
18 can be written in a unifying form, that is
[Equation]
For the last component in Eq.
18, we have
[Equation]
Therefore, the Objective E P real ( x ,y ) [ − log P θ ( y | x )] is also equivalent to
[Equation]
[Equation]
+
[Equation]
[Equation]
21) where µ ¯ θ =
h ¯ θ ( x )[ K +1] (cid:80)
Kk =
 h ¯ θ ( x )[ k ] .
B P ROOFS FOR S TABLE -3EF
As we have claimed in Sec. 3 and Sec. 4, we expand the original 3EF to have multiple backward and forward prototypes (dubbed as Stable-3EF), which stabilizes the training process and improves the performance. Here we give a formal illustration of the expandable form and provide proof of it.
First, at the t th incremental session, instead of creating one backward prototype p b to measure the confidence for all the old tasks, we create a backward prototype for each old task.
This expansion prompts the new module to learn a discriminator for each old task, thereby improving the performance and stability of training especially when there are clear domain shifts among old tasks.
Therefore, there are t − 1 backward prototypes for measuring the confidence of old tasks.
Furthermore, in Stable3EF, we create multiple forward prototypes during the training.
Concretely, for samples generated through the marginal distribution defined by the energy function, we assign the most-likely pseudo labels to them.
We set the number of forward prototypes to a constant number F .
After introducing these multiple prototypes, i.e ., p b s and p f s, we should redefine h θ : X −→ ∆ K + F +( t − 2) .
Given an input-label pair ( x , y ) ∈ ∪ ti =1 X i × ∪ ti =1 Y
i , we define the energy E θ ( x , y ) as
[Equation]
where σ : ∪ ti =1 Y i −→ K + F + ( t − 1) maps new labels to their K class-corresponding prototypes and maps old labels to their t − 1 task-corresponding backward prototypes.
Therefore, similar to Eq.
2, the conditional probability density and marginal probability density for
Stable-3EF can be re-formulated as :
[Equation]
And then we can induce that the energy function in P θ ( x ) is formulated as
[Equation]
Like Eq.
4, Stable-3EF also estimates the joint distribution P θ ( x , y ) , that is
[Equation]
And same to the original 3EF, Stable-3EF also aims to find its gradient equivalent optimization objective to avoid the intractable normalizing constant in the joint distribution defined by the energy
E θ ( x , y ) .
Theorem B.1 (Marginal Distribution Maximum Likelihood Estimation for Stable-3EF) .
Defining
E ′ θ ( x ) = − log (cid:80) K
+ t − 2+ F k = K + t − 1 h θ ( x )[ k ] and its corresponding marginal distribution as P ′ θ ( x ) , the optimization of E P real ( x ) [ − log P θ ( x )] is equivalent to that of E P real ( x ) (cid:104) − log (cid:80) K
+ t − 2 k
[Equation]
[Equation]
λ ¯ θ E P
[Equation]
x )
[Equation]
− log
[Equation] h θ ( x )[ k ] (cid:105) when gradient descend is applied, where λ ¯ θ is the ratio of the normalizing constants determined by E ′ θ ( x ) and E θ ( x ) , and ¯ θ means that parameters of θ is frozen ( i.e ., instances sampled from P ′ ¯ θ ( x ) are detached).
Proof.
Since P θ ( x ) = exp( − E θ ( x ))
(cid:80) x ′ exp( − E θ ( x )) , we have
[Equation]
Take the gradient of the right part in Eq.
[Equation]
and we have
[Equation]
[Equation]
where ¯ θ represents that θ is frozen ( i.e ., gradients are not computed),
Z ′ ¯ θ = (cid:80) x ′ exp( − E ′ θ ( x ′ )) is the normalizing constant for energy E ′
θ ( x ) , and Z θ is the normalizing constant for energy E θ ( x ) .
Hence, the objective E P real ( x ) [ − log P θ ( x )] is equivalent to
[Equation]
where λ ¯ θ = Z ′ ¯
θ
Z ¯ θ .
Theorem B.2 (Conditional Distribution Maximum Likelihood Estimation for Stable-3EF) .
With preliminaries from Thm.
B.1, the optimization of E P real ( x ,y ) [ − log P θ ( y | x )] is equivalent to that of
E
[Equation]
[Equation]
when gradient descend is applied, where µ ¯
[Equation]
Proof.
Considering the definition of P θ ( y | x ) in Eq.
2, we have
[Equation] The gradient of E P
[Equation]
( x ,
[Equation]
[Equation]
(cid:105) with respect to θ is equal to that of
E P real ( x ,y ) ∩ y ∈Y i [ − log h θ ( x )[ σ ( y )]]
( i = 1 , 2 , . . .
, t − 1 ) no matter whatever |Y i | is.
Therefore, the first t components in Eq.
29 can be written in a unifying form [Equation]
that is
[Equation]
where For the last component in Eq.
[Equation]
we have
[Equation]
Therefore
[Equation]
the Objective E P real ( x ,y ) [ − log P θ ( y | x )] is also equivalent to
E P real ( x ,y )
[Equation]
[Equation]
where
[Equation]
Combining Thm.
28 and Thm.
32, we get the final optimization objective
[Equation] Note that h θ ( x )[ i ] ≥ 0 for all i = 1 , 2 , . .
, K + t − 2 + F , and then we get
[Equation]
Hence, Eq.
33 is upper bounded by
[Equation]
Taking Eq.
36 as the optimization objective
[Equation]
we expand the expansion phase in original 3EF into a more stable form with multiple backward and forward prototypes, namely the expansion phase in
Stable-3EF.
Here we further discuss how we achieve the expanded fusion phase in Stable-3EF.
We also assume that we have trained a unifying model h θ o for all the prior tasks and there is a σ o maps labels to output index of h θ o .
Considering that those t − 1 backward prototypes measures the confidence for t − 1 prior tasks, similar to Eq.
10, we define
[Equation]
Then we have
[Equation]
[Equation]
α i and β i are obtained through the minimization of the negative log-likelihood on the exemplar-set.
C M ORE E XPERIMENTAL S ETTINGS C. E XEMPLAR SELECTION .
As we claimed, all the prior methods assume that all the old samples are available when selecting exemplars.
Typically, they apply the exemplar selection strategy proposed in Rebuffi et al.
(2017), where exemplars are carefully selected by greedily minimizing the derivation of the feature center between the selected exemplars and all the old samples.
Besides, they assume all the old categories are equal and store the same number of exemplars for each old class. This violates the truth in many application scenarios, where available exemplars are usually imbalanced and even some instances for old categories become unavailable at the following sessions.
Therefore, the robustness to the imbalance or lack of some categories is important for CIL methods.
Assuming there are k old classes
and we want to store m exemplars for each old class.
We design three protocols for the imbalance of exemplar-set: (1) half-half : in this protocol, half of k classes are allowed to store more than m exemplars and the other half of k classes are only allowed to store less than m exemplars.
Specifically, defining the balance factor γ ∈ [0 , 1] , half of k classes store (1 + γ ) m exemplars while the other store (1 − γ ) m exemplars.
(2) exp : in this protocol, we use a negative exponential sequence of length k as the weight for each category.
Specifically, the i th class has the weight exp( − γi ) and its number of exemplars is defined by
[Equation]
(3) random : we uniformly sample km exemplars from all available old instances.
This protocol is the most similar to the original setting since the expectation of the number of exemplars for each category is m statistically.
C. I MPLEMENTATION DETAILS .
Our method and all baselines are implemented with Pytorch (Paszke et al., 2017).
For ImageNet, we adopt the standard ResNet-18 (He et al., 2016) as our feature extractor and set the batch size as 256.
The learning rate starts from 0.1 and gradually decays to zero with a cosine annealing scheduler (Loshchilov & Hutter, 2016) (170 epochs in total).
For CIFAR-100, we use ResNet-32 (He et al., 2016) as our feature extractor and set the batch size to 128.
The learning rate also starts from 0.1 and gradually decays to zero with a cosine annealing scheduler (170 epochs in total).
For both
ImageNet and CIFAR-100, we use SGD with the momentum of 0.9 and the weight decay of 5e-4 at the expansion phase.
At the fusion phase, we use SGD with a momentum of 0.9 and set the weight decay to 0, and train the fused model on the exemplar-set for 30 epochs.
For data augmentation, we apply the same data augmentation as Rebuffi et al.
For generating samples from P ′ θ ( x ) when learning the energy manifold, we follow Stochastic Gradient Langevin Dynamics (SGLD)
(Welling & Teh, 2011).
However, to facilitate the sampling process, we use representations of real instances as the starting point of SGLD at the hidden layer, which can be seen as a disturbance to the original hidden representation of the real samples.
When applying energy alignment at the evaluation phase, we also use the SGLD at the hidden layer.
For the SGLD in sampling process, we use Adam (Kingma & Ba, 2014) to consider the accumulated gradient and set the learning rate and the derivation to be 1 e − 1 , and 1 e − 3 , respectively.
And the number of updates is set to be 5.
Therefore, we are able to generate new samples rather efficiently when learning the energy manifold.
When applying energy alignment at the evaluation phase, we use a constant step size of 2 and a standard deviation of 1e-3.
The number of updates in each SGLD round is set to be 30.
To further stabilize the training process and improve the performance, we expand the original 3EF into Stable-3EF with multiple backward and forward prototypes.
The number of backward prototypes is set to be the number of old tasks, and the number of forward prototypes is set to a constant number F .
The detailed illustration and proof are provided in Appendix B. Besides, to simplify the training procedure and avoid the intractability of normalizing constants, we use a trade-off coefficient λ to replace λ ¯ θ and µ ¯ θ in our implementation.
By default, we set λ to be 1 e − 1 and F to be 50 .
D M ORE D ISCUSSIONS D. C ONTRIBUTIONS OF 3EF
Our contribution is three-fold: 1) We propose a novel training paradigm for CIL: We efficiently train a specific module for the current task while achieving bi-directional compatibility and then fuse it with the prior model under minimal effort.
We provide a theoretical proof showing that the training cost is inherently the modeling of an energy-based model and apply the energy model to align the test samples and reduce their energy ( i.e ., open world uncertainty) to further boost the performance.
3) We achieve state-of-the-art performance with lower training cost and maintain the performance when only randomly sampled old data is available while other methods fail dramatically.
D. D IFFERENCE AND CONNECTIONS WITH PRIOR METHODS
Here we discuss the crucial difference and connections between our method and prior works.
Though
Wang et al.
(2021) has proposed to use an energy-based extra dimension to model the open world uncertainty in OOD detection, it is not suitable for CIL where old categories should be considered to alleviate forgetting.
Besides, we take the joint distribution P ( x , y )
=
P ( x ) P ( y | x ) into consideration, which forces the model to learn discrimination ( P ( y | x ) ) and be sensitive to the input distribution shift ( P ( x ) ) and thus modules that do not match the input distribution will capture the distribution shift and weaken their influence on the ultimate predictions.
Similar to Joseph et al.
(2022), we also utilize the energy-based model to achieve energy alignment at the evaluation phase.
While our energy-based model is inherently built at the training phase, Joseph et al.
(2022) need to train an extra energy aligner after the training phase, introducing additional training costs.
Apart from that, despite they assume that the task-ids of given samples are known in their implementation, we do not require that.
Zhou et al.
(2022) utilize multiple virtual prototypes to reserve feature space for future unseen categories, thus achieving forward compatibility.
Our method can be seen as a bi-directional compatible method, with one prototype p b measuring the confidence for old categories and the other prototype p f
modeling the out-of-distribution probability.
D. W HY 3EF IS ROBUST TO THE IMBALANCED EXEMPLAR - SET
Here we give an explanation about why 3EF demonstrates strong robustness to the imbalance of exemplar-set.
Previous methods, whether based on dynamic-structure or regularization, rely on a unifying feature representation and classifier.
When the exemplar-set is unbalanced, that is, when the number of samples in some categories is very small or does not exist, the feature representation and classifier of the class are damaged.
Although our method still needs to fuse all modules into a unified classifier, each module is decoupled from each other and is responsible for different tasks.
Thus, when training new tasks, even if samples of some old classes are not available, the old modules responsible for these classes are not affected.
In addition, when training new modules, we classify all old classes into a special backward prototype.
This special backward prototype aims to learn the characteristics shared by categories in old tasks, which is insensitive to the imbalance of old samples.
In the model fusion phase, the confidence of backward prototype is uniformly added to the output of old modules, thus acting as a soft task discriminator without affecting the predictions of old modules among old tasks.
D. W HY 3EF IS EFFICIENT
Here we explain why 3EF is efficient.
Previous methods, whether based on knowledge distillation or dynamic structure, require the forward propagation of old modules when learning new tasks, to achieve a unifying classifier.
Besides, with the increasing number of modules retained, dynamicstructure-based methods suffer from increasing training costs.
As we claimed, the training of 3EF consists of two phases, expansion and fusion.
At the expansion phase, we only need to train the newly created module independently without the involvement of prior modules, so that the training cost at the expansion phase is equivalent to tuning a single module and will not increase at incremental stages.
Though we follow Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011) to generate samples from P ′ θ ( x ) when learning the energy manifold.
However, instead of initializing samples randomly as the starting point, we use representations of real instances as the starting point of SGLD at the hidden layer, which can be seen as a disturbance to the original hidden representation of the real samples and greatly accelerate the generation process.
Besides, we use the Adam optimizer to automatically adjust the step size and consider the cumulative gradient to further improve the generation efficiency.
We run only 5 updates to get the generated samples.
At the fusion phase, we only need to learn two factors including scale factor α and bias factor β through training on the randomly selected exemplar-set for about 30 epochs, whose training cost is minimal compared to the expansion phase.
Particularly, we compare the training cost of 3EF with the methods including the classical dynamicstructure-based method DER Yan et al.
and regularization-based methods like iCaRL (Rebuffi et al., 2017) and WA (Wu et al., 2019).
Without loss of generality, we take the t th incremental stage as an example.
Although all the old modules are frozen in DER, the training process still requires the forward propagation of all t − 1 old modules and the forward propagation of the new module and then applies backpropagation to update the new module.
Apart from that, DER also needs to be the final classifier with a balanced reserved dataset.
Therefore, the training process of DER at t th incremental stage is: t × forward propagation, 1 x backpropagation, 1 × finetune classifier.
Similarly, due to the requirements of knowledge distillation in regularization-based methods like iCaRL, the training cost is: 2 × forward propagation, 1 × backpropagation.
In contrast, in 3EF, we first train the new module in a decoupled manner, which only requires: 1 × forward propagation, 1 × backward propagation.
Then, the fusion phase only needs to tune two parameters with a small subset, whose cost is minimal and is even more efficient than the process of simply tuning the final classifier in DER.
Therefore, the training cost of 3EF consists of: 1 × forward propagation, 1 × backward propagation, 1 × finetune α and β .
Specifically, we compare the training time of BiC (regularization-based), DER (dynamic-structurebased) and 3EF at each incremental session on CIFAR-100 B0 10 steps protocol, the results are shown in Table.
D.4.
We can observe that the average training time of 3EF is lower than both the regularization-based method BiC and the dynamic-structure-based method DER.
[Table]
E M ORE EXPERIMENTAL RESULTS E. D ETAILED PERFORMANCE ON STANDARD CIFAR-100 B50 PROTOCOLS .
We report the detailed accuracy on standard CIFAR-100 B50 protocols with 5, 10, 25 incremental sessions in Fig.

   

[Figure]
  
 

 $ FF X U DF \      [Figure]

   
 1XPEHURIFODVVHV     $ FF X U DF \      5HSOD\L&D5/%L&:$ '(53RG1HW() (b) CIFAR-100 B50 10 steps     
 1XPEHURIFODVVHV     $ FF X U DF \      5HSOD\L&D5/%L&:$ '(53RG1HW() (c) CIFAR-100 B50 25 steps
E. P ERFORMANCE OF 3EF-D ISTILL WITH BACKBONES IN VARIOUS SIZES .
We report the detailed accuracy of 3EF-Distill with different backbones, including ResNet 20, ResNet 32, ResNet 44, ResNet 56, and ResNet 110 in Table E.2.
We can see that the training strategy of 3EF is consistently effective in backbones of various sizes. The accuracy of each incremental session has an upward trend as the model becomes larger.
Accuracy in each session (%)
Backbones Average
ResNet77.
ResNet80.
ResNet81.
ResNet82.
ResNet83.
E. D ETAILED PERFORMANCE ON STANDARD I MAGE N ET -100 PROTOCOLS .
We report the detailed accuracy on standard ImageNet-100 Fig.
E.1.

   
 1XPEHURIFODVVHV   
 

 $ FF X U DF \     
[Figure]
'(53RG1HW()
[Figure]

[Figure]
E. M ORE RESULTS ON IMBALANCED PROTOCOLS .
We report more results on imbalanced protocols in Fig.

     1XPEHURIFODVVHV
    
 
$ FF X U DF \      [Figure]
'(53RG1HW()
[Figure]

     1XPEHURIFODVVHV
    
 
$ FF X U DF \     
[Figure]