ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection
Abstract
Class-incremental learning (CIL) learns a classiﬁcation model with training dataof different classes arising progressively.
Existing CIL either suffers from seriousaccuracy loss due to catastrophic forgetting, or invades data privacy by revisitingused exemplars.
Inspired by linear learning formulations, we propose an analyticclass-incremental learning (ACIL) with absolute memorization of past knowledgewhile avoiding breaching of data privacy (i.e., without storing historical data).
The absolute memorization is demonstrated in the sense that class-incrementallearning using ACIL given present data would give identical results to that fromits joint-learning counterpart which consumes both present and historical samples.
This equality is theoretically validated.
Data privacy is ensured since no historicaldata are involved during the learning process.
Empirical validations demonstrate
ACIL’s competitive accuracy performance with near-identical results for variousincremental task settings (e.g., 5-50 phases).
This also allows ACIL to outperformthe state-of-the-art methods for large-phase scenarios (e.g., 25 and 50 phases).
Introduction
Class-incremental learning (CIL) [26, 16] trains a network phase-by-phase with training data ineach phase having distinctive classes.
The CIL has received an increasing popularity owing to theneed to adapt learned models to unseen data classes without needing to train from scratch, allowingresource-saving and environmentally-friendly machine learning.
Developing CIL is a natural callin our dynamic world where data and respective target category or task are usually available in aspeciﬁc location or time slot.
In addition, the CIL is intuitively motivated as it resembles real humanlearning processes where a person could continuously adopt knowledge of new object categories ontop of the learned information.
Merits of CIL come with costs.
The CIL could struggle with the notorious catastrophic forgetting[1] rendering the network losing grasp of the learned knowledge when accepting new tasks, whichis also known as the task-recency bias.
To mitigate the forgetting issue, several branches of CILmethods, such as the bias correction-based [2] and replay-based (or exemplar-based) [16] CIL, havebeen proposed.
These CIL techniques are allowed to store a small number of samples from previoustasks to ﬁght the forgetting of old knowledge.
In particular, the replay-based CIL has achieved thestate-of-the-art performance [12].
However, such competitive results have been obtained at the costof revisiting the historical samples, which has brought concerns in terms of data privacy protection.
Data Privacy in CIL.
Data privacy is becoming more of value in our interconnected modern world,which naturally applies to CIL problems asking for exemplar-free learning.
Note that the “privacy” in
CIL (i.e., cannot re-use past exemplars) may be different from the deﬁnition of other ﬁelds (such asdata encryption).
The increasing concern of data privacy contradicts many existing CIL techniques,such as the replay-based CIL.
Several methods from the regularization-based CIL [9] respect privacy as they only impose regularization terms on the loss functions.
However, without re-accessing thetrained samples, their accuracy performance cannot compete with that of the replay-based CIL.
Another exemplar-free CIL branch is the generative adversarial network (GAN)-based learning [21],which preserves privacy by generating historical samples using GANs.
This CIL relies heavily on
GANs’ performance and has not been tested in challenging datasets such as ImageNet [3].
In summary, existing CIL techniques either invade data privacy (e.g., replay-based CIL) or cannotprovide satisfactory accuracy performance (e.g., regularization-based CIL).
In addition, as theforgetting issue persists (though mitigated), CIL’s performance experiences a signiﬁcant degradationas learning phases increase, a pattern shared by many existing CIL techniques (e.g., [4]).
Theperformance degradation escalates in large-phase scenarios—a learning scenario with a large numberof learning phases for increment (e.g., 50 learning phases [24]).
This has motivated us to ﬁnd new
CIL methods that well tackles the catastrophic forgetting without invading data privacy.
In this paper, we propose an analytic class-incremental learning (ACIL) to handle the issue offorgetting and privacy invasion in CIL.
The ACIL is inspired by analytic learning [29, 5], a techniquethat formulates network training into learning of linear stacks.
The analytic learning componentallows the ACIL to conduct CIL in a recursive learning manner that can absolutely memorize theknowledge of every historical sample (i.e., address catastrophic forgetting) while avoiding the breachof data privacy (i.e., without storing any past data).
The key contributions are summarized as follows.
• We introduce the ACIL, which holds absolute memorization of previous knowledge when accepting
new tasks.
• The ACIL does not store past samples, thereby achieving data privacy protection, a rare but valuable
CIL property.
• We provide theoretical validation of ACIL’s absolute memorization, showing that the CIL using
ACIL given present data provides an identical result to that from its joint-learning counterpart thatadopts data from both present and historical phases.• Experiments on benchmark datasets show that the ACIL gives competitive CIL results that donot degrade over increment of data classes during learning phases.
In particular, it outperforms thestate-of-the-arts by a considerable margin for relatively large-phase scenarios (e.g., 25 or 50 phases).
Related Works2.1 Class-Incremental Learning
Bias correction-based CIL mainly tries to address the task-recency bias.
The end-to-end incrementallearning [2] reduces the bias by introducing a balance training stage where only an equal numberof samples for each class is used.
The bias correction (BiC) [23] includes an additional trainablelayer which aims to correct the bias.
The method named LUCIR proposed in [6] ﬁghts the bias bychanging the softmax layer into a cosine normalization one.
Replay-based CIL stores a small subset of data from the previously accessed tasks to reinforcethe network’s memory of old knowledge.
This CIL branch quickly draws attention due to theappealing ability to resist the catastrophic forgetting.
For instance, the PODNet [4] adopts an efﬁcientspatial-based distillation loss to reduce forgetting, with a focus on the large-phase setting, achievingreasonably good results.
The AANets [11] employs a new architecture containing a stable blockand a plastic block to balance the stability and plasticity.
On top of the replay-based CIL, methodsexploring exemplar storing techniques [13] are also fruitful.
For instance, the reinforced memorymanagement (RMM) [12] seeks a dynamic memory management using reinforcement learning.
Byplugging it onto PODNet and AANets, the RMM attains a state-of-the-art performance.
Regularization-based CIL imposes additional constraints on the loss functions to avoid forgetting.
The regularization can be imposed on weights by estimating the parameters’ importance so relevantweights do not drift signiﬁcantly.
The elastic weight consolidation (EWC) [8] captures the priorimportance using an diagonally approximated Fisher information matrix.
The EWC is improved by[10] through ﬁnding a better approximation of the Fisher information matrix.
The regularization canalso be imposed on activations to prevent activation drift, which outperforms its weight-regularizationcounterpart in general.
The learning without forgetting (LwF) [9] prevents activations of the old network from drifting while learning new tasks.
The less-forgetting learning [7] penalizes theactivation difference except the fully-connected layer.
GAN-based CIL replays past samples by generating them using GANs.
The deep generative replay[17] generates synthetic samples using an unconditioned GAN.
It is later improved by memory replay
GAN [22] adopting a label-conditional GAN.
In general, the GAN-based CIL relies heavily on
GAN’s generative performance, and is only tested on relatively small datasets, such as MNIST (i.e.,handwritten digits).
The bias correction-based and replay-based CILs allow storing exemplars, leading to privacy invasion.
The exemplar-free methods (e.g., regularization-based and GAN-based CIL) do not give competitiveresults.
Our ACIL can memorize historical knowledge without re-accessing the data from previoustasks, allowing it to perform CIL with absolute memorization and privacy reservation.
2.2 Analytic Learning
The analytic learning has been developed to circumvent limitations imposed by back-propagation(BP), such as gradient vanishing/exploding, divergence during iteration and long training time (i.e.,need many epochs).
The analytic learning also goes by other names such as pseudoinverse learning[5] due to the use of matrix inverse.
The analytic learning starts with the shallow learning.
Onequick example is the radial basis network [15], which trains the parameters using a least-squares (LS)estimation after conducting a kernel transformation in the ﬁrst layer.
The multilayer analytic learning[18, 20] converts the nonlinear network learning into linear segments that can be solved adopting
LS techniques in a one-epoch training style.
For instance, the dense pseudoinverse autoencoder [19]trains a stacked autoencoder layer-by-layer by concatenating shallow and deep features using LSsolutions.
The analytic learning could experience out-of-memory issue as the weights are learnedinvolving the entire dataset at once.
Such a memory issue can be addressed by the block-wiserecursive Moore-Penrose learning (BRMP) [28] by replacing the joint learning with a recursive one.
This much resembles the replacement of gradient descent with stochastic gradient descent to reducememory usage, but differs in that the BRMP can exactly reproduce its joint-learning result.
The analytic learning and its recursive formulation (e.g., BRMP) brings inspiration to the CIL realm.
The BRMP can stream new samples to update the weight without weakening the impact of previoussamples.
This matches the ACIL’s need for the enhanced memorization of previously trained data.
By bridging the analytic learning and its recursive formulation, our ACIL can be built to absolutelyremember historical samples, without needing to re-access them.
The Proposed Method
This section presents the algorithmic details of ACIL, including a base training agenda and a CILagenda .
Our presentation of ACIL is mainly rooted in CNNs which contain a CNN backbone (featureextractor) followed by a fully-connected network (FCN) layer (classiﬁer) for classiﬁcation problems.
An overview of ACIL is depicted in Figure 1.
3.1 The Base Training Agenda
The base training agenda of ACIL has two stages in a sequential oder, namely a base training via BPand an analytic re-alignment base training (ARaBT), which are illustrated in Figure 1(a) and Figure1(b) respectively.
Base Training via BP .
The ﬁrst stage (see Figure 1(a)) of the base training agenda duplicates theconventional BP training on the base dataset.
That is, the network is trained with a BP-based iterationalgorithm (e.g., SGD with momentum) for multiple epochs with an appropriate learning rate scheduler(e.g., step decay scheduler).
Let W CNN and W FCN represent the weights for the CNN backbone andthe FCN classiﬁer.
After the BP-base base training, given an input X , the output of the network is
[Equation]
where f CNN ( X , W CNN ) indicates the output of the CNN backbone with an X passing through it; f ﬂatis a ﬂattening … f l a tt e n
SoftmaxClassifier
[Figure]
n e x p
[Figure]
a
[Figure]
e n e x p a n s
[Figure]
n
[Figure]
[Figure] f l a tt e n e x p a
n s i o n [Figure]
DataPhase
[Figure]
[Figure]
Analytic Re-alignment Base Training .
The second stage of base training (see Figure 1(b)), the
ARaBT, is the key to the formulation of ACIL.
In this stage, the ARaBT “re-aligns” the network’slearning to match the learning dynamics of an analytic learning.
Prior to our development, some deﬁnitions related to CIL are introduced.
A K -phase CIL indicatesthat a network is trained for K phases where training data of each phase comes with differentclasses.
Let D train k ∼ { X train k , Y train k } and D test k ∼ { X test k , Y test k } be the training and testing datasetsat phase k
( k = 1 , 2 , . . .
, K ).
X k ∈ R N k × w × h × c
(e.g., images with a shape of w × h × c ) and
Y k ∈ R N k × d yk (with phase k including d y k classes) are stacked input and label (one-hot) tensors.
Speciﬁcally, D train∼ { X train, Y train} represents the base training set, which will be utilized to conductthe ARaBT.
The ﬁrst step is to extract the feature matrix (denoted by X (cnn)) by feeding the input tensor X train 0through
the trained CNN backbone, followed by a ﬂattening operation, i.e.,
[Equation]
where X (cnn)∈ R N 0 × d cnn .
Instead of building one FCN layer to map the feature onto the classiﬁcationterminal, we conduct a feature expansion (FE) process by inserting an additional FCN layer whichexpands the feature space into a higher one.
That is, the feature X (cnn)is expanded to X (fe) 0 as follows
[Equation]
where X (fe)∈ R N 0 × d (fe) with d (fe) being the expansion size (with d cnn ≤ d (fe) ).
f act is an activationfunction (we adopt ReLU in this paper), and W fe is the FE matrix expanding the CNN-extractedfeature.
The need for FE process can be justiﬁed by the fact that analytic-learning methods requiremore parameters to achieve their maximum performance.
For the FE matrix, we determine d (fe) witha very simple trick by drawing every element from a normal distribution.
Such a randomizationtechnique has been shown to capture useful information for classiﬁcation problems (e.g., see [5, 29]).
Finally, the expanded feature X (fe) 0 is mapped onto the label matrix Y trainusing a linear regressionprocedure via solvingargmin
[Equation]
where (cid:107)·(cid:107) 2 indicates the l 2 -norm, and γ regularizes the above objective function.
Also, · T is thematrix transpose operator.
The optimal solution to (3) can be found in
[Equation]
where ˆ W (0) FCN indicates the estimated FCN weight of the ﬁnal classiﬁer layer.
3.2 The Class-Incremental Learning Agenda
With the network learning aligned with the analytic learning (see (4)), we may proceed to CIL inan analytic learning fashion.
To this end, assume that we are given D train 0 , . . .
, D train k − 1 , the learningproblem in (3) can be extended toargmin
[Equation] [Equation]
where
[Equation]
Note that the stacked label matrix in (5) has a sparse structure due to the fact that datasets fromdifferent phases are mutually exclusive.
The solution to (5) can be written as
[Equation]
where ˆ W ( k − 1) FCN ∈ R d (fe) × (cid:80) k − 1 i =1 d yi with a column size proportional to k .
Equation (7) gives an LS-based analytical solution for joint learning on D train 0: k − 1 .
The goal of ACIL isto calculate the analytical solution that satisﬁes (5) at phase k based on ˆ W ( k − 1)
FCN given D train k withoutany samples from D train 0: k − 1 .
Speciﬁcally, we aim to obtain ˆ W ( k ) FCN recursively based on ˆ W ( k − 1) FCN anddata X (fe) k , Y train k that are available only at the current learning phase.
However, the updated weightˆ W ( k ) FCN must satisfy the joint learning in (5) given D train 0: k .
Let
[Equation]
be the regularized feature autocorrelation matrix (RFAuM) at learning phase k − 1 .
Then our solutioncan be summarized in the following Theorem.
Theorem 3.1.
The FCN weight recursively obtained by
[Equation]
is identical to that obtained by (7) at phase k .
The RFAuM R k can also be recursively calculated by
[Equation]
Proof.
See the supplementary materials.
As shown in Theorem 3.1, the proposed ACIL constructs a recursive update of the FCN weight matrixwithout any loss of historical information.
One can ﬁrst conduct the base training agenda on the basedataset (e.g., compute ˆ W (0) FCN ), and perform CIL afterwards adopting the recursive formulation toobtain ˆ W ( k ) FCN for k > 0 .
The computational steps of ACIL is summarized in Algorithm 1.
Absolute Memorization .
As observed in Theorem 3.1, the CIL in (9) yields an identical result tothat of the joint learning in (7).
This allows the ACIL to operate with absolute memorization in thesense that the recursive formulation (i.e., the incremental learning) gives the same answer as the one
Algorithm and obtained by its joint analytic learning counterpart.
Such an absolute memorization differentiates ourmethod from the existing CIL techniques that are struggling to ﬁght the forgetting issue.
To the bestof our knowledge, the ACIL is the ﬁrst CIL that achieves absolute memorization.
Data Privacy Protection .
Another beneﬁt of our ACIL lies in privacy protection.
Algorithm 1 showsthat, during the CIL agenda, no historical samples are granted.
Instead, the R k is cached to encryptinformation for historical samples.
However, it is impossible to reverse-engineer the process to obtainthe original samples based on the R k only, avoiding possible breaching of data privacy.
This is anattractive feature as data privacy has attracted increasing concern in the CIL community.
Although methods in regularization-based CIL (e.g., LwF) could also protect data privacy, theaccuracy performance is less ideal.
In comparison, as latter shown in the experiments (e.g., Table1), the proposed ACIL preserves data privacy while achieving very competitive results.
In addition,the RFAuM holds a ﬁxed shape (i.e., a square matrix of R d ce × d ce ) regardless of the sample size.
Thistakes up less storage room than that of the replay-based CIL.
An Analytic-Learning Branch of CIL.
We may categorize the ACIL into a new branch (i.e.,analytic-learning branch) of CIL.
Unlike other branches, the ACIL does not forget any historicalinformation at all.
It also attains privacy protection, a rare but valuable CIL feature.
Even with severalappealing features, the ACIL is naturally not as powerful as the BP-based joint learning.
The ACIL isfacilitated but also constrained by the fact that it freezes the training of CNN weights.
As seen in (6),the feature matrix X (fe) i during the CIL agenda is constructed purely based on a transfer learning w.r.t.the CNN backbone trained on the base dataset.
That is, the ACIL extracts the feature of new taskclasses using a somewhat obsolete feature extractor.
This would lead to certain performance drop.
However, we would argue that the beneﬁts of ACIL greatly outweigh the potential accuracy loss.
Ourargument are well supported by the experiments, displaying very competitive results using ACIL.
Experiments
We evaluate the proposed ACIL on CIFAR-100, ImageNet-Subset and ImageNet-Full datasets whichare benchmark datasets for CIL.
We compare the ACIL with several state-of-the-art CIL techniques,including LwF [9], EWC [8], Semantic Drift Compensation (SDC) [25], BIC [23], iCaRL [16], LUCIR [6], Mnemonics [13], PODNet [4], AANets [11] and RMM [12].
GAN-based CIL is notincluded as it is only tested on less challenging datasets (e.g., MNIST) and its performance reliesheavily on the GAN training. Benchmark methods . The LwF adopts distillation-based loss functions to prevent forgetting. The
EWC uses Fisher information matrix. The SDC studies and compensates the semantic drift of features. The LwF, EWC, SDC and the proposed ACIL belong to the privacy-preserving methods. The othermethods, i.e., BIC, iCaRL, Mnemonics, PODNet, AANets and RMM, are replay-based methods,requiring storage of past exemplars.
4.1 Datasets and Implementation Details
Datasets.
CIFAR-100 contains 100 classes of 32 × 32 color images with each class having 500 and100 images for training and testing respectively.
ImageNet-Full has 1000 classes, and 1.3 million [Table] images for training with 50,000 images for testing.
ImageNet-Subset, in particular, is constructed byselecting 100 speciﬁc classes from ImageNet-Full based on what deﬁned in [4].
Network Architecture.
The architectures for CIL in the experiments are ResNet-32 on CIFAR-100and ResNet-18 on both ImageNet-Full and its subset.
These two architectures are commonly adoptedfor CIL performance comparison.
Our ACIL imposes a slight change (i.e., inserts an expansion FCNlayer), but the CNN backbones are identical to those from the selected ResNet architectures.
Training Details.
For conventional BP training in the base training agenda, we train the networkusing SGD for 160 (90) epochs for ResNet-32 (ResNet-18).
The learning rate starts at 0.1 and it isdivided by 10 at epoch 80 (30) and 120 (60).
We adopt a momentum of 0.9 and weight decay of5 × 10 − 4 ( 1 × 10 − 4 ) with a batch size of 128.
The input data are augmented with random cropping,random horizontal ﬂip and normalizing.
For fair comparison, this base training setting is identicalto that of many CIL methods (e.g., [6, 13]).
For the ARaBT and ACIL’s incremental learning steps,no data augmentation is adopted, and the training ends within only one epoch.
The results for the ACIL are measured by the average of 3 runs on an RTX 2080Ti GPU workstation.
Note that in ACIL,the CNN is only trained during the BP base training phase.
After that, the parameters of the CNNbackbone are ﬁxed during the incremental phases (i.e, phase #1 to #K) including the ARaBT.
CIL Protocol.
We follow the protocol adopted in [4, 11].
The network is ﬁrst trained (i.e., phase #0)on the base dataset containing half of the full classes from the original dataset.
Subsequently, thenetwork gradually learns the remaining classes evenly for K phases (i.e., K -phase CIL), with thedataset in each phase containing disjoint classes from one another.
Most existing methods only reportresults for K = 5 , 10 , 25 .
We include K = 50 as well to validate ACIL’s absolute memorization.
4.2 Evaluation Metric
Two metrics are adopted to evaluate the ACIL.
The overall accuracy performance is evaluated bythe average incremental accuracy (or average accuracy)
¯ A (%): ¯ A = 1 K +1 (cid:80)
Kk =
A k where A kindicates the average test accuracy of the network incrementally trained at phase k by testing it on
D test 0: k .
The ¯ A evaluates the overall performance of CIL algorithms.
A higher ¯ A score is preferredwhen evaluating CIL algorithms.
The other evaluation metric is the forgetting rate F (%) deﬁned in[13]: F =
A ZK − A Z 0 where A Zk denotes the average accuracy at phase k by testing it on D test 0 .
Theforgetting rate reveals the degree to which a CIL method forgets the base classes.
Hence, it is a goodindicator to evaluate CIL’s forgetting issue.
4.3 Result Comparison
We tabulate the average incremental accuracy ¯ A and the forgetting rate F from the comparedmethods in Table 1.
As shown in the upper panel, overall, the “combo” CIL techniques—techniques [Figure] [Figure] I
[Figure] t _ S u b s e t [Figure] I m a g e
N e t - F u ll
[Figure]
that combine more than one CIL methods—give very competitive ¯ A scores.
In particular, the“POD+AANets+RMM” combo, which incorporates PODNet [4], AANets [11] into RMM [12],obtains the most competitive results that can be treated as current the state-of-the-art counterpart.
As shown in the upper panel of Table 1, the ACIL gives a slightly worse average accuracy for 5-phase
CIL in general.
However, ACIL’s performance catches up with those of the state-of-th-arts as Kincreases, and begins to lead for K ≥ 25 (i.e., large-phase CIL scenarios).
For instance, for 5-phase
CIL, the ACIL gives an accuracy of 66.30% on CIFAR-100, which is slightly worse than the resultsfrom several combo techniques such as the “POD+AANets” combo (66.31%) by 0.01% and the“POD+AANets+RMM” combo (68.36%) by 2.06%.
However, for 25-phase CIL, the ACIL (with65.95%) outperforms these combo methods, e.g., outperforming the second best by 1.83% (64.12%from the “POD+AANets+RMM” combo).
Such an overtaking pattern is naturally expected owingto ACIL’s absolute memorization.
That is, the accuracy of ACIL remains unchanged for different
K values, while other CIL methods experience various levels of forgetting issue that intensiﬁes as
K increases.
Note that there could be a very mild ¯
A degradation from ACIL as K increase (e.g.,66.30% → 65.95).
Although theoretically the ACIL should give identical results regardless of K , thepossible mild drop is likely caused by quantization errors since large K indicates more computationrounds hence more quantization operations (e.g., see TABLE VI [28]).
This pattern on CIFAR-100 is quite consistent with those on ImageNet-Subset and ImageNet-Full.
On ImageNet-Full, the ACIL begins to outperform the compared methods for K ≥ 10 , and leads(with 64.63%) the second best result (63.08% from the “LUCIR+Mnemonics” combo) by 1.55% for25-phase learning.
On ImageNet-Subset, the ACIL falls behind the “POD+AANets+RMM” combeven for 25-phase learning, but the gap is very small (74.59% v.s. 75.01%).
The overtaking pattern isfurther detailed in Figure 2.
In addition, we report the 50-phase CIL for the proposed ACIL.
As expected, the average accuracies(i.e., 66 .
13% and 64 .
35% on CIFAR-100, ImageNet-Subset and ImageNet-Full) are veryclose to those trained with K = 5 , 10 or 25 .
This allows the ACIL to further outperform the comparedmethods that are not specializing in large-phase incremental problems.
Although the PODNet alsoaims at large-phase problems, its performance cannot compete with ACIL’s (57.98% v.s. 66.01% on
CIFAR-100, and 62.48% v.s. 74.13% on ImageNet-Subset).
Why ACIL Performs Well.
Conventionally, the analytic learning cannot compete with BP [28].
However, if the feature extractor (e.g., CNN layers) is pre-trained with BP with the classiﬁer headdesigned by analytic learning related techniques, the performance can catch up [14].
Such a scenarioﬁts well in the CIL procedure in this paper, explaining why our ACIL performs well.
The expansion size d (fe) from the FE process has a huge impact on the CIL performance.
As plotted in
Figure 3, the ¯ A on ImageNet-100 and ImageNet-Full increases with larger d (fe) ) .
The performance on
CIFAR-100 starts to decline for d (fe) )
> 10k, likely because the expansion ratio for ResNet-32 caseis unreasonably large (e.g., d (fe) )
/d (cnn) = 15k / 64 ) compared with that of ResNet-18 (15k / 512 ) on
ImageNet datasets.
As observed in Figure 3, the ACIL on ImageNet datasets should work better with even larger d (fe) ) , but our 11GB GPU experiences memory leak.
Still, the expansion up to d (fe) )
=
15kallows the ACIL to give very competitive results (see Table 1).
The forgetting rate F is also presented in the bottom panel of Table 1.
Our ACIL demonstratesthe lowest F scores on all the three benchmark datasets.
This is a further evidence supporting ourabsolute-memorization claim.
Note that the absolute memorization does not lead to F = 0 .
Evena healthy joint learning would reduce the performance on the base classes.
This can be explainedby the example as follows.
Let M 50 and M 100 be the two networks jointly trained on the base 50classes
and the 100 full classes from CIFAR-100 respectively.
Testing M 100 on the base datasetwould still experience performance loss compared with that obtained by testing M 50 on the basedataset, i.e., F > 0 .
Hence, although our ACIL perfectly remembers the pass samples, non-zeroforgetting rate still applies when incrementally learning new classes.
Nonetheless, the forgetting ratehas been shown to be much lower than the existing CIL methods.
For instance, for 5-phase learningon ImageNet-Full, the “LUCIR+Mnemonics” combo exhibits 13.63% forgetting, but our ACIL onlyhas 2.75%!
Yet, the low F score does not suggest strong resistance for learning new tasks since theaverage accuracy has been shown to be comparable to state-of-the-art results (see upper panel of
Table 1).
That is, the ACIL bears a relatively good stability-plasticity balance.
Data Privacy Protection.
Apart from the competitive incremental accuracy, the ACIL is in strongsupport of data privacy.
As indicated in Algorithm 1, the incremental learning surrenders any samplesfrom previous tasks, allowing data privacy across learning phases or platforms.
As shown in Table 1,privacy-preserving CIL (e.g., LwF, EWC and SDC) suffers much more intensively (e.g., 45.51% from LwF of 25-phase CIL on CIFAR-100) than the replay-based CIL (e.g., 64.12% from RMM combo of25-phase CIL on CIFAR-100).
Our ACIL maintains the privacy while providing comparable or betteraccuracy performance (e.g., 65.95% of 25-phase CIL on CIFAR-100).
Such a comfortable balancewould certainly attract attention as we are living in a world that values and protects data privacy.
[Figure] [Figure]
[Table]
[Figure]
Memory for Storage.
The ACIL stores R k instead of exemplars.
As an example, for ﬁx-exemplarsetting, the memory used by storing R k (8 k ) on CIFAR-100/CUB200-2011/ImageNet is 8 k × 8 k
=
64million (M) tensor elements, while other methods consume 6.1M/301.1M/3010.6M respectively (e.g.,on ImageNet 224 × 224 × 3 × 20 × 1000 ≈ 3010 .
6 M).
This shows that our method is memory-friendlyto large-shaped image datasets (e.g., ImageNet).
4.4 Ablation Study
In the proposed ACIL, the FE process governed by d (fe) and the regularization controlled by γ areessential.
To show this, we adopt an ablation study by conducting a 5 -phase CIL of ResNet-32 (withd (fe) = 8 k) on CIFAR-100 to observe the performance shift w.r.t.
these modules.
As reported in
Table 2, without the FE process (see the ﬁrst two rows in Table 2), the average incremental accuracyexperiences a sharp drop (e.g., 66.30% → 52.99%).
The need for the FE process was enlightened bythe fact that the analytic learning is naturally prone to under-ﬁtting due to simple linear regression[29].
Widening the feature size helps to capture the missing discriminative information.
The regularization factor γ , on the other hand, plays an important role but behaves robustly during the
CIL experiments.
As shown rows 2-5 in Table 2, the ACIL performs robustly for a considerably widerange of γ values (e.g., 10 − 3 - 10 − 1 ).
However, it would be unwise to remove the regularization as the
ACIL could suffer from a strong accuracy reduction without its support (e.g., 66.30% → 51.12%). [Table]
4.5 Exploring Limitations of ACIL
Our ACIL freezes the CNN backbone trained on the base dataset, so if the incoming data classes arevery different from the base ones, there might be performance degradation.
As a demonstration, a5-phase mix-data experiment is conducted in Table 3 with the base training containing 50 classesfrom CIFAR-100, and CIL with 50 classes randomly selected from Places365 [27]. There is indeedperformance drop (66.37% → 57.43%), though not catastrophically. This is likely caused by ﬁxingthe CNN network, and is a limitation of ACIL. It will be our future work to mitigate such a limitation.
Positive and Negative Societal Impact
A main characteristic of ACIL is privacy-preserving.
It allows researchers to avoid breaching userprivacy while pushing forward their learning methods.
From this angle, our method gives a positivesocietal impact by protecting privacy.
We do not foresee obvious negative societal impacts.
However,our ACIL does rely certainly on the CNN feature extractor in a similar way of transfer learning.
Inthis case, there might be various follow-up research aiming to improve the extraction power by manytrials and errors, leading to certain electricity pressure consumed by GPU operations.
Conclusion
In this paper, we have presented an analytic class-incremental learning (ACIL) which bears twovaluable features (i.e., the absolute memorization and the data privacy protection) for addressingseveral existing limitations of class-incremental learning.
The analytic learning has been incorporatedas a key component to conduct incremental learning of new tasks in a recursive manner.
Such arecursive learning style allows the ACIL to have absolute memorization.
That is, the incrementallearning of ACIL given present data would produce identical results to that of a joint learningwhich accesses both present and historical data, a property that has been theoretically validated.
The recursive formulation has also the merit of not storing any samples from historical tasks, thusavoiding the breach of data privacy.
Experiments have been conducted to validate our claims.
Overall,our ACIL gives very competitive accuracy results.
In particular, it outperforms the state-of-the-artmethods for large-phase scenarios (e g., incremental learning with 50 phases).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See hyperparameter analysis in Section 5.3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] See the Supplementary material.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[No] Will
release the code and instructions shortly.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes] Results are reported by the average of 3 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use ResNet and
have cited the paper.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All datasets are publicly accessible.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] All datasets are publicly accessible with no
personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]