A Theoretical Study on Solving Continual Learning
Abstract
Continual learning (CL) learns a sequence of tasks incrementally.
There are two popular CL settings, class incremental learning (CIL) and task incremental learning (TIL).
A major challenge of CL is catastrophic forgetting (CF).
While several techniques are available to effectively overcome CF for TIL, CIL remains to be challenging due to the additional difficulty of inter-task class separation .
So far little theoretical work has been done to provide a principled guidance and necessary and sufficient conditions for solving the CIL problem.
This paper performs such a study.
It first probabilistically decomposes the CIL problem into two subproblems: within-task prediction (WP) and task-id prediction (TP).
It further proves that TP is correlated with out-of-distribution (OOD) detection.
The key result is that regardless of whether WP and TP or OOD detection are defined explicitly or implicitly by a CIL algorithm, good WP and good TP or OOD detection are necessary and sufficient for good CIL performances.
Additionally, TIL is simply
WP.
Based on the theoretical result, new CIL methods are also designed, which outperform strong baselines in both CIL and TIL settings by a large margin.
Introduction
Continual learning aims to incrementally learn a sequence of tasks [1].
Each task consists of a set of classes to be learned together.
A major challenge of CL is catastrophic forgetting (CF).
Although a large number of CL techniques have been proposed, they are mainly empirical.
Limited theoretical research has done on how to solve CL.
This paper performs such a theoretical study about the necessary and sufficient conditions for effective CL.
There are two main CL settings that have been extensively studied: class incremental learning (CIL) and task incremental learning (TIL) [2].
In
CIL, the learning process builds a single classifier for all tasks/classes learned so far.
In testing, a test instance from any class may be presented for the model to classify.
No prior task information (e.g., task-id) of the test instance is provided.
Formally, CIL is defined as follows.
Class incremental learning (CIL).
CIL learns a sequence of tasks, 1 , 2 , ..., T .
Each task k has a training dataset D k
= { ( x ik , y ik ) n k i =1 } , where n k is the number of data samples in task k , and x ik ∈ X is an input sample and y ik ∈ Y k
(the set of all classes of task k ) is its class label.
All Y k ’s are disjoint
( Y k ∩ Y k ′ =
∅ , ∀ k ̸ = k ′ ) and (cid:83) Tk =1 Y k
=
Y .
The goal of CIL is to construct a single predictive function or classifier f : X → Y that can identify the class label y of each given test instance x .
In the TIL setup, each task is a separate classification problem.
For example, one task could be to classify different breeds of dogs and another task could be to classify different types of animals (the tasks may not be disjoint).
One model is built for each task in a shared network.
In testing, the task-id of each test instance is provided and the system uses only the specific model for the task (dog or animal classification) to classify the test instance.
Formally, TIL is defined as follows.
Task incremental learning (TIL).
TIL learns a sequence of tasks, 1 , 2 , ..., T .
Each task k has a training dataset D k
= { (( x ik , k ) , y ik ) n k i =1 } , where n k is the number of data samples in task k ∈ T = { 1 , 2 , ..., T } , and x ik ∈ X is an input sample and
y ik ∈ Y k
⊂ Y is its class label.
The goal of TIL is to construct a predictor f : X × T → Y to identify the class label y ∈ Y k for ( x, k ) (the given test instance x from task k ).
Several techniques are available to effectively overcome CF for TIL (with almost no CF)
However, CIL remains to be highly challenging due to the additional problem of Inter-task Class
Separation (ICS) (establishing decision boundaries between the classes of the new task and the classes of the previous tasks) because the previous task data are not accessible.
Before discussing the proposed work, we recall the closed-world assumption made by traditional machine learning, i.e., the classes seen in testing must have been seen in training [1, 5].
However, in many applications, there are unknowns in testing, which is called the open world setting [1, 5].
In open world learning, the training (or known) classes are called in-distribution (IND) classes.
A classifier built for the open world can (1) classify test instances of training/IND classes to their respective classes, which is called IND prediction , and (2) detect test instances that do not belong to any of the IND/known classes but some unknown or out-of-distribution (OOD) classes, which is called OOD detection .
Many OOD detection algorithms can perform both IND prediction and OOD detection [6, 7, 8, 9].
The commonality of OOD detection and CL is that they both need to consider future unknowns.
This paper conducts a theoretical study of CIL, which is applicable to any CIL classification model.
Instead of focusing on the traditional PAC generalization bound [10, 11], we focus on how to solve the CIL problem.
We first decompose the CIL problem into two sub-problems in a probabilistic framework: Within-task Prediction (WP) and Task-id Prediction (TP).
WP means that the prediction for a test instance is only done within the classes of the task to which the test instance belongs, which is basically the TIL problem.
TP predicts the task-id.
TP is needed because in CIL, task-id is not provided in testing.
This paper then proves based on the popular cross-entropy loss that (1) the CIL performance is bounded by WP and TP performances, and (2) TP and task OOD detection performance bound each other (which connects CL and OOD detection).
The key result is that regardless of whether WP and TP or OOD detection are defined explicitly or implicitly by a CIL algorithm, good WP and good TP or OOD detection are necessary and sufficient conditions for good
CIL performances.
This result is applicable to both batch/offline and online CIL and to CIL problems with blurry task boundaries.
The intuition is also quite simple because if a CIL model is perfect at detecting OOD samples for each task (which solves the ICS problem), then CIL is reduced to WP.
This theoretical result provides a principled guidance for solving the CIL problem, i.e., to help design better CIL algorithms that can achieve strong WP and TP performances.
Since WP is basically IND prediction for each task and most OOD techniques perform both IND prediction and OOD detection, to achieve good CIL accuracy, a strong OOD performance for each task is necessary.
Based on the theoretical guidance, several new CIL methods are designed, including techniques based on the integration of a TIL method and an OOD detection method for CIL, which outperform strong baselines in both the CIL and TIL settings by a large margin.
This combination is particularly attractive because TIL has achieved no forgetting, and we only need a strong OOD technique that can perform both IND prediction and OOD detection to learn each task to achieve strong CIL results.
Related Work
Although numerous CL techniques have been proposed, little study has been done to provide a theoretical guidance on how to solve the problem.
Existing approaches mainly belong to several categories.
Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].
Memorizing some old examples and using them to jointly train the new task is another popular approach (called replay ) [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].
Some systems learn to generate pseudo training data of old tasks and use them to jointly train the new task, called pseudo-replay [41, 42, 43, 44, 45, 46, 21, 47, 48].
Orthogonal projection learns each task in an orthogonal space to other tasks [49, 50, 51].
Our theoretical study is applicable to any continually trained classification models.
Parameter isolation is yet another popular approach, which makes different subsets (which may overlap) of the network parameters dedicated to different tasks using masks [3, 52, 53, 4, 54].
This approach is particularly suited for TIL.
Several methods have almost completely overcome forgetting.
HAT [3] and CAT [52] protect previous tasks by masking the important parameters to those tasks.
PackNet [53], CPG [54] and SupSup [4] find an isolated sub-network for each task.
HyperNet [55] initializes task-specific parameters conditioned on task-id.
ADP [56] decomposes parameters into shared and adaptive parts to construct an order robust TIL system.
CCLL [57] uses task-adaptive calibration in convolution layers.
Our methods designed based on the proposed theory make use of two parameter isolation-based TIL methods and two OOD detection methods.
A strong OOD detection method CSI in [6] helps produce very strong CIL results.
CSI is based on data augmentation [58] and contrastive learning [59].
Excellent surveys of OOD detection include [60, 61].
Some methods have used a TIL method for CIL with an additional task-id prediction technique.
iTAML [62] requires each test batch to be from a single task.
This is not practical as test samples usually come one by one.
CCG [63] builds a separate network to predict the task-id.
Expert Gate [64] constructs a separate autoencoder for each task.
HyperNet [55] and PR-Ent [65] use entropy to predict the task id.
Since none of these papers is a theoretical study, they did not know that strong
OOD detection is the key.
Our methods based on OOD detection perform dramatically better.
Several theoretical studies have been made on lifelong/continual learning.
However, they focus on traditional generalization bound.
[10] proposes a PAC-Bayesian framework to provide a learning bound on expected error in future tasks by the average loss on the observed tasks.
The work in [66] studies the generalization error by task similarity and [11] studies the dependence of generalization error on sample size or number of tasks including forward and backward transfer.
[67] shows that orthogonal gradient descent gives a tighter generalization bound than SGD.
Our work is very different as we focus on how to solve the CIL problem, which is orthogonal to the existing theoretical analysis.
CIL by Within-Task Prediction and Task-ID Prediction
This section presents our theory.
It first shows that the CIL performance improves if the within-task prediction (WP) performance and/or the task-id prediction (TP) performance improve, and then shows that TP and OOD detection bound each other, which indicates that CIL performance is controlled by WP and OOD detection.
This connects CL and OOD detection.
Finally, we study the necessary conditions for a good CIL model, which includes a good WP, and a good TP (or OOD detection).
3.1 CIL Problem Decomposition
This sub-section first presents the assumptions made by CIL based on its definition and then proposes a decomposition of the CIL problem into two sub-problems.
A CL system learns a sequence of tasks { ( X k , Y k ) } k
=1 ,...,T , where X k is the domain of task k and Y k are classes of task k
as Y k
=
{ Y k,j } , where j indicates the j th class in task k .
Let X k,j to be the domain of j th class of task k , where
X k = (cid:83) j X k,j .
For accuracy, we will use x ∈ X k,j instead of Y k,j in probabilistic analysis.
Based on the definition of class incremental learning (CIL) (Sec. 1), the following assumptions are implied,
Assumption 1.
The domains of classes of the same task are disjoint, i.e., X k,j ∩ X k,j ′ = ∅ , ∀ j ̸ = j ′ .
Assumption 2.
The domains of tasks are disjoint, i.e., X k ∩ X k ′ = ∅ , ∀ k ̸ = k ′ .
For any ground event D , the goal of a CIL problem is to learn P ( x ∈ X k,j | D ) .
This can be decomposed into two probabilities, within-task IND prediction (WP) probability and task-id prediction (TP) probability.
WP probability is P ( x ∈ X k,j | x ∈ X k , D ) and TP probability is P ( x ∈ X k | D ) .
We can rewrite the CIL problem using WP and TP based on the two assumptions,
[Equation]
where k 0 means a particular task and j 0 a particular class in the task.
Some remarks are in order about Eq.
2 and our subsequent analysis to set the stage.
Remark 1 .
Eq.
2 shows that if we can improve either the WP or TP performance, or both, we can improve the CIL performance.
Remark 2 .
It is important to note that our theory is not concerned with the learning algorithm or the training process, but we will propose some concrete learning algorithms based on the theoretical result in the experiment section.
Remark 3 .
Note that the CIL definition and the subsequent analysis are applicable to tasks with any number of classes (including only one class per task) and to online CIL where the training data for each task or class comes gradually in a data stream and may also cross task boundaries (blurry tasks [68]) because our analysis is based on an already-built CIL model after training.
Regarding blurry task boundaries, suppose dataset 1 has classes {dog, cat, tiger} and dataset 2 has classes {dog, computer, car}.
We can define task 1 as {dog, cat, tiger} and task 2 as {computer, car}.
The shared class dog in dataset 2 is just additional training data of dog appeared after task 1.
Remark 4 .
Furthermore, CIL =
WP * TP in Eq.
2 means that when we have WP and TP (defined either explicitly or implicitly by implementation), we can find a corresponding CIL model defined by
WP * TP.
Similarly, when we have a CIL model, we can find the corresponding underlying WP and
TP defined by their probabilistic definitions.
In the following sub-sections, we develop this further concretely to derive the sufficient and necessary conditions for solving the CIL problem in the context of cross-entropy loss as it is used in almost all supervised CL systems.
3.2 CIL Improves as WP and/or TP Improve
As stated in Remark 2 above, the study here is based on a trained CIL model and not concerned with the algorithm used in training the model.
We use cross-entropy as the performance measure of a trained model as it is the most popular loss function used in supervised CL.
For experimental evaluation, we use accuracy following CL papers.
Denote the cross-entropy of two probability distributions p and q as
[Equation]
For any x ∈ X , let y to be the CIL ground truth label of x , where y k 0 ,j
=
if x ∈ X k 0 ,j
otherwise y k,j = 0 , ∀ ( k, j ) ̸
= ( k 0 , j 0 ) .
Let ˜ y be the WP ground truth label of x , where ˜ y k 0 ,j 0 = 1 if x ∈ X k 0 ,jotherwise ˜ y k 0 ,j = 0 ,
∀ j ̸
= j 0 .
Let ¯ y be the TP ground truth label of x , where ¯ y k
= 1 if x ∈ X kotherwise ¯ y k
= 0 , ∀ k ̸ = k 0 .
Denote
[Equation]
where H WP , H CIL and H TP are the cross-entropy values of WP, CIL and TP, respectively.
We now present our first theorem.
The theorem connects CIL to WP and TP, and suggests that by having a good WP or TP, the CIL performance improves as the upper bound for the CIL loss decreases.
Theorem 1.
If H TP ( x ) ≤ δ and H WP ( x ) ≤ ϵ , we have H CIL ( x ) ≤ ϵ + δ.
The detailed proof is given in Appendix A.1.
This theorem holds regardless of whether WP and
TP are trained together or separately.
When they are trained separately, if WP is fixed and we let ϵ
= H WP ( x ) , H CIL ( x ) ≤ H WP ( x ) + δ , which means if TP is better, CIL is better.
Similarly, if
TP is fixed, we have H CIL ( x ) ≤ ϵ + H TP ( x ) .
When they are trained concurrently, there exists a functional relationship between ϵ and δ depending on implementation.
But no matter what it is, when ϵ + δ decreases, CIL gets better.
Theorem 1 holds for any x ∈ X
=
(cid:83) k X k that satisfies H TP ( x ) ≤ δ or H WP ( x ) ≤ ϵ .
To measure the overall performance under expectation, we present the following corollary.
Corollary 1.
Let U ( X ) represents the uniform distribution on X .
i)
If E x ∼ U
( X )
[ H TP ( x )] ≤ δ , then E x ∼ U ( X ) [ H CIL ( x )] ≤ E x ∼ U ( X )
[ H WP ( x )]
+ δ .
Similarly, ii) E x ∼ U
( X )
[ H WP ( x )] ≤ ϵ , then E x ∼ U ( X )
[ H CIL ( x )] ≤ ϵ + E x ∼ U ( X ) [ H TP ( x )] .
The proof is given in Appendix A.2.
The corollary is a direct extension of Theorem 1 in expectation.
The implication is that given TP performance, CIL is positively related to WP.
The better the WP is, the better the CIL is as the upper bound of the CIL loss decreases.
Similarly, given WP performance, a better TP performance results in a better CIL performance.
Due to the positive relation, we can improve CIL by improving either WP or TP using their respective methods developed in each area.
3.3 Task Prediction (TP) to OOD Detection
Building on Eq. 2, we have studied the relationship of CIL, WP and TP in Theorem 1.
We now connect TP and OOD detection.
They are shown to be dominated by each other to a constant factor.
We again use cross-entropy H to measure the performance of TP and OOD detection of a trained network as in Sec.
3.2 To build the connection between H TP ( x ) and OOD detection of each task, we first define the notations of OOD detection.
We use P ′ k
( x ∈ X k | D ) to represent the probability distribution predicted by the k th task’s OOD detector.
Notice that the task prediction (TP) probability distribution P ( x ∈ X k | D ) is a categorical distribution over T tasks, while the OOD detection probability distribution P ′ k ( x ∈ X k | D ) is a Bernoulli distribution.
For any x ∈ X , define
[Equation]
In CIL, the OOD detection probability for a task can be defined using the output values corresponding to the classes of the task.
Some examples of the function is a sigmoid of maximum logit value or a maximum softmax probability after re-scaling to 0 to 1.
It is also possible to define the OOD detector directly as a function of tasks instead of a function of the output values of all classes of tasks, i.e.
Mahalanobis distance.
The following theorem shows that TP and OOD detection bound each other.
Theorem 2.
i)
If H TP ( x ) ≤ δ , let P ′ k
( x ∈ X k | D )
=
P ( x ∈ X k | D ) , then H OOD,k ( x ) ≤ δ, ∀ k = 1 , . .
, T . ii)
If H OOD,k ( x ) ≤ δ k , k = 1 , . . .
, T , let P ( x ∈ X k | D ) =
P ′ k ( x ∈ X k | D ) (cid:80) k P ′ k
( x ∈ X k | D ) , then
H TP ( x ) ≤ ( (cid:80) k 1 x ∈ X k e δ k )( (cid:80) k 1 − e − δ k ) , where 1 x ∈ X k is an indicator function.
See Appendix A.3 for the proof.
As we use cross-entropy, the lower the bound, the better the performance is.
The first statement (i) says that the OOD detection performance improves if the
TP performance gets better (i.e., lower δ ).
Similarly, the second statement (ii) says that the TP performance improves if the OOD detection performance on each task improves (i.e., lower δ k ).
Besides, since ( (cid:80) k 1 x ∈ X k e δ k )( (cid:80) k 1 − e − δ k ) converges to 0 as δ k ’s converge to 0 in order of
O ( | (cid:80) k
δ k | ) , we further know that H TP and (cid:80)
k
H OOD,k are equivalent in quantity up to a constant factor.
In Theorem 1, we studied how CIL is related to WP and TP.
In Theorem 2, we showed that TP and
OOD bound each other.
Now we explicitly give the upper bound of CIL in relation to WP and OOD detection of each task.
The detailed proof can be found in Appendix A.4.
Theorem 3.
If H OOD,k ( x ) ≤ δ k , k = 1 , . . .
, T and H WP ( x )
≤ ϵ , we have
[Equation]
where 1 x ∈ X k is an indicator function.
3.4 Necessary Conditions for Improving CIL
In Theorem 1, we showed that good performances of WP and TP are sufficient to guarantee a good performance of CIL.
In Theorem 3, we showed that good performances of WP and OOD are sufficient to guarantee a good performance of CIL.
For completeness, we study the necessary conditions of a well-performed CIL in this sub-section.
Theorem 4.
If H CIL ( x ) ≤ η , then there exist i) a WP, s.t.
H WP ( x ) ≤ η , ii) a TP, s.t.
H TP ( x ) ≤ η , and iii) an OOD detector for each task, s.t.
H OOD,k ≤ η, k = 1 , . . .
, T .
The detailed proof is given in Appendix A.5.
This theorem tells that if a good CIL model is trained, then a good WP, a good TP and a good OOD detector for each task are always implied.
More importantly, by transforming Theorem 4 into its contraposition, we have the following statements: If for any WP, H WP ( x ) > η , then H CIL ( x ) > η .
If for any TP, H TP ( x ) >
η , then H CIL ( x ) >
η .
If for any OOD detector, H OOD,k ( x ) >
η, k
= 1 , . .
, T , then H CIL ( x ) >
η .
Regardless of whether
WP and TP (or OOD detection) are defined explicitly or implicitly by a CIL algorithm, the existence of a good WP and the existence of a good TP or OOD detection are necessary conditions for a good
CIL performance.
Remark 5 .
It is important to note again that our study in this section is based on a CIL model that has already been built.
In other words, our study tells the CIL designers what should be achieved in the final model.
Clearly, one would also like to know how to design a strong CIL model based on the theoretical results, which also considers catastrophic forgetting (CF).
One effective method is to make use of a strong existing TIL algorithm, which can already achieve no or little forgetting (CF), and combine it with a strong OOD detection
algorithm (as mentioned earlier, most OOD detection methods can also perform WP).
Thus, any improved method from the OOD detection community can be applied to CIL to produce improved CIL systems (see Sections 4.3 and 4.4).
Recall in Section 2, we reviewed prior works that have tried to use a TIL method for CIL with a task-id prediction method [55, 64, 62, 63, 65].
However, since they did not know that the key to the success of this approach is a strong OOD detection algorithm, they are quite weak (see Section 4).
New CIL Techniques and Experiments
Based on Theorem 3, we have designed several new CIL methods, each of which integrates an existing CL algorithm and an OOD detection algorithm.
The OOD detection algorithm that we use can perform both within-task IND prediction (WP) and OOD detection.
Our experiments have two goals: (1) to show that a good OOD detection method can help improve the accuracy of an existing
CIL algorithm, and (2) to fully compare two of these methods (see some others in Sec. 4.5) with strong baselines to show that they outperform the existing strong baselines considerably.
4.1 Datasets, CL Baselines and OOD Detection Methods
Datasets and CIL Tasks.
Four popular benchmark image classification datasets are used, from which six CIL problems are created following recent papers [25, 34, 26].
(1) MNIST consists of handwritten images of 10 digits with 60,000/10,000 training/testing samples.
We create a CIL problem ( M-5T ) of 5 tasks with 2 consecutive classes/digits as a task.
(2) CIFAR-10 consists of 32x32 color images of 10 classes with 50,000/10,000 training/testing samples.
We create a CIL problem ( C10-5T ) of 5 tasks with 2 consecutive classes as a task.
(3) CIFAR-100 consists of 60,000 32x32 color images with 50,000/10,000 training/testing samples.
We create two CIL problems by splitting 100 classes into 10 tasks ( C100-10T ) and 20 tasks ( C100-20T ), where each task has 10 and 5 classes, respectively.
(4) Tiny-ImageNet has 120,000 64x64 color images of 200 classes with 500/50 images per class for training/testing.
We create two CIL problems by splitting 200 classes into 5 tasks ( T-5T ) and 10 tasks ( T-10T ), where each task has 40 and 20 classes, respectively.
Baseline CL Methods.
We include different families of CL methods: regularization , replay , orthogonal projection , and parameter isolation .
MUC [25] and PASS [26] are regularization-based methods.
For replay methods, we use LwF [13], iCaRL [29], Mnemonics [69], BiC [32], DER++ [34], and Co 2 L [37].
For orthogonal projection, we use OWM [49].
Finally, for parameter isolation, we use
CCG [63], HyperNet [55], HAT [3], SupSup [4] (Sup), and PR [65].
We use the official codes for the baselines except for Co 2 L, CCG, and PR.
For these three systems, we copy the results from their papers as the code for CCG is not released and we are unable to run Co 2 L and PR on our machines.
OOD Detection Methods .
Two OOD detection methods are used.
We combine them with the above existing CL algorithms.
Both these methods can also perform within-task IND prediction (WP).
ODIN :
Researchers have proposed several methods to improve the OOD detection performance of a trained network by post-processing [7, 70, 71].
ODIN [7] is a representative method.
It adds perturbation to input and applies a temperature scaling to the softmax output of a trained network.
CSI :
It is a recently proposed OOD detection technique [6] that is highly effective.
It is based on data and class label augmentation and supervised contrastive learning [72].
Its rotation data augmentations create distributional shifted samples to act as negative data for the original samples for contrastive learning.
The details of CSI is given in Appendix D.
4.2 Training Details and Evaluation Metrics
Training Details.
For the backbone structure, we follow [4, 26, 34].
AlexNet-like architecture [73] is used for MNIST and ResNet-18 [74] is used for CIFAR-10.
For CIFAR-100 and Tiny-ImageNet,
ResNet-18 is also used as CIFAR-10, but the number of channels are doubled to fit more classes.
All the methods use the same backbone architecture except for OWM and HyperNet, for which we use their original architectures.
OWM uses AlexNet.
It is not obvious how to apply the technique to the
ResNet structure.
HyperNet uses a fully-connected network and ResNet-32 for MNIST and other datasets, respectively.
We are unable to change the structure due to model initialization arguments unexplained in the original paper.
For the replay methods, we use memory buffer 200 for MNIST and
CIFAR-10 and 2000 for CIFAR-100 and Tiny-ImageNet as in [29, 34].
We use the hyper-parameters suggested by the authors.
If we could not reproduce any result, we use 10% of the training data as a validation set to grid-search for good hyper-parameters.
For our proposed methods, we report the hyper-parameters in Appendix G.
All the results are averages over 5 runs with random seeds.
Evaluation Metrics.
Average classification accuracy over all classes after learning the
last
task.
The final class prediction depends prediction methods
(see below
We also report forgetting rate
in
Appendix J.
Average AUC (Area Under the ROC Curve) over all task models for the evaluation of OOD detection.
AUC is the main measure used in OOD detection papers.
Using this measure, we show that a better OOD detection method will result in a better CIL performance.
Let AUC k be the AUC score of task k .
It is computed by using only the model (or classes) of task k to score the test data of task k as the in-distribution (IND) data and the test data from other tasks as the out-of-distribution (OOD) data.
The average AUC score is: AUC = (cid:80)
k AUC k /n , where n is the number of tasks.
It is not straightforward to change existing CL algorithms to include a new OOD detection method that needs training, e.g., CSI, except for TIL (task incremental learning) methods, e.g., HAT and Sup.
For HAT and Sup, we can simply switch their methods for learning each task with CSI (see Sec.4.4).
Prediction Methods.
The theoretical result in Sec.
3 states that we use Eq.
2 to perform the final prediction.
The first probability (WP) in Eq. 2 is easy to get as we can simply use the softmax values of the classes in each task.
However, the second probability (TP) in Eq.
2 is tricky as each task is learned without the data of other tasks.
There can be many options.
We take the following approaches for prediction (which are a special case of Eq. 2, see below): (1).
For those approaches that use a single classification head to include all classes learned so far, we predict as follows (which is also the approach taken by the existing papers.)
[Equation]
where f ( x ) is the logit output of the network.
For multi-head methods (e.g., HAT, HyperNet, and Sup), which use one head for each task, we use the concatenated output as
[Equation]
where (cid:76) indicate concatenation and f ( x ) k is the output of task k .
These methods (in fact, they are the same method used in two different settings) is a special case of
Eq.
2 if we define OOD k as σ (max f ( x ) k ) , where σ is the sigmoid.
Hence, the theoretical results in
Sec. 3 are still applicable.
We present a detailed explanation about this prediction method and some other options in Appendix C.
These two approaches work quite well.
4.3 Better OOD Detection Produces Better CIL Performance
The key theoretical result in Sec.
3 is that better OOD detection will produce better CIL performance.
Recall our considered methods ODIN and CSI can perform both WP and OOD detection.
Applying ODIN.
We first train the baseline models using their original algorithms, and then apply temperature scaling and input noise of ODIN at testing for each task (no training data needed).
More precisely, the output of class j in task k changes by temperature scaling factor τ k of task k
as s
[Equation]
x
[Equation]
(10) and the input changes by the noise factor ϵ k as
[Equation]
(11) where ˆ y is the class with the maximum output value in task k .
This is a positive adversarial example inspired by [75].
The values τ k and ϵ k are hyper-parameters and we use the same values for all tasks except for PASS, for which we had to use a validation set to tune τ k
(see Appendix B).
[Table]
ODIN 77.
HyperNet
Original 71.
ODIN 72.
Sup
Original 79.
ODIN 80.
Tab. 1 gives the results for C100-10T.
The CIL results clearly show that the CIL performance increases if the
AUC increases with ODIN.
For instance, the CIL of
DER++ and Sup improves from 53.71 to 55.29 and 44.
to 46.74, respectively, as the AUC increases from 85.
to 88.21 and 79.16 to 80.58.
It shows that when this method is incorporated into each task model in existing trained CIL network, the CIL performance of the original method improves.
We note that ODIN does not always improve the average AUC.
For those experienced a decrease in AUC, the CIL performance also decreases except LwF.
The inconsistency of LwF is due to its severe classification bias towards later tasks as discussed in BiC [32].
The temperature scaling in ODIN has a similar effect as the bias correction in BiC, and the CIL of LwF becomes close to that of BiC after the correction.
Regardless of whether ODIN improves AUC or not, the positive correlation between AUC and CIL (except LwF) verifies the efficacy of Theorem 3, indicating better OOD detection results in better CIL performances.
Applying CSI.
We now apply the OOD detection method CSI.
Due to its sophisticated data augmentation, supervised constrative learning and results ensemble, it is hard to apply CSI to other baselines without fundamentally change them except for HAT and Sup (SupSup) as these methods are parameter isolation-based TIL methods.
We can simply replace their model for training each task with CSI wholesale (the full detail is given in Appendix D).
As mentioned earlier, both HAT and SupSup as TIL methods have almost no forgetting.
Tab. 2 reports the results of using CSI and ODIN.
ODIN is a weaker OOD method than CSI.
Both
HAT and Sup improve greatly as the systems are equipped with a better OOD detection method
CSI.
These experiment results empirically demonstrate the efficacy of Theorem 3, i.e., the CIL performance can be improved if a better OOD detection method is used.
4.4 Full Comparison of HAT+CSI and Sup+CSI with Baselines
We now make a full comparison of the two strong systems (HAT+CSI and Sup+CSI) designed based on the theoretical results.
These combinations are particularly attractive because both HAT and Sup are TIL systems and have little or no CF.
Then a strong OOD method (that can also perform WP (within-task/IND prediction) will result in a strong CIL method.
Since HAT and Sup are exemplarfree CL methods, HAT+CSI and Sup+CSI also do not need to save any previous task data.
Tab.
[Table] [Table]
c
[Table]
+CSI+
[Table]
[Table] [Table]
shows that HAT and Sup equipped with CSI outperform the baselines by large margins.
DER++, the best replay method, achieves 66.0 and 53.7 on C10-5T and C100-10T, respectively, while HAT+CSI achieves 87.8 and 63.3 and Sup+CSI achieves 86.0 and 65.1.
The large performance gap remains consistent in more challenging problems, T-5T and T-10T. We note that Sup works very poorly on
M-5T, but Sup+CSI improved it drastically, although still very weak compared to HAT+CSI.
Due to the definition of OOD in the prediction method and the fact that each task is trained separately in HAT and Sup, the outputs f ( x ) k from different tasks can be in different scales, which will result in incorrect predictions.
To deal with the problem, we can calibrate the output as α k f ( x ) k
+ β k and use
OOD k
= σ ( α k f ( x ) k + β k ) .
The optimal α ∗ k and β ∗ k for each task k can be found by optimization with a memory buffer to save a very small number of training examples from previous tasks like that in the replay-based methods.
We refer the calibrated methods as HAT+CSI+c and Sup+CSI+c.
They are trained by using the memory buffer of the same size as the replay methods (see Sec. 4.2).
Tab.shows that the calibration improves from their memory free versions, i.e., without calibration.
We provide the details about how to train the calibration parameters α k and β k in Appendix E.
As shown in Theorem 1, the CIL performance also depends on the TIL (WP) performance.
We compare the TIL accuracies of the baselines and our methods in Tab. 4.
Our systems again outperform the baselines by large margins on more challenging datasets (e.g., CIFAR100 and Tiny-ImageNet).
[Table]
4.5 Implications for Existing CL Methods, Open-World Learning and Future Research
Implication for regularization and replay methods.
Regularization-based (exemplar-free) methods try to protect important parameters of old tasks to mitigate CF.
However, since the training of each task does not consider OOD detection, TP will be weak
[Table]
which causes difficulty for inter-task class separation (ICS) and thus low CL accuracy.
Replay-based methods are better as the replay data from old tasks can be naturally regarded as OOD data for the current task, then a better OOD model is built, which improves TP.
However, since the replay data is small.
the OOD model is sub-optimal, especially for earlier tasks as their training cannot see any future task data.
Thus for both approaches, it will be beneficial to consider CF and OOD together in learning each task (e.g., [76]).
Implication for open-world learning.
Since our theory says that CL needs OOD detection, and OOD detection is also the first step in open-world learning (OWL), CL and OWL naturally work together to achieve self-motivated open-world continual learning [5] for autonomous learning or AI autonomy.
That is, the AI agent can continually discover new tasks (OOD detection) and incrementally learn the tasks (CL) all on its own with no involvement of human engineers.
Further afield, this work is also related to curiosity-driven self-supervised learning [77] in reinforcement learning and 3D navigation.
Limitation and future work.
The proposed theory provides a principled guidance on what needs to be done in order to achieve good CIL results, but it gives no guidance on how to do it.
Although two example techniques are presented and evaluated, they are empirical.
There are many options to define WP and TP (or OOD).
An idea in [40] may be helpful in this regard.
[40] argues that a continual learner should learn holistic feature representations of the input data, meaning to learn as many features as possible from the input data.
The rationale is that if the system can learn all possible features from each task, then a future task does not have to learn those shared/intersecting features by modifying the parameters, which will result in less CF and also better ICS.
A full representation of the IND data also improves OOD detection because the OOD score of a data point is basically the distance between the data point and the IND distribution.
Only capturing a subset of features (e.g., by cross entropy) will result in poor OOD detection [78] because those missing features may be necessary to separate IND and some OOD data.
In our future work, we will study how to optimize
WP and TP/OOD and find the necessary conditions for them to do well.
Conclusion
This paper proposed a theoretical study on how to solve the highly challenging continual learning (CL) problem.
class incremental learning (CIL) (the other popular CL setting is task incremental learning (TIL)).
The theoretical result provides a principled guidance for designing better CIL algorithms.
The paper first decomposed the CIL prediction into within-task prediction (WP) and task-id prediction (TP).
WP is basically TIL.
The paper further theoretically demonstrated that TP is correlated with out-of-distribution (OOD) detection.
It then proved that a good performance of the two is both necessary and sufficient for good CIL performances.
Based on the theoretical result, several new CIL methods were designed.
They outperform strong baselines in CIL and also in TIL by a large margin.
Finally, we also discussed the implications for existing CL techniques and open-world learning.
Acknowledgments
The work of Gyuhak Kim, Zixuan Ke and Bing Liu was supported in part by a research contract from
KDDI, two NSF grants (IIS-1910424 and IIS-1838770), and a DARPA contract HR001120C0023.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ??.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
4.2 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Theorems and Corollaries A.1 Proof of Theorem
Proof.
Since
H
[Equation]
and
[Equation]
we have
[Equation]
A.2 Proof of Corollary 1.
Proof.
By proof of Theorem 1, we have
[Equation]
Taking expectations on both sides, we have i)
[Equation]
and ii)
[Equation]
A.3 Proof of Theorem 2.
Proof.
i)
Assume x ∈ X k 0 .
For k = k 0 , we have
[Equation]
For k ̸ = k 0 , we have
[Equation]
ii) Assume x ∈ X k 0 .
For k = k 0 , by H OOD,k 0 ( x ) ≤ δ k 0 , we have
[Equation]
which means
P
[Equation]
For k ̸ = k 0 , by H OOD,k ( x ) ≤ δ k , we have
[Equation]
which means
[Equation]
Therefore, we have
[Equation]
Hence,
[Equation]
A.4 Proof of Theorem 3.
Proof.
Using Theorem 1 and 2,
[Equation]
A.5 Proof of Theorem 4.
Proof.
i)
Assume x ∈ X k 0 ,j 0 ⊂ X k 0 .
Define P ( x ∈ X k,j | x ∈ X k , D ) =
P ( x ∈ X k,j | D ) .
According to proof of Theorem 1,
[Equation]
Hence, we have
[Equation]
ii)
Assume x ∈ X k 0 ,j 0 ⊂ X k 0 .
Define P ( x ∈ X k | D )
= (cid:80) j P ( x ∈ X k,j | D ) .
According to proof of Theorem 1,
H TP (
[Equation]
= −
[Equation]
X
[Equation] [Equation]
D ) ,
H
[Equation]
= − log
[Equation]
∈
X k 0 ,j 0 | D ) .
Hence, we have
H TP (
[Equation]
= −
[Equation] [Equation]
X
[Equation]
| D )
= − log (cid:88)
j
P ( x ∈ X k
[Equation]
j | D ) ≤ − log P ( x ∈ X k
[Equation]
j 0 | D )
=
H CIL ( x ) ≤ η. iii)
Assume x ∈ X k 0 ,j 0 ⊂ X k 0 .
Define P ′
i ( x ∈ X k | D ) =
P ( x ∈ X k | D ) =
(cid:80) j P ( x ∈ X k,j | D ) .
According to proof of Theorem 4 ii), we have
H TP ( x ) ≤ η.
According to proof of Theorem 2 i), we have
H OOD,i ( x ) ≤ H TP ( x ) .
Therefore,
H OOD,i ( x ) ≤ H TP ( x ) ≤ η.
B Additional Results and Explanation Regarding Table 1 in the Main Paper
In Sec. 4.3, we showed that a better OOD detection improves CIL performance.
For the postprocessing method ODIN, we only reported the results on C100-10T due to space limitations.
Tab.shows the results on the other datasets.
A continual learning method with a better AUC shows a better CIL performance than other methods with lower AUC.
For instance, original HAT achieves AUC of 82.47 while HyperNet achieves 78.
on C10-5T. The CIL for HAT is 62.67 while it is 53.40 for HyperNet.
However, there are some exceptions that this comparison does not hold.
An example is LwF. Its AUC and CIL are 89.39 and 54.67 on C10-5T. Although its AUC is better than HAT, the CIL is lower.
This is due to the fact that
CIL improves with WP and TP according to Theorem 1.
The contraposition of Theorem 4 also says if the cross-entropy of TIL is large, that of CIL is also large.
Indeed, the average within-task prediction (WP) accuracy for LwF on C10-5T is 95.2 while the same for HAT is 96.7.
Improving WP is also important in achieving good CIL performances.
For PASS, we had to tune τ k using a validation set.
This is because the softmax in Eq.
10 improves
AUC by making the IND (in-distribution) and OOD scores more separable within a task, but deteriorates the final scores across tasks.
To be specific, the test instances are predicted as one of the classes in the first task after softmax because the relative values between classes in task 1 is larger than the other tasks in PASS.
Therefore, larger τ 1 and smaller τ k , for k > 1 , are chosen to compensate the relative values.
[Table]
C Definitions of TP
As noted in the main paper, the class prediction in Eq.
2 varies by definition of WP and TP.
The precise definition of WP and TP depends on implementation.
Due to this subjectivity, we follow the prediction method as the existing methods in continual learning, which is the arg max over the output.
In this section, we show that the arg max over output is a special case of Eq.
We also provide CIL results using different definitions of TP.
We first establish another theorem.
This is an extension of Theorem 2 and connects the standard prediction method to our analysis.
Theorem 5 (Extension of Theorem 2) .
i)
If H TP ( x ) ≤ δ , let P ′ k
( x ∈ X k | D )
=
P ( x ∈ X k | D ) 1 /τ k , ∀ τ k > 0 , then H OOD,k ( x ) ≤ max( δ/τ k , − log(1 − (1 − e − δ ) 1 /τ k ) , ∀ k = 1 , . .
, T . ii)
If H OOD,k ( x ) ≤ δ k , k = 1 , . . .
, T , let P ( x ∈ X k | D ) =
[Equation]
x
[Equation]
X k
[Equation] [Equation]
j P
[Equation]
x
[Equation]
|
[Equation] [Equation]
τj , ∀ τ k > 0 , then
H TP ( x ) ≤ (cid:80) k
[Equation]
where 1 x ∈ X k is an indicator function.
In Theorem 5 (proof appears later), we can observe that δ/τ k decreases with the increase of τ k , while − log(1 − (1 − e − δ ) 1 /τ k ) increases.
Hence, when TP is given, let δ
= H TP ( x ) , we can find the optimal τ i to define OOD by solving δ/τ k
= − log(1 − (1 − e − δ ) 1 /τ k ) .
Similarly, given OOD, let δ k
= H OOD,k ( x ) , we can find the optimal τ 1 , . .
, τ T to define TP by finding the global minima of (cid:80) k 1 x ∈ X
k δ k τ k + (cid:80) k
(1 − e − δk ) 1 /τk (cid:80) k 1 x ∈ X k (1 − (1 − e − δk ) 1 /τk ) .
The optimal τ k can be found using a memory buffer to save a small number of previous data like that in a replay-based continual learning method.
In Theorem 5 (ii), let P ′ k
( x ∈ X k | D )
=
σ (max f ( x ) k ) , where σ is the sigmoid and f ( x ) k is the output of task k and choose τ k ≈ 0 for each k .
Then P ( x ∈ X k | D ) becomes approximately 1 for the task k where the maximum logit value appears and 0 for the rest tasks.
Therefore, Eq. 2 in the paper
P
[Equation]
k,
[Equation]
x
[Equation] [Equation]
x
[Equation] [Equation]
P
[Equation]
) is zero for all classes in tasks k ′ ̸
= k .
Since only the probabilities of classes in task k are non-zero, taking arg max over all class probabilities gives the same class as arg max over output logits.
[Table]
HyperNet
[Table]
We have also tried another definition of WP and TP.
The considered WP is
[Equation] [Equation]
where ν k is a temperature scaling parameter for task k , and the TP is
[Equation] [Equation]
where P ′ k ( x ∈ X k | D )
= max j e f
[Equation]
x
[Equation]
kj /τ k / (cid:80) j e f ( x ) kj /τ k and τ k is a temperature scaling parameter.
This is the maximum softmax of task k .
We choose ν k = 0 .
1 and τ k = 5 for all k .
A good τ and ν can be found using grid search on a validation set.
However, one can also find the optimal values by optimization using some past data saved for memory buffer.
The CIL results for the new prediction method is in Tab. 6.
Proof of Theorem 5.
i)
Assume x ∈ X k 0 .
For k = k 0 , we have
[Equation]
For k ̸ = k 0 , we have
[Equation]
ii) Assume x ∈ X k 0 .
For k = k 0 , by H OOD,k 0 ( x ) ≤ δ k 0 , we have
[Equation]
which means
P
[Equation]
For k ̸ = k 0 , by H OOD,k ( x ) ≤ δ k , we have
[Equation]
, which means
[Equation]
Therefore
[Equation]
we have
[Equation]
Hence,
[Equation]
D Details of HAT, Sup, and CSI
We have proposed two highly effective new CIL methods, HAT+CSI and Sup+CSI, by integrating the existing parameter isolation based continual learning (CL) method HAT [3] or Sup [4] with the strong
OOD detection method CSI [6].
We replaced the training loss of HAT and Sup by that of CSI while applying the continual learning techniques of the respective method.
In this section, we overview
Sup, HAT, and CSI, and explain how to train them continually.
Figure 1 shows the overall training frameworks of Sup+CSI and HAT+CSI.
Denote feature extractor by h , classifier by f , and the parameters by W .
In the main paper, we denote the output of task k by f ( x ) k for both a single-head or multi-head method (e.g., Eq. 9) for consistency.
In this section, we use f ( x, k ) to indicate the output of task k to be more explicit as both
HAT and Sup are multi-head methods (one head for each task) designed for task incremental learning (TIL).
[Figure]
is by taking concatenated the network, the training batch is augmented to give different views of samples for contrastive training in the OOD detection algorithm CSI
The training consists of two steps following CSI
The first step is learning the feature extractor.
In this step, the Edge Popup algorithm [79] is applied to find a sparse network for each
The sparse networks, which are indicated by edges of different colors in the diagram
The second fixed prediction is also made by over the concatenated output each method.
Due to the OOD detection algorithm CSI, the overall training process is similar to Sup
CSI except that it applies the hard attention algorithm [3].
In training feature extractor task embeddings are applied to find hard masks at each layer.
Then given the learned feature representations fine-
D.1 Sup
SupSup (Sup) [4] trains supermasks by Edge Popup algorithm
More precisely, given initial W , find binary masks M k for task k to minimize the cross-entropy loss
[Equation]
(14) where X k is the training data for task k , and
[Equation]
where ⊗ indicates an element-wise product.
The masks are obtained by selecting the top p % of entries in the score matrices V .
The p value determines the sparsity of the mask M k .
The subnetwork found by Edge Popup algorithm is indicated by different colors in Figure 1(a).
Given the task-id k of a test instance at inference, the system (which is referred as Sup GG in the original Sup paper) uses the task-specific mask M k to obtain the classification output.
By integrating the OOD detection method, CSI, during training, Sup+CSI does not require to know the task-id of test instance, which makes Sup+CSI applicable to CIL (class incremental learning).
D.2 HAT
We now discuss the hard attention (mask) mechanism of HAT [3].
It finds binary masks a kl for each layer l and task k , and uses them to block/unblock information flow at forward and backward pass.
More precisely, the hard attention is defined as
[Equation]
where σ is the sigmoid, s is a positive constant, and e kl is a learnable embedding.
To approximate the binary mask, the system uses a large s value.
The attention is applied to the output at each layer as
[Equation] [Equation]
where ⊗ is an element-wise product, and
[Equation]
The neurons with attention value 1 is important for task k
while those with zero attention value are not necessary for the task, and thus they can be freely changed without affecting the output value h ′ l .
The system needs to know which neurons are important to protect the previous knowledge from forgetting.
Denote the accumulated attentions of all previous tasks by
[Equation]
where a 0 l is the zero vector and max is an element-wise maximum.
The gradients of parameters corresponding to important neurons is modified as
[Equation]
where a <ki,l is the i ’th unit of a <kl and l = 1 , · · · , L − 1 .
The hard attention is not applied to the last layer L since it is a task-specific classification layer.
To encourage sparsity in a kl , the system uses regularization as
[Equation]
21) where λ k is a hyper-parameter.
The system minimizes the loss
[Equation]
22) where L ce is the cross-entropy loss.
The overall framework of the algorithm is shown in Figure 1(b).
D.3 CSI
We now explain the OOD detection method CSI, and how to incorporate it in HAT and Sup.
CSI is based on contrastive learning [59, 58] and data augmentation due to their excellent performance [6].
Since this section focuses on how to learn a single task based on OOD detection, we omit the task-id unless necessary.
The OOD training process is similar to that of contrastive learning.
It consists of two steps: 1) learning the feature representation by the composite g ◦ h , where h is a feature extractor and g is a projection to contrastive representation, and 2) learning a linear classifier f mapping the feature representation of h to the label space.
This two step training process is outlined in Figure 1(a) and (b).
In the following, we describe the training process: contrastive learning for feature representation learning (1), and OOD classifier building (2).
We then explain how to make a prediction based on an ensemble method to further improve prediction.
D.3.1 Contrastive Loss for Feature Learning.
This is step 1.
Supervised contrastive learning is used to try to repel data of different classes and align data of the same class more closely to make it easier to classify them.
A key operation is data augmentation via transformations.
Given a batch of N samples, each sample x is first duplicated and each version then goes through three initial augmentations (horizontal flip, color changes, and Inception crop [80]) to generate two different views x 1 and x 2 (they keep the same class label as x ).
Denote the augmented batch by B , which now has 2 N samples.
In [81, 6], it was shown that using image rotations is effective in learning
OOD detection models because such rotations can effectively serve as out-of-distribution (OOD) training data.
For each augmented sample x ∈ B with class y of a task, we rotate x by 90 ◦ , 180 ◦ , 270 ◦ to create three images, which are assigned three new classes y 1 , y 2 , and y 3 , respectively.
This results in a larger augmented batch ˜ B .
Since we generate three new images from each x , the size of ˜ B is 8 N .
For each original class, we now have 4 classes.
For a sample x ∈ ˜ B , let ˜ B ( x ) = ˜
B\{ x } and let
P ( x ) ⊂ ˜
B\{ x } be a set consisting of the data of the same class as x distinct from x .
The contrastive representation of a sample x is z x = g ( h ( x, t )) /
∥ g ( h ( x, t ))
∥ , where t is the current task.
In learning, we minimize the supervised contrastive loss [72] of task t .
[Equation]
where τ is a scalar temperature, · is dot product, and × is multiplication.
The loss is reduced by repelling z of different classes and aligning z of the same class more closely.
L c basically trains a feature extractor with good representations for learning an OOD classifier.
Since the feature extractor is shared across tasks in continual learning, a protection is needed to prevent catastrophic forgetting.
HAT and Sup use their respective technique to protect their feature extractor from forgetting.
Therefore, the losses L of Eq.
14 and L ce of Eq.
22 are replaced by Eq.while the forgetting prevention mechanisms still hold.
D.3.2 Learning the Classifier.
This is step 2.
Given the feature extractor h trained with the loss in Eq.
23, we freeze h and only fine-tune the linear classifier f , which is trained to predict the classes of task t and the augmented rotation classes.
f maps the feature representation to the label space in R 4 |C t | , where 4 is the number of rotation classes including the original data with 0 ◦ rotation and |C t | is the number of original classes in task t .
We minimize the cross-entropy loss,
[Equation]
where ft indicates fine-tune, and
[Equation]
where f ( h ( x, t ))
∈ R 4 |C t | .
The output f ( h ( x, t )) includes the rotation classes.
The linear classifier is trained to predict the original and the rotation classes.
Since individual classifier is trained for each task and the feature extractor is frozen, no protection is necessary.
D.3.3 Ensemble Class Prediction.
We describe how to predict a label y ∈ C t (TIL) and y ∈ C (CIL) ( C is the set of original classes of all tasks).
We assume all tasks have been learned and their models are protected by masks.
We discuss the prediction of class label y for a test sample x in the TIL setting first.
Note that the network f ◦ h in Eq.
25 returns logits for rotation classes (including the original task classes).
Note also for each original class label j k ∈ C k (original classes) of a task k , we created three additional rotation classes.
For class j k , the classifier f will produce four output values from its four rotation class logits, i.e., f j k , 0 ( h ( x 0 , k )) , f j k , 90 ( h ( x 90 , k )) , f j k , 180 ( h ( x 180 , k )) , and f j k , 270 ( h ( x 270 , k )) , where 0, 90, 180, and 270 represent 0 ◦ , 90 ◦ , 180 ◦ , and 270 ◦ rotations respectively and x 0 is the original x .
We compute an ensemble output f j k ( h ( x, k )) for each class j k ∈ C k of task k ,
[Equation]
We use Eq.
9 to make the CIL class prediction, where the final class prediction is made as
[Equation]
E Output Calibration
In this section, we discuss the output calibration technique used in Sec.
4.4 to improve the final prediction accuracy.
Even if an OOD detection of each task was perfect (i.e. the model accept and reject IND and OOD samples perfectly), the system could make incorrect class prediction if the magnitudes of outputs across different tasks are different.
To ensure that the output values are comparable, we calibrate the outputs by scaling α k and shifting β k for each task.
The optimal parameters ( α k , β k ) ∈
R × R can be found by solving optimization problem using samples in memory buffer.
More precisely, denote the memory buffer M and calibration parameters ( α, β ) ∈ R T × R T , where T is the number of learned tasks.
After training T th task, we find optimal calibration parameters by minimizing the cross-entropy loss,
[Equation]
[Table]
where p ( c | x ) is computed using the softmax,
[Equation]
29) where (cid:76) indicates the concatenation and f ( x ) k is the output of task k as Eq. 9.
Given the optimal parameters ( α ∗ , β ∗ ) , we make final prediction as
[Equation]
If we use OOD k = σ ( α ∗ k f ( x ) k
+ β ∗ k ) , where σ is the sigmoid, and TP k =
OOD k / (cid:80) k ′
OOD k ′ , the theoretical results in Sec.
3 hold.
F TIL (WP) Results
The TIL (WP) results of all the systems are reported in Tab. 7.
HAT and Sup show strong performances compared to the other baselines as they leverage task-specific parameters.
However, as shown in
Theorem 1, the CIL depends on TP (or OOD).
Without an OOD detection mechanism in HAT or
Sup, they perform poorly in CIL as shown in the main paper.
The contrastive learning in CSI also improves the IND prediction (i.e., WP), and this along with OOD detection results in the strong CIL performance.
G Hyper-parameters
Here we report the hyper-parameters that we did not report in the main paper due to space limitations.
We mainly report the hyper-parameters of the proposed methods, HAT+CSI, Sup+CSI, and their calibrated versions.
For all the experiments of the proposed methods, we use the values chosen by the original CSI [6].
We use LARS [82] optimization with learning rate 0.1 for training the feature extractor.
We linearly increase the learning rate by 0.1 per epoch for the first 10 epochs.
After that, we use cosine scheduler [83] without restart as in [6, 59].
After training the feature extractor, we train the linear classifier for 100 epochs with SGD with learning rate 0.1 and reduce the rate by 0.1 at 60, 75, and 90 epochs.
For all the experiments except MNIST, we train the feature extractor forepochs with batch size 128.
For the following hyper-parameters, we use 10% of training data for validation to find a good set of values.
For the number of epochs and batch size for MNIST, Sup+CSI trains for 1000 epochs with batch size of 32 while HAT+CSI trains for 700 epochs with batch size of 256.
The hard attention regularization penalty λ i in HAT is different by experiments and task i .
For MNIST, we use λ 1 = 0 .
25 , and λ 2 = · · · = λ 5 = 0 .
For C10-5T, we use λ 1 = 1 . 0 , and λ 2 = · · · = λ 5 = 0 .
For C100-10T, λ 1 = 1 . 5 , and λ 2 = · · · = λ 10 = 1 . 0 are used.
For C100-20T, λ 1 = 3 . 5 , and
[Table]
Sup
[Table]
+
[Table]
λ 2 = · · · = λ 20 = 2 .
5 are used.
For T-5T, λ i = 0 .
75 for all tasks, and lastly, for T-10T, λ 1 = 1 . 0 , and λ 2 = · · · = λ
10 = 0 .
75 are used.
We use larger λ 1 for the first task than the later tasks as we have found that the larger regularization on the first task results in better accuracy.
This is by the definition of regularization in HAT.
The earlier task gives lower penalty than later tasks.
We manually give larger penalty to the first task.
We did not search hyper-parameter λ t for tasks t ≥ 2 .
For sparsity in Sup+CSI, we simply choose the least sparsity value of 32 used in the original Sup paper without parameter search.
Calibration methods (HAT+CSI+c and Sup+CSI+c) are based on its memory free versions (i.e.
HAT+CSI and Sup+CSI).
Therefore, the model training part uses the same hyper-parameters as their calibration free counterparts.
For calibration training, we use SGD with learning rate 0.01,training iterations, and batch size of 15 for HAT+CSI+c for all experiments.
For Sup+CSI+c, we use the same values for all the experiments except for MNIST.
For MNIST, we use learning rate 0.05, batch size of 8, and run 280 iterations.
For the baselines, we use the hyper-parameters reported in the original papers or in their code.
If the hyper-parameters are unknown or the code does not reproduce the result (e.g., the baseline did not implement a particular dataset or the code had undergone significant version change), we search for the hyper-parameters as we did for HAT+CSI and Sup+CSI.
H Computes and Resources Used in Experiments
This paper provides a guidance on how to solve the CIL problem, backed by theoretical justifications.
Based on the guidance, we have proposed some new CIL methods.
Two outstanding ones are
HAT+CSI and Sup+CSI.
These methods achieve state-of-the-art CIL performances, but by no mean, they are the only approaches.
Many CIL algorithms can be designed following the analysis as it is general to any CL model.
Despite the generality of our work, we report the execution time and required memory for HAT+CSI and Sup+CSI.
The report is based on a machine with NVIDIA RTX 3090 on C10-5T experiments.
HAT+CSI takes 28.68 hours while Sup+CSI runs for 18.41 hours, which are slower than baselines.
Contrastive learning and extensive data augmentation in CSI are the major reason for the slow execution time.
However, if other more efficient OOD detection algorithms can replace CSI, the running time can be improved with the new OOD detection methods.
As noted in Sec.
4.2, all the methods use the same backbone architecture with the same width and depth except for OWM and HyperNet for the reasons explained in the main paper.
We report the number of parameters of each method required for inference after learning the last task.
Sup and
Sup+CSI uses a very small number of parameters because Sup finds a sparse subnetwork for each task.
Our methods HAT+CSI introduces 7.7K, 17.6K, 68.0K, 47.5K, 191.0K, and 109.0K parameters on
M-5T, C10-5T, C100-10T, C100-20T, T-5T, and T-10T, respectively, at each task.
Sup+CSI introduces 56.3K, 284.9K, 590.3K, 580.0K, 607.7K, and 605.1K parameters on the same experiments.
The calibrated methods HAT+CSI+c and Sup+CSI+c introduce 2 parameters ( α k , β k ) per task.
For HAT and HAT+CSI, the reported number of parameters is based on the network at full capacity.
The hard attention masks consume 71.10, 86.31, 98.89, 99.71, 92.94, and 98.67% of the total network capacity on average over 5 runs for HAT on M-5T, C10-5T, C100-10T, C100-20T, T-5, and T-10T, respectively.
Similarly, 99.39, 99.56, 99.56, 50.68, 94.94, and 99.18% of the total network capacity are used for HAT+CSI on the same datasets on average.
I Negative Societal Impacts
The goal of continual learning is to learn a sequence of tasks incrementally.
Like many machine learning algorithms, our proposed methods could be affected by bias in the input data as this work does not deal with fairness or bias in the data.
A possible solution to mitigate the problem is to check bias in data before training.
J Forgetting Rate
We discuss forgetting rate (i.e., backward transfer) [28], which is defined for task t as
[Equation]
(31) where A init k is the classification accuracy of task k ’s data after learning it for the first time and A tk is the accuracy of task k ’s data after learning task t .
We report the forgetting rate after learning the last task.
M-5T
C
[Figure]
Figure 2 shows the forgetting rates of each method.
Some methods (e.g., OWM, iCaRL) experience less forgetting than the proposed methods HAT+CSI and Sup+CSI on M-5T. On this dataset, all the systems performed well.
For instance, OWM and iCaRL achieve 95.8% and 96.0% accuracy while HAT+CSI and HAT+CSI+c achieve 94.4 and 96.9% accuracy.
As we have noted in the main paper, Sup+CSI and Sup+CSI+c achieve only 80.7 and 81.0 on M-5T although they have improved drastically from 70.1% of the base method Sup.
OWM and HyperNet show lower forgetting rates than HAT+CSI+c and Sup+CSI+c on T-5T and
T-10T. However, they are not able to adapt to new classes as OWM and HyperNet achieve the classification accuracy of only 10.0% and 7.9%, respectively, on T-5T and 8.6% and 5.3% on T-10T.
HAT+CSI+c and Sup+CSI+c achieves 51.7% and 49.2%, respectively, on T-5T and 47.6% and 46.2% on T-10T.
In fact, the performance reduction (i.e., forgetting) in our proposed methods occurs not because the systems forget the previous task knowledge, but because the systems learn more classes and the classification naturally becomes harder.
The continual learning mechanisms (HAT and Sup) used in the proposed methods experience little or no forgetting because they find independent subset of parameters for each task, and the learned parameters are not interfered during training.
For the forgetting rate results in the TIL setting, refer to our earlier workshop paper [84].