O PTIMIZING SPCA - BASED CONTINUAL LEARNING : A THEORETICAL APPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Catastrophic forgetting and the stability-plasticity dilemma are two major obstacles to continual learning.
In this paper we ﬁrst propose a theoretical analysis of a
SPCA-based continual learning algorithm using high dimensional statistics.
Second, we design OSCL (Optimized Spca-based Continual Learning) which builds on a ﬂexible task optimization based on the theory.
By optimizing a single task, catastrophic forgetting can be prevented theoretically.
While optimizing multitasks, the trade-off between integrating knowledge from the new task and retaining previous knowledge of the old tasks can be achieved by assigning appropriate weights to corresponding tasks in compliance with the objectives.
Experimental results conﬁrm that the various theoretical conclusions are robust to a wide range of data distributions.
Besides, several applications on synthetic and real data show that the proposed method while being computationally efﬁcient, achieves comparable results with some state of the art.
I NTRODUCTION
Continual learning paradigm.
Machine learning methods generally learn from samples of data randomly drawn from a stationary distribution.
However, this scenario is rare in reality.
Continual learning (CL) is a particular machine learning paradigm in which data continuously arrive in a possibly non i.i.d.
way and the knowledge is accumulated over time (Schlimmer & Fisher, 1986;
Ebrahimi et al., 2019; Lee et al., 2020; De Lange et al., 2021).
The study of continual learning is essential to develop artiﬁcial intelligence systems that emulate humans and to design machine learning systems that are appropriate for real-world applications.
On the one hand, humans continue to acquire knowledge and solve new problems throughout their lifetimes.
The goal of continual learning is to mimic the capacity of humans to learn from a non-stationary data stream without forgetting catastrophically the learned knowledge (Titsias et al., 2019; Lee et al., 2020).
On the other hand, when deploying a trained model on real applications, the distribution of the data will consistently drift over time.
Therefore, the machine learning algorithm must be able to adapt continuously to these changes (Kirkpatrick et al., 2017; Lesort et al., 2020).
Challenges in continual learning.
One of the major challenges of continual learning is to avoid catastrophic forgetting (McCloskey & Cohen, 1989; Chen & Liu, 2018; Aljundi, 2019).
This occurs when the performance of previous tasks is severely degraded during the learning process.
To introduce more ﬂexibility and to take into account both the current task and the previous tasks, the stability-plasticity dilemma was introduced (Nguyen et al., 2017; Lopez-Paz & Ranzato, 2017;
Rajasegaran et al., 2019).
More speciﬁcally, plasticity refers to the ability of integrating new knowledge, and stability to the capacity of retaining previous knowledge (which is related to catastrophic forgetting ).
Note that the term catastrophic forgetting , although strongly referenced in the literature in deep neural network models, is a fairly general concept that can occur in any machine learning algorithm as it has been noted in shadow single-layered models, such as self-organizing feature maps (Richardson & Thomas, 2008; Chen & Liu, 2018).
State of the art.
Most works on CL have focused on purely supervised tasks which is the focus of the present work (Kirkpatrick et al., 2017;
Nguyen et al., 2017; Ebrahimi et al., 2019; Lee et al., 2020).
Supervised CL can be classiﬁed into three main categories based on how knowledge and data are updated and stored: replay, regularization, and dynamic architecture based methods (De Lange et al., 2021).
Replay-based approaches address the catastrophic forgetting by saving and reusing the previously seen data while learning a new task (Lopez-Paz & Ranzato, 2017; Isele & Cosgun, 2018; Titsias et al., 2019).
Regularization-based methods penalize the update of crucial weights to alleviate forgetting by introducing an extra regularization term in the loss function (Kirkpatrick et al., 2017; Ebrahimi et al., 2019).
Dynamic architecture-based methods ﬂexibly update the learning model as new tasks are added based on the task complexity and the relation between the tasks (Rajasegaran et al., 2019; Lee et al., 2020).
Although successful cases of supervised continual learning were reported in articles, the algorithms tend to feature unpredictable behavior, generally requiring a host of additional inconvenient hyperparameters and requiring a high computational cost.
More importantly, supervised continual learning methods lack theoretical guarantees of avoiding catastrophic forgetting .
Contributions of the paper.
In this paper, we introduce a novel CL method based on Supervised
Principal Component Analysis (SPCA).
We provide a theoretical analysis of the proposed method using high dimensional statistics.
This analysis allows us to predict in advance the performance of the algorithm.
Furthermore, we develop a label optimization scheme based on the theory that allows us to guarantee theoretically that catastrophic forgetting is avoided.
As a result, we obtain a simple and efﬁcient continual learning algorithm free of hyperparameters, at a low computational cost named OSCL .
Moreover, the theory allows us to weight the tasks during the learning process according to the user’s preferences and priorities.
This is in line with the resolution of the stabilityplasticity dilemma.
As such, the main contributions of the paper can be summarized as follows
• We propose a simple continual learning algorithm, computationally inexpensive based on supervised principal component analysis, and provide a theoretical analysis (exact classiﬁcation error rather than bounds) using high dimensional statistics.
• Using the theoretical result, we develop a label optimization scheme that provably prevent catastrophic forgetting and also allows for a weighting of the tasks during learning.
• Several applications are presented to corroborate the practical usefulness of the approach in terms of efﬁciency and ﬂexibility.
Outlines.
The remainder of the paper is organized as follows.
Section 2 discusses several works in the continual learning literature and highlights the difference and contributions of the present paper.
Section 3 introduces and formalizes the continual learning framework and proposes a simple continual learning algorithm based on Supervised Principal Component Analysis.
Section 4 proposes a theoretical analysis of the this algorithm as well as ﬂexible optimization tools to beneﬁt from all tasks.
Section 5 provides several applications to corroborate the different conclusions of the paper.
Notations.
Matrices will be represented by bold capital letters ( e.g., matrix A ).
Vectors will be represented in bold lowercase letters ( e.g., vector v ) and scalars will be represented without bold letters ( e.g., variable a ).
The canonical vector of size n is denoted by e [ n ] m ∈ R n , 1 ≤ m ≤ n , where the i -th element is 1 if i = m , and 0 otherwise.
The diagonal matrix with diagonal x andelsewhere is denoted by D x .
Generally, the subscript t refers to the task number, and the superscript j to the class index.
As an example, x ( j ) t(cid:96) denotes the (cid:96) -th sample of class j for Task t .
1 n ∈ R n is the vector of all ones and the matrix Σ v ∈ R n × n denotes the covariance matrix of the random vector v .
[ n ] denotes the set { 1 , . .
. , n } .
R ELATED WORKS 2. C ONTINUAL LEARNING
As mentioned in the introduction, we focus on the state of the art of supervised CL, which is divided into three main concepts that we develop in detail in this section.
Appendix provides a more detailed picture for interested readers.
For the sake of completeness, we also discuss meta-learning-based continual learning in this section.
Replay-based methods.
Replay-based approaches aim to retain a certain amount of historical examples, extracted features, or generated examples to reduce forgetting when training the model with new data.
Three challenges need to be solved.
The ﬁrst problem is selecting appropriate previous samples.
In this context, Isele & Cosgun (2018) proposed four strategies for choosing which data will be stored.
Aljundi et al.
(2019) proposed sample selection as a constraint reduction problem.
In order to improve sample diversity, (Bang et al., 2021) propose a new memory management strategy that incorporates classiﬁcation uncertainty per sample and data augmentation
The second challenge is to ﬁnd a suitable format to store the data from past tasks.
In this direction, Shin et al.
Ayub & Wagner (2020) used a deep generative model to sample pseudo-data points to replay the past experience while Farajtabar et al.
(2020) stored a set of gradients of each task to minimize catastrophic forgetting .
The last challenge is to ﬁnd a better way to use historical data.
Lopez-Paz & Ranzato (2017); Chaudhry et al.
; Farajtabar et al.
(2020) avoided interference between tasks by projecting the new gradients in the feasible region constrained by the gradient(s) of previous tasks.
Wang et al.
(2022) achieved a better stability-plasticity trade-off using an attention=based framework and dual-classiﬁer structure.
(Hersche et al., 2022) composed of a frozen feature extractor, a trainable ﬁxed size fully connected layer, and a re-writable dynamically growing EM (Explicit
Memory) that stores as many vectors as the number of classes encountered. (Wan et al., 2022) enforce the three losses for a continual learner for long term embedding consistency. (Tang et al., 2022) developed a learnable feature generator and use semantic contrastive learning to prevent DNN from forgetting when learning new tasks. It is important to note that our method does not use any samples from the previous tasks and only stores statistical data.
Therefore our method is different from the aforementioned algorithms.
Regularization-based methods.
The regularization-based approaches generally impose a regularization constraint on the objective function to maintain a balance between stability and plasticity .
The literature focuses on two main issues.
In the ﬁrst place, it is necessary to evaluate the importance of parameters in the previous model and, in the second place, to limit the adjustment of critical weights.
With Kirkpatrick et al.
(2017), such importance is computed from the diagonal of the
Fisher information matrix and important weights are constrained to change slowly while learning new tasks.
Nguyen et al.
(2017) extended online variational inference to handle more general continual learning tasks.
Ebrahimi et al.
; Zeno et al.
(2018) used uncertainty as the importance of the weights and subsequently, adapted the learning rate whereas Jung et al.
(2020) considers the average activation value as the importance of a node.
Our algorithm does not use any regularization and therefore differs from regularization-based approaches.
Dynamic architecture-based methods.
Dynamic architecture-based approaches adjust the model architecture according to the complexity of the tasks, and the relation between them.
Two questions are critical in this approach: the timing to adjust the model architecture and the procedure to do so.
In Fernando et al.
(2017) the changing procedure uses evolutionary strategies to select pathways through the network and freeze older pathways.
In contrast, Yoon et al.
(2018) and Lee et al.
(2019) don’t modify systematically the architecture given new tasks but take into account the difﬁculty of the tasks and the relation between them.
More speciﬁcally, Yoon et al.
(2018) used group sparse regularization to dynamically expand the architecture when the accumulated knowledge alone cannot sufﬁciently explain the new task.
Similarly, Lee et al.
(2019) expands the number of experts under a Bayesian nonparametric framework, which determines the complexity of the model based on the data.
Our method keeps the same model from one task to another and does not ﬁt into the approaches based on dynamic architectures.
Meta-learning based methods.
Meta-learning refers to algorithms that learn how to learn across a set of related tasks.
Meta-learning-based CL approaches combine the ability of meta-learning to learn common knowledge across tasks with the capability of CL to continuously adapt to new tasks.
Integrating the new knowledge with the accumulated one and using this accumulation efﬁciently to facilitate rapid learning are the principal issues for meta-learning-based methods.
Jerfel et al.
(2019) developed a mixture of hierarchical models and extended the latent variable model to a nonparametric setting.
Caccia et al.
(2020) proposed a more general meta-learning-based continual learning scenario as well as a baseline model by pre-training and learning continuously.
KJ &
N Balasubramanian (2020) learned in the parameter meta-space with consolidation and generated model weights from the latent space for solving downstream tasks.
Although the idea of label optimization has been introduced in a transfer learning framework (Tiomoko et al., 2020), this paper is the ﬁrst to propose a line of label optimization in a CL context.
As a result, our study is not directly part of the aforementioned CL classiﬁcation methods but can be considered a new research line for CL.
2. S UPERVISED P RINCIPAL C OMPONENT A NALYSIS
Our continual learning approach is based on Supervised Principal Component Analysis.
Barshan et al.
(2011a) is one of the ﬁrst to propose SPCA, which is a generalization of principal component analysis (PCA) and aims at ﬁnding the principal components with maximum dependence on the response variables.
Ritchie et al.
(2019) presented a manifold optimization approach to SPCA that simultaneously solves the prediction and dimension reduction problems, which is general enough for both regression and classiﬁcation settings.
For a more detailed review, we refer to Ghojogh &
Crowley (2019) which explains PCA, SPCA, kernel PCA, and kernel SPCA.
The advantage of SPCA in a CL context is its ease of transferring knowledge from previous tasks to the next in addition to its simplicity of theoretical analysis, which is a major advantage over more complicated algorithms such as SVM, LDA, logistic regression, or neural networks.
2. O N LARGE DIMENSIONAL ANALYSIS OF MACHINE LEARNING ALGORITHMS
The ﬁeld of theoretical analysis of machine learning algorithms has become increasingly large with a signiﬁcant impact on the understanding and optimization of algorithms.
Different ﬁelds of research are explored, among which we ﬁnd approaches using the theory of random matrices (Mai & Couillet, 2021; Couillet & Benaych-Georges, 2016; Liao & Couillet, 2019;
Seddik et al., 2020; Tiomoko et al., 2020; Niyazi et al., 2021), approaches based on physical statistics (Cocco et al., 2018; Zdeborov ´ a & Krzakala, 2016; Malzahn & Opper, 2005; Agliari et al., 2020; Carleo et al., 2019; Agliari et al., 2020; Taheri et al., 2021), and other approaches (Convex Gaussian Min Max theorem notably (Thrampoulidis et al., 2014;
Deng et al., 2020))
These approaches generally have the same goal: to predict in advance the classiﬁcation error or to predict the mean square error in a regression framework to anticipate the behavior of the algorithm and improve it.
Our study is part of this wave of asymptotic analyses and is more speciﬁcally related to the approaches of the theory of random matrices.
Furthermore, using Bayesian and information theoretic arguments, Pentina & Lampert (2014; 2015); Doan et al.
(2021) establishes bounds for the classiﬁcation error in a continual learning framework.
These works, though, only provide loose bounds and orders of magnitude, which, while convenient for deciding whether an objective is impossible, provide no satisfying accurate evaluations of performance.
In this paper, we provide a precise and exact theoretical classiﬁcation error of Spca-based continual learning.
P RELIMINARIES 3. C ONTINUAL LEARNING FRAMEWORK
Let consider a m -class classiﬁcation problem in a d -dimensional space with training data of task t denoted X [ t ] =
[ X [ t ]1 , . . .
, X [ t ] m ] ∈ R d × n t where X [ t ] j
=
[ x ( j ) t 1 , . .
, x ( j ) tn tj ] ∈ R d × n tj are the n tj vectors of class j ∈ { 1 , . . .
, m } .
To each x ( j ) t(cid:96) ∈
R d is attached a corresponding “label” (or score) y
( j ) t(cid:96) ∈ R .
We denote in short y t =
[ y (1) t 1 , . . .
, y ( m ) tn t ] T ∈ R n t the vector of all labels of task t .
In a continual learning setting, when dealing with task t , the user only has access to the data of task t , i.e., X [ t ] and the associated label y t , and needs to predict label ˆ y for a test data x ∈ R d coming from an already seen task τ ∈ [ t ] .
If the task identity of the test data is known a priori, we refer the setting to the Task Incremental Learning (TIL) .
The unknown task label during inference scenario is divided into the Domain Incremental Learning (DIL) where all tasks share the same label space and
Class Incremental Learning (CIL) when it changes.
van de Ven & Tolias (2019) details more deeply the relationships between the different settings.
Continual learning requires knowledge to be transmitted, not data.
This is one of the main advantages of SPCA that we explain in the next section.
3. S PCA - BASED CONTINUAL LEARNING ALGORITHM
For simplicity, we will limit ourselves to the binary case for now.
Multi-class learning is handled by using binary-based approaches such as one-versus-one and one-versus-all which are explained in Section 4 and fully discussed in Appendix C. SPCA (Barshan et al., 2011b) extracts the principal components of the data that are most dependent on the target variable.
More speciﬁcally, in a binary classiﬁcation context, let the data matrix
X =
[ X [1] , . . .
, X [ t ] ] ∈ R d × n , with n = (cid:80) tτ = 1 n τ , be the collection of the training data of the t tasks already seen and their associated labels or scores y =
[ y T 1 , . . .
, y T t ] T ∈ R n , the decision function of SPCA is given for a new test data x by
[Equation]
Since x ( j ) t 1 , . . .
, x ( j ) tn t are i.i.d.
data vectors, we impose equal scores y ( j ) t 1 = . . .
=
y ( j ) tn tj denoted ˜ y
tj within every class j of task t .
As such, we may reduce the score vector y ∈ R n under the form
[Equation]
for ˜ y =
[˜ y 11 , . . . ,
˜
y t 2 ] T ∈ R 2 t .
This consists in assigning to each class of each task a unique and common label.
In a classical way, one class of all tasks will be assigned a score of − 1 , while the other class will be given a score of 1 , i.e., ˜ y =
[ − 1 , 1 , − 1 , 1 , . . . , − 1 , 1] T .
The class allocation is therefore performed based on the sign of the decision function g ( x ) .
We will see that the classical choice of labels leads to catastrophic forgetting and does not allow us to solve the stability-plasticity dilemma.
For this reason, we leave a free choice for the labels or scores ˜
y
and we will optimize them using the theory similarly as done in Tiomoko et al.
This optimization of labels can be intuitively seen as a weights of different classes with respect to their importance in the knowledge preservation.
This being said, Equation 1 can then be rewritten in a more convenient manner as
[Equation]
Therefore, when dealing with task t , the user just needs to have access to the empirical means ˆ µ τj and the label ˜
y τj for τ ≤ t , which allows an important memory saving compared to sending the raw data.
CL has the important challenge of building a ﬂexible optimization method that solves the catastrophic forgetting and the stability-plasticity dilemma.
To achieve this goal, we propose a theoretical analysis of the SPCA-based continual learning algorithm.
Precisely, we will characterize the theoretical classiﬁcation error of each task such that a ﬂexible optimization scheme can be performed based on the theoretical analysis.
To that end, we will make use of the following assumption on the data distribution.
Assumption 1 (Concentration of X ) For class C j , j ∈ { 1 , 2 } of task t , we assume that all vector x ( j ) t 1 , . . .
, x ( j ) tn tj ∈ C
j are i.i.d.
such that Cov( x ( j ) t 1 ) =
Σ tj and E [ x ( j ) t 1 ] = µ tj .
Moreover, we assume that there exist two constants C, c > 0 (independent of n, d ) such that, for any 1 -Lipschitz function f : R d → R ,
[Equation]
where m Z is a median of the random variable Z .
Assumption 1 notably encompasses the following scenarios: the columns of X are (a) independent Gaussian random vectors with identity covariance, (b) independent random vectors uniformly distributed on the R d sphere of radius √ d , and, most importantly, (c) any Lipschitz continuous transformation thereof, such as GAN as it has been recently theoretically shown in Seddik et al. An intuitive explanation of Assumption 1 is that the transformed random variables f ( x ) for any f : R d → R Lipschitz has a variance of order O (1) . In particular, it implies that it does not depend on the initial dimension d .
Furthermore, we place ourselves into the following large dimensional regime:
Assumption 2 (High-dimensional asymptotics) As n → ∞ , we consider the regime where d =
O ( n ) and assume d/n → c 0 > 0 .
Furthermore, we assume that n tj /n
→ c tj .
This assumption of the commensurable relationship between the number of samples and their dimension corresponds to a realistic regime and differs from classical asymptotic where the number of samples is often assumed to be exponentially larger than the feature size.
Note that this chosen asymptotic regime classical in Random Matrix Theory ﬁts most real-life applications and has been successfully applied in telecommunications (Couillet & Debbah, 2011), ﬁnance (Potters et al., 2005) and more recently in machine learning (Liao, 2019; Mai & Couillet, 2021;
Tiomoko et al., 2020).
Under Assumptions 1–2, Section 4 proposes a theoretical analysis of the classiﬁcation error of
SPCA-based continual learning algorithm.
T HEORETICAL ANALYSIS AND OPTIMIZATION FRAMEWORK 4. T HEORETICAL ANALYSIS OF THE S PCA - BASED CONTINUAL LEARNING
Theorem 1 Under Assumptions 1–2, for x a test data vector of class j in task τ ∈
[ t ] , following
Assumption 1 with E [ x ] = µ τj and Σ x = Σ τj , [Equation]
in distribution where,
[Equation]
where s =
[ s 11 , . . .
, s 2 t ] ∈ R 2 t , with s τj = 1 n tr( Σ τj Σ x ) ,
c
=
[ c 11 , . .
c t 2 ] .
D c and D s are the diagonal matrix with c and s as diagonal elements.
And M = [ µ 11 , µ 12 , . .
. , µ t 1 , µ t 2 ] ∈ R d × 2 t .
It is interesting to note that the theoretical high-dimensional performance of SPCA-based continual learning depends only on small data statistics (of at most size 2 t with t the number of tasks seen so far).
This is in line with many analyses generally conducted in the ﬁeld of large-dimensional analysis.
More speciﬁcally, in our case these sufﬁcient statistics are: 1) the correlation between the task means through M T M , 2) the correlation between the covariance matrices through the vector s and 3) the interaction between the means and covariance through M T Σ x M .
Note that these quantities incorporate the interaction between the tasks and the classes.
We would like to mention that these statistics are easily estimated using empirical means and empirical covariance matrices as explained in Remark 1 of the Appendix B. More importantly, performance depends on labels ˜
y we have chosen to leave as parameters.
Next section will discuss the optimization of these labels in order to avoid catastrophic forgetting and resolve the stability-plasticity dilemma.
Sectionillustrates this experimentally.
Since g ( x ) has a Gaussian limit centered around m τj , the asymptotic standard decision for x to be allocated to class 1 or class 2 for task τ is obtained by
[Equation]
for some threshold γ .
To minimize the classiﬁcation error, the optimal threshold γ (cid:63) for separating two Gaussians lies at the intersection between these two Gaussians and can be determined numerically as a function of m τj and σ τj .
The resulting classiﬁcation error rate for task τ is given as
[Equation]
with m τj and σ τj as deﬁned in Theorem 1 and where Q ( z ) = 1 √ 2 π (cid:82) z
−∞ e − t 2 / 2 dt is the cumulative distribution function of standard normal distribution.
As already mentioned, we defer the estimation of m τj and σ τj to Remark 1 in the Appendix B.
4. O PTIMIZATION FRAMEWORK
Having access a priori to the theoretical classiﬁcation error depending on the label
˜ y , one can minimize the theoretical classiﬁcation error as function of ˜ y .
Interestingly, for the speciﬁc case where the weights of the tasks are ω =
e [ t ] τ (which refers to optimizing the single task τ ), the optimal label has a closed form solution (see details of the derivation in Appendix B):
[Equation]
where ˜ y [ t ] τ (cid:63) denotes the optimal label for test task τ when t tasks are seen.
Theorem 2 ( On catastrophic forgetting ) Under Assumptions 1–2, ∀ t and τ ∈ [ t ] ,
[Equation]
where (cid:15) τ
(˜ y [ t ] τ (cid:63) ) is the classiﬁcation error rate of task τ after the model learned on task t .
Proof-sketch.
If we denote by ¯ ˜ y [ t +1] τ =
[˜ y
[ t ] τ (cid:63) T , 0 , 0] T ∈ R 2( t +1) , the vector ˜ y
[ t ] τ (cid:63) to which we add two-zeros for both classes of task t + 1 , we have: (cid:15) τ
(˜ y [ t ] τ (cid:63) ) =
(cid:15) τ (¯˜ y
[ t +1] τ ) ≥ (cid:15) τ (˜ y
[ t +1] τ (cid:63) ) .
The equality due to the fact that the addition of label 0 to both classes of a task doesn’t contribute to the decision function (see equation 2).
The inequality comes from that ˜ y
[ t +1] τ (cid:63) is the minimum vector of a convex problem over the space of 2( t + 1) -dimensional vectors.
Therefore, Theorem 2 ensures theoretically that catastrophic forgetting is prevented when adding more tasks.
This will be veriﬁed empirically in Section
In the case of a generic weight vector ω (to optimize the multi-tasks), the optimization of ˜ y is performed using Scipy optimization toolbox (Bressert, 2012) with the cost function given as L (˜ y )
= (cid:80) tτ
=1 ω τ (cid:15) τ
(˜ y ) , where the weights ω τ depending on the importance that user wants to put on task τ and (cid:15) τ
(˜ y ) is given by equation 4.
The optimization framework becomes ﬂexible, allowing the stability and plasticity problems to be traded off and this will be illustrated in the Section 5.
4. O N THE MULTI - CLASS EXTENSION
Input: Training samples of current task t i.e., X [ t ] =
[ X (1)[ t ] , . . . , X ( m ) [ t ] ] , X ( (cid:96) ) [ τ ] ∈ R p × n τ(cid:96) , test data x and empirical means and covariance matrices of previous tasks ˆ µ τj , ˆ Σ τj for τ < t . Output: Estimated class ˆ (cid:96) ∈ { 1 , . . .
{+, m } of x . for j = 
+}
to m do for j (cid:48) ∈ { 1 , . .
, m } \ { j (cid:48) } do
Estimate: M T M , M T Σ x M and s according to Remark 1.
Create optimal scores
˜ y
(cid:63) ( j (cid:48) , j ) =
arg min ˜ y (cid:80) τs
=1 ω s (cid:15) s (˜ y ) .
Compute classiﬁcation scores according to equation 2, deduce the predicted class c ( j, j (cid:48) )
=
j or c ( j, j (cid:48) ) = j (cid:48) based on the decision rule in equation 3. Update the mean matrix M and Σ ij by sending the empirical end for end for
[Equation]
[Equation]
[Equation]
Having an optimized framework for binary classiﬁer, our approach to multi class setting aims to resort on one-versus-one or alternatively on one-versus-all approach.
In a one-versus-one scheme, 12 m ( m − 1) binary classiﬁers are trained (one for each pair class j and class j (cid:48) , solving a binary classiﬁcation problem).
For each test sample, each binary classiﬁer decides on or votes for the more relevant class.
The test sample is then attributed to the class having the majority of votes.
Each binary classiﬁer is optimized by choosing appropriate labels and decision thresholds, thereby largely improving over the basal classiﬁer performance.
In one-versus-all approach applied to m classes consists to build m binary classiﬁers where for classiﬁer j ∈ [ m ] , the one class is class j and all other classes j (cid:48) (cid:54) =
j are gathered into one common class.
The ﬁnal decision is obtained by taking the argmin over the m classiﬁers.
An inherent problem of one-versus-all is the scales and alignments of the m binary classiﬁers to be compared are different.
We solve this problem thanks to the theoretical statistics of g ( x ) per classiﬁer depend on Theorem 1 in addition to the optimization of the basal classiﬁer.
This procedure is thoroughly discussed in the Appendix C. Our algorithm is summarized as Algorithm 1.
A PPLICATIONS 5. D ATASETS D ESCRIPTION
Through the experimental part, we will use 5 data sets described as follows (see more details in the
Appendix D):
Synthetic: Gaussian mixture model with two classes with means µ τj = ( − 1)
j µ τ with µ τ = β t µ + (cid:112) 1 − β 2 t µ ⊥ for β t a relatedness parameter and with µ = e [ p ] 1 , µ ⊥ =
e [ p ] p .
• Permuted MNIST: denoted PMNIST (Zenke et al., 2017), a variant of the MNIST dataset of handwritten digits where various permutations are used to model different tasks.
• Split MNIST: denoted SMNIST .
Which split the original MNIST-dataset into 5 tasks and each task contains two consecutive MNIST-digits (Lopez-Paz & Ranzato, 2017).
• Rotated MNIST: denoted RMNIST works in a similar manner to Permuted MNIST , but all the images are rotated to represent a variety of tasks (Hadsell et al., 2020).
• Split Fashion MNIST: denoted as SFMNIST .
Farquhar & Gal (2018) build the SFMNIST as CL benchmark which consists of 5 tasks, each task has two consecutive classes of Fashion MNIST dataset.
5. T HEORETICAL VERSUS EMPIRICAL PREDICTION
The purpose of this section is twofold.
Our ﬁrst goal is to demonstrate the robustness of the theoretical analysis when applied to synthetic and real data by comparing the theoretical and empirical predictions.
Secondly, we would like to highlight the signiﬁcant improvement brought by the optimization of labels and illustrate that catastrophic forgetting is avoided.
Figure 1 represents the empirical histogram of the decision function with the theoretical prediction for classical and optimized schemes.
It is noticeable that the concentration random vector assumption is quite realistic since the theoretical and empirical predictions are close to each other.
Next we illustrate how catastrophic forgetting is avoided.
To do so, we represent 3 examples in
Figure 2.
In the synthetic case we compare the average theoretical and empirical classiﬁcation accuracy both for the classical and optimized scheme as function of an increasing number of tasks.
Here the optimization assumes equal weight for all tasks encountered.
We can see a close ﬁt between theory and practice while remarking that the performance of OSCL never decreases as opposed of its counterpart non optimized (SPCA) which forgets knowledge as new tasks are encountered.
In the second and third picture (preformed on PMNIST and RMNIST ), we represent the trajectory of the accuracy for each seen tasks as the function of the task number we observed so far. And
Table 1 shows the average forgetting measure where the values are all negative.
We remark that by adding new tasks, we always improve the accuracy previously obtained, thus avoiding catastrophic forgetting .
5. O N THE FLEXIBITY OF THE FRAMEWORK
In this section, we investigate the ﬂexibility of the optimization framework by assigning different weights to different tasks.
Considering two-task binary classiﬁcation problem, Figure 3 shows the [Figure] [Figure]
[Table] classiﬁcation error of task 1 compared with task 2 . Each coordinates of the point represent the classiﬁcation errors of task 1 and 2 along with the weight ( ω 1 , ω 2 ) next to the point.
In the same ﬁgure, a red cross represents the classiﬁcation error of the classical label.
A task that is given more weight is prioritized by the algorithm over a task that is given less weight.
As a result of this ﬂexibility, certain tasks can be favored at the user’s discretion.
5. C OMPARISON WITH SOME STATE OF THE ART ALGORITHMS
In this section, we compare our method to a number of state-of-the-art approaches to show the performance and the low computational cost of our algorithm.
Four datasets and three scenarios (TIL, DIL and CIL) are used to fully validate the properties of our mothod. [Figure] [Figure]
[Table] [Table]
For SMNIST and PMNIST , the experimental protocols following KJ & N Balasubramanian (2020).
and the task number of RMNIST and SFMNIST are 10 and 5, respectively.
More details about experimental setting will be discussed in Appendix D. From Table 2 and Table 3, as we can see,
OSCL achieves competitive results with some baseline methods while being simpler in terms of complexity.
Since we use a linear classiﬁer we have extracted HOG features (Dalal & Triggs, 2005) for some images to fairly compare with deep neural network based approaches.
C ONCLUDING REMARKS
In this paper, a SPCA-based continual learning algorithm is analyzed from a high-dimensional perspective.
Using the theoretical analysis, we propose an optimization of the labels that avoids catastrophic forgetting .
In addition, the label optimization ﬂexibly weight tasks during learning, thereby resolving the stability-plasticity dilemma.
The method shows competitive results with the state of the art while being computationally inexpensive.
This work can be extended to the analysis of continual learning algorithm based on SVM and the integration of nonlinear kernels.
A BSTRACT
The appendix contains the main technical arguments omitted in the core of the article due to space limitation and is organized as follows.
Section 2 complements the related works with a diagram illustrating some other CL methods.
Section B derives the asymptotic classiﬁcation error of the SPCA-based CL algorithm.
To this end, Section B.1 proves the Gaussian distribution of the classiﬁcation score under concentrated random vector assumptions.
Section B.2 and B.3 computes respectively the ﬁrst and second order moment of the classiﬁcation score.
Section C complements the explanation on the multi-class extensions more precisely the one-versus-all approach.
Section D provides additional experiments and additional insights to the experiments derived in the main paper.
A C OMPLEMENTARY SECTION TO RELATED WORKS
To better summarize the section on the state of the art, we provide a rather didactic diagram in Figure 4 that we have constructed to classify the different methods of continual learning.
Our method could be classiﬁed in the theoretical part but with important algorithmic implications of efﬁciency, ﬂexibility and computation time.
B D ERIVATION OF THE TECHNICAL RESULTS
In this section, we perform the theoretical analysis of the SPCA-based continual learning.
In order to derive the theoretical classiﬁcation error, we need to understand the statistical behavior of the decision function g ( x ) =
y T X T x in particular its distribution and computing its ﬁrst and second order statistics.
To that end we recall the assumption on the data (Assumption 3) and the growth rate of the dimension (Assumption 4)
Assumption 3 (Concentration of X ) For class C j , j ∈ { 1 , 2 } of task t , we assume that all vector x ( j ) t 1 , . . .
, x ( j ) tn tj ∈ C
j are i.i.d.
such that Cov( x ( j ) t 1 ) =
Σ tj and E [ x ( j ) t 1 ] = µ tj .
Moreover, we assume that there exist two constants C, c > 0 (independent of n, d ) such that, for any 1 -Lipschitz function f : R d → R ,
[Equation]
where m Z is a median of the random variable Z .
Assumption 4 (High-dimensional asymptotics) As n → ∞ , we consider the regime where d =
O ( n ) and assume d/n → c 0 > 0 .
Furthermore, we assume that n tj /n
→ c tj and we will denote c =
[ c 11 , . .
c t 2 ] .
B. O N THE DISTRIBUTION OF THE DECISION SCORE .
In this section, the goal is to study the asymptotic behavior of g ( x ) | x ∈ C j , for i ∈ [ t ] .
Since conditionally on the training data X , y T X T x is expressed as the projection of the deterministic vector Xy on the gaussian random vector x , it follows that y T X T x is asymptotically Gaussian.
The proof under the concentrated random vector unfolds from the following proposition.
Theorem 3 (CLT for concentrated random vectors (Klartag, 2007; Fleury et al., 2007))
If x ∈ R p is a concentrated random vector with E [ x ] = 0 , E [ xx T ]
= I p with an observable diameter of order O (1) and σ be the uniform measure on the sphere S p − 1 ⊂ R p of radius 1 , then for any integer k , small compared to p , there exist two constants C, c and a set Θ ⊂ ( S p − 1 ) k
⊂ R p × k such that
[Equation]
[Equation]
√ pCe − c √ p and for all θ
= ( θ 1 , . .
, θ k ) ∈ Θ
[Equation]
where F 0 , 1 is the cumulative distribution function of the standard normal distribution N (0 , 1) .
Since g ( x ) is asymptotically Gaussian, we will focus on computing its ﬁrst and second order moments.
B. F IRST ORDER MOMENT OF THE DECISION SCORE .
Due to the independence between the training and test dataset X and x , the mean reads as
[Equation]
where D c stands for the diagonal matrix containing on its diagonal elements the element of vector c = [ n 11 /n, . .
. , n 2 t /n ] .
B. S ECOND ORDER MOMENT OF THE DECISION SCORE .
The variance of the decision score g ( x ) reads as [Equation] x )
[Equation]
Moreover we have that
[Equation]
Therefore
Var
[Equation]
where D T =  
Tr ( Σ 11 Σ x ) ...
[Equation]
We deduce the ﬁnal expression of the variance as
[Equation]
with s =
[ s 11 , . . .
, s 2 t ] ∈ R 2 t , with s τj = 1 n tr( Σ τj Σ x )
Remark 1 (Estimation of m τj and σ τj ) All the quantities deﬁned in Theorem are known a priori except the scalar products M T M , M T Σ x M and the vector s .
Note that 1 n tr( ˆ Σ τj ˆ Σ x ) is a consistent estimator for s τj with ˆ Σ τj and ˆ Σ x , the empirical covariance matrices.
The estimation of
M T M and M T Σ x M .For large Gaussian data, M T M can be effectively estimated as Tiomoko et al.
[Equation]
with q = 2( τ − 1) + j and q (cid:48)
= 2( τ (cid:48) − 1) + j (cid:48) different and X τj =
[ X τj ;1 , X τj ;2 ] an even-sized division of X τj .
B. D ERIVATION OF THE OPTIMAL LABEL
In the case of identical covariance for both classes i.e., Σ τ 1 = Σ τ 2 , the optimal threshold γ is given as
[Equation]
The associated classiﬁcation error is given by
[Equation]
Therefore
[Equation]
The previous optimization is known as the optimization of Rayleigh quotient (Li, 2015) which is convex and the optimal solution is given as
[Equation]
C O N THE MULTI - CLASS EXTENSION
The literature Bishop & Nasrabadi (2006) describes broad groups of approaches to deal with classiﬁcation with m > 2 classes.
We focus here on the most common method, namely the one-versus-all approach.
The complete optimization of one-versus-all being theoretically heavy to handle and demanding prior knowledge on the decision output statistics, the method inherently suffers from sometimes severe practical limitations; these are partly tackled here exploiting the large dimensional analysis performed in this article.
In an one-versus-one setting, 12 m ( m − 1) binary classiﬁers are trained (one for each pair j, j (cid:48) of classes, solving a binary classiﬁcation).
For each test sample, each binary classiﬁer decides on – or “votes” for – the more relevant class.
The test sample is then attributed to the class having the majority of votes.
Although the number of binary classiﬁers is greater than in the one-versus-all approach, the training process for each classiﬁer is faster since the training database is much smaller for each binary classiﬁer.
Besides, the method is more robust to class imbalances (since only pairwise comparisons are made) but suffers from an undecidability limitation in the case of equal numbers of majority votes for two or more classes.
In the one-versus-all method, focusing on Task t , m individual binary classiﬁers, indexed by (cid:96) = 1 , . . .
, m , are trained, each of them separating Class C t(cid:96) from the other m − 1 classes C t(cid:96) (cid:48) , (cid:96) (cid:48) (cid:54) =
(cid:96) .
Each test sample is then allocated to the class index corresponding to the classiﬁer reaching the highest of the m classiﬁer scores.
Although quite used in practice, the approach ﬁrst suffers a severe unbalanced data bias when using binary ( ± 1 ) labels as the set of negative labels in each binary classiﬁcation is on average m − 1 times larger than the set of positive labels, and also suffers a center-scale issue when ultimately comparing the outputs of the m decision functions, the average locations and ranges of which may greatly differ; these issues lead to undesirable effects, as reported in (Bishop & Nasrabadi, 2006, section 7.1.3)).
These problems are here simultaneously addressed: speciﬁcally, having access to the large dimensional statistics of the classiﬁcation scores allows us to appropriately center and scale the scores.
Each centered-scaled binary classiﬁer is then further optimized by appropriately selecting the class labels (different from ± 1 ) so to minimize the resulting classiﬁcation error.
See Figure 5 for a convenient illustration of the improvement induced by this centering-scaling and label optimization approach.
C. O NE - VERSUS - ALL MULTI - CLASS OPTIMIZATION
For each target task t , in a one-to-all approach, m SPCA-based CL binary classiﬁcations are solved with the target class C τ(cid:96) (renamed “class C (cid:96)τ 1 ”), against all other C (cid:96)τ 2 classes (combined into a single “ C (cid:96) τ 2 class”).
Calling g
( (cid:96) )
x the output of the classiﬁer (cid:96) for a new datum x , the class allocation decision is traditionally based on the largest among all scores g (1) x , . . .
, g ( m ) x ,t .
However, this presumes that the distribution of the scores g (1) x when x ∈ C 1 , g (2) x when x ∈ C 2 , etc., more or less have the same statistical mean and variance.
This is not the case in general, as depicted in the ﬁrst column of Figure 5, where data from class C 1 are more likely to be allocated to class C 3 (compare the red curves).[-− 1 . 5 − 1 − 0 . 
-]
[-5 − 1 − 0 . 
-]
By providing an accurate estimate of the distribution of the scores g ( (cid:96) ) x for all (cid:96) ’s and all genuine classes of x , Theorem 1 of the main article allows us to predict the various positions of the Gaussian curves in Figure 5.
In particular, it is possible, for each binary classiﬁer (cid:96) to center and scale g ( (cid:96) )
x when x ∈ C (cid:96) .
This operation averts the centering and scaling biases depicted in the ﬁrst column of
Figure 5: the result of the center-scale operation appears in the second column of Figure 5.
This ﬁrst improvement step simpliﬁes the algorithm which now boils down to determining the index of the largest g ( (cid:96) )
x − m ( (cid:96) )
τ 1 , (cid:96) ∈ { 1 , . .
, m } , while limiting the risks induced by the center-scale biases.
This being said, our theoretical analysis further allows to adapt the input labels ˜
y
( (cid:96) ) in such a way to optimize the expected output.
Ideally, assuming x genuinely belongs to class C (cid:96) , one may aim to increase the distance between the output score g ( (cid:96) )
x and the other output scores g ( (cid:96) (cid:48) )
x for (cid:96) (cid:48) (cid:54) = (cid:96) .
This is performed by maximizing the distance between the output score g ( (cid:96) )
x for x ∈ C (cid:96) and the scores g ( (cid:96) )
x for x (cid:54)∈ C (cid:96) .
By “mechanically” pushing away all wrong decisions, this ensures that, when x ∈ C (cid:96) , g ( (cid:96) ) x is greater than g ( (cid:96) (cid:48) )
x for (cid:96) (cid:48) (cid:54) = (cid:96) .
This is visually seen in the third column of Figure 5, where the distances between the rightmost Gaussians and the other two is increased when compared to the second column, and we retrieve the desired behavior.
Algorithm 1 and C.1 summarizes the
OSCL for one-versus one one all schemes Remark 2 (On the complexity of OSCL ) The complexity of OSCL depends on the multi-class extension chosen. OSCL may be sequentially described as performing 12 m ( m − 1) times (for the oneversus-one) and m times (for the one-versus-all) the following procedure in 1) computing the decision score g ( x ) as per equation 3, 2) estimation of M (cid:62) M , M (cid:62) Σ x M and s , 3) optimization of ˜ y .
The complexity of computing g ( x ) is a matrix-vector multiplication, yielding complexity O ( n 2 ) . The computation of M (cid:62) M (as well as M (cid:62) Σ x M and the vector s ) is of complexity O ( dn + d ) (estimation + product). The optimization of the labels being performed using the BFGS algorithm is of complexity O
( t 3 ) , t being the number of tasks. The overall complexity is O ( m 2 T 3 ) + O ( n 2 m 2 ) for the one-versus-one scheme and O ( mT 3 ) + O ( n 2 m ) for the one-versus-all.
Note that competitors (using for most of them deep neural networks) can difﬁculty computes algorithmic complexity since it depends on the architecture chosen and depends on the internal optimization of the deep neural networks. But as conﬁrmed with the running time (both for training and inference), our method achieves a lower computational cost. − 1 . 5 − 1 − 0 . {+
+}
Classical g t ( x ) | x[-. for j = 
 to m do for j (cid:48)-] ∈ C 1 g t ( x ) | x ∈ Cg t ( x ) | x ∈ C−−  40 .0 .0 .0 .
Scaled scores {+−−  40 .0 .0 .0 . Optimized labels − 1 . 5 − 1 − 0 . 
+}
−−  40 .0 .0 .0 .−−
.−
5 − 1 − 0 .
−−  40 .0 .0 .0 .−−
often be misclassiﬁed
C
“optimized of
Input: Training
.e
, test empirical covariance
Output for j = [-
-]1 to m do
Estimate: M T M , M T Σ x M and s according to Remark 1.
Create optimal scores
˜ y
(cid:63) ( j ) =
arg min ˜ y (cid:80) ts =1 ω s (cid:15) s (˜ y ) .
Compute classiﬁcation scores g ( x , (cid:96) )
according to equation 2.
Compute the theoretical statistics m ( (cid:96) ) tj and σ ( (cid:96) ) tj as per Theorem 1, center and scale the decision function as g ( x ,(cid:96) )
− m ( (cid:96) ) tj σ ( (cid:96) ) tj .
Update the mean matrix M and the covariance matrices Σ ij by sending the empirical means ˆ µ tj and covariance matrices ˆ Σ tj for use in the next task) end for
Output: ˆ (cid:96) =
arg max (cid:96) ∈{ 1 ,...,m } g ( x ; (cid:96) ) .
D E XPERIMENTAL PART D. T HE DESCRIPTION OF DATA SET AND EXPERIMENTAL SET UP
The Permuted MNIST dataset has 70,000 images of handwritten digits from 0 to 9, with 60, being used for training and 10,000 for testing.
Different from the original MNIST dataset, this dataset uses random permutations of pixels as inputs for each of the ten tasks.
The Spilt MNIST This benchmark is based on 5 subsets of consecutive digits within the MNIST training data set.
Learning to distinguish between two consecutive digits from 0 to 10 is the goal of the ﬁve tasks.
In the Rotated MNIST , the digits were rotated by an angle generated uniformly between 0 and 180 degrees and data for different tasks with different rotation angles.
Based on this, the rotation angle and the factors of variation already included in MNIST are the factors of variation.
The Fashion MNIST dataset has the same size as MNIST, but is based on different (and more challenging) 10 classes.
Similar to the Split MNIST , we will split the the Fashion MNIST into 5 tasks, and the ﬁve binary classiﬁcation tasks are: T-shirt/Trouser, Pullover/Dress, Coat/Sandals,
Shirt/Sneaker, and Bag/Ankle boots.
Across all MNIST variants, 1000 samples were used as training data.
For Rotated MNIST and
Permuted MNIST , 10 tasks were generated.
And regrading to Split MNIST and Split Fashion
MNIST , 5 tasks are generated, as in KJ & N Balasubramanian (2020);
Sokar et al.
Each experiment was run ﬁve times randomly to obtain the results.
For a fair comparison, given that we are a linear model, we used HOG (Dalal & Triggs, 2005) to extract features for Rotated MNIST ,
Split MNIST , and Split Fashion MNIST , and raw data for Permuted MNIST (without any feature extraction).
D. E VALUATION M ETRICS
We evaluate our algorithm by average accuracy A across tasks.
A = 1 T (cid:80) Tt =
1 A t , where A t = 1 t (cid:80) tτ a t,τ and a t,τ is the accuracy of task τ after the model is trained on t th task
[Table]
D. S UPPLEMENTARY EXPERIMENTS
[Table] [Table]