Continual learning: a feature extraction formalization, an efﬁcient algorithm, and barriers
Abstract
Continual learning is an emerging paradigm in machine learning, wherein a model is exposed in an online fashion to data from multiple different distributions (i.e. environments), and is expected to adapt to the distribution change.
Precisely, the goal is to perform well in the new environment, while simultaneously retaining the performance on the previous environments (i.e. avoid “catastrophic forgetting”).
While this setup has enjoyed a lot of attention in the applied community, there hasn’t be theoretical work that even formalizes the desired guarantees.
In this paper, we propose a framework for continual learning through the framework of feature extraction—namely, one in which features, as well as a classiﬁer, are being trained with each environment.
When the features are linear, we design an efﬁcient gradient-based algorithm DPGrad , that is guaranteed to perform well on the current environment, as well as avoid catastrophic forgetting.
In the general case, when the features are non-linear, we show such an algorithm cannot exist, whether efﬁcient or not.
Introduction
In the last few years, there has been an increasingly large focus in the modern machine learning community on settings which go beyond iid data .
This has resulted in the proliferation of new concepts and settings such as out-of-distribution generalization [16], domain generalization [3], multi-task learning [41], continual learning [25] and etc.
Continual learning , which is the focus of this paper concerns learning through a sequence of environments, with the hope of retaining old knowledge while adapting to new environments.
Unfortunately, despite a lot of interest in the applied community—as evidenced by a multitude of
NeurIPS and ICML workshops [26, 12, 30]—approaches with formal theoretical guarantees are few and far between.
The main reason, similar encountered as its cousin ﬁelds like out-of-distribution generalization or multi-tasks learning, usually come with some “intuitive” desiderata — but no formal deﬁnitions.
What’s worse, it’s often times clear that without strong data assumptions—the problem is woefully ill-deﬁned.
The intuitive desiderata the continual learning community has settled on is that the setting involves cases where an algorithm is exposed (in an online fashion) to data sequentially coming from different distributions (typically called “environments”, inspired from a robot/agent interacting with different environments).
Moreover, the goal is to keep the size of the model being trained ﬁxed, and make sure the model performs well on the current environment while simultaneously maintaining a good performance in the previously seen environments.
In continual learning parlance, this is termed “resistance to catastrophic forgetting”.
It is clear that some of the above desiderata are shared with well-studied learning theory settings (e.g. online learning, lifelong learning), while some aspects differ.
For example, in online learning, we don’t care about catastrophic forgetting (or we only do so in some averaged sense); in lifelong learning, it’s not necessary to keep the size of the model ﬁxed.
It is also clear that absent
some assumptions on the data and the model being trained, these desiderata cannot possibly be satisﬁed: why would there even exist a model of some ﬁxed size that performs well on both past environments, and current ones — let alone one that gets updated in an online fashion.
A feature-extraction formalization of continual learning Our paper formalizes a setting for continual learning through the lens of feature extraction : the model maintains a ﬁxed number of (trainable) features, as well as a linear classiﬁer on top of said features.
The features are updated for every new environment, with the objective that the features are such that a good linear classiﬁer exists for the new environment, while the previously trained linear classiﬁers (on the updated features) are still good for the past environments.
The reason the linear classiﬁers from previous rounds are not allowed to be updated is storage efﬁciency: in order to tune the prompts, one needs to store the training data from previous tasks, this would bring a storage overhead and potentially privacy concerns.
This is a very common approach in practice—examples of this are systems involving large amounts of data of a streaming nature (e.g. Google searches, Youtube, a robotic agent interacting with a continual stream of environments), and it be would prohibitive to store it for later ﬁne tuning.
The number of features is kept ﬁxed for the same reason: if we are to expand with new features for every new environment, the model size (and hence storage requirements) would grow.
We prove two main results for our setting.
When the features are a linear function of the input data, and a good set of features exist, we design an efﬁcient algorithm , named doubly projected gradient descent, or DPGrad , that has a good accuracy on all environments, and resists catastrophic forgetting.
Our algorithm, while being novel, bears some resemblance to a class of projection-based algorithms used in practice [11, 5] – we hope the theoretical analysis can shed insight onto large scale continual learning.
When the features are allowed to be a non-
linear function of the input
, we show that continual learning is not possible—in general.
Namely, we construct an instance for which even if a good set of features exists, the online nature of the setting, as well as the fact that the linear classiﬁers for past environments are not allowed to be updated, makes it possible for the algorithm to “commit” to linear classiﬁers, such that either catastrophic forgetting, or poor performance on the current environment has to occur.
Our results 2.1 Problem formulation
In a continual learning problem, the learner has sequential access to k environments.
In the i -th ( i 2 [ k ] ) environment, the data is drawn i.i.d.
from the underlying distribution D i over R d ⇥ R , denoted as ( x, y )
⇠ D i , where x 2 R d is the input and y 2 R is the label.
Motivated by the empirical success of representation learning [2, 9], we formulate the continual learning problem through the feature extraction view: The learner is required to learn a common feature mapping (also known as representation function)
R : R d !
R r that maps the input data x 2 R d to a low dimensional representation R ( x ) 2 R r ( r ⌧ d ), along with a sequence of task-dependent linear classiﬁers (also known as linear prompts) v 1 , . .
, v k 2 R r on top of the representation.
Precisely, the prediction of the i -th environment is made by f ( x ) =
h v i , R ( x ) i .
As this is the ﬁrst-cut study, we focus on the realizable and the proper learning setting.
1 That is, we assume the existence of a feature mapping R in the function class H (which is known in advance) and a sequence of linear predictor v 1 , . .
, v k such that for any i 2 [ k ] and any data ( x, y )
⇠ D i , y =
h
v i , R ( x ) i (realizable).
The learner is required to output a function R that belongs to the hypothesis class H (i.e. the learner is proper).
Remark 2.1 (Known environment identity) .
Our model requires the knowledge of environment identity at test time, and thus can be classiﬁed into the category of incremental task learning.
We note there is also empirical research focusing on unknown environment identity, which would be an interesting direction for future work (See Section 7).
The guarantee that we wish our learning algorithm to obtain is as follows:
Deﬁnition 2.2 (Goal of Continual Learning) .
Let d, k, r 2 N , r ⌧ d, k , ✏ 2 (0 , 1 / 2) .
Let H be a function class in which the feature mappings from R d to R r lie.
The continual learning problem involves k environments D 1 , . . .
, D k .
We assume there exists a function R ?
2 H and a sequence of linear classiﬁers v ? 1 , . .
. , v ?
k 2 R r such that for any ( x, y )
⇠ D i ( i 2 [ k ] ), the label satisﬁes
y = h
[Equation] [Equation]
The continual learner has sequential access to environments D 1 , . . .
, D k , as well as access to arbitrarily many samples per environment 2 .
The goal is to learn a representation function R 2 H and a sequence of linear prompts v 1 , . .
, v k 2 R r that achieve a good accuracy on the current task and do not suffer from catastrophic forgetting.
Formally, in the i -th environment ( i 2 [ k ] ), the learner optimizes the feature mapping R and the linear classiﬁer v i (without changing v 1 , . .
. , v i  1 ) and aims to satisfy
• Avoid catastrophic forgetting : During the execution of the i -th task, the algorithm guarantees that
[Equation]
• Good accuracy on the current task: At the end of i -th task, the algorithm guarantees that
[Equation]
For linear feature mapping, the representation function can be written in a linear form R ( x ) =
U > x for some U 2 R d ⇥ r , and it implies the i -th environment is generated by a linear model.
That is, deﬁning w i =
Uv i 2 R d , one can write y
=
h
v i , U > x i
=
h w i , x i .
Remark 2.3 (The beneﬁt of continual learning with linear feature) .
Note, for linear features, it’s in principle possible to just learn a sequence of linear classiﬁers w 1 , . . .
, w k 2 R d separately— without learning a low-dimensional featurizer.
Choosing an r -dimensional featurizer confers memory efﬁciency ( O ( kr + dr ) vs. O ( dk ) ) and sample efﬁciency ( O ( r ) vs. O ( d ) samples per task in the asymptotic regime k ! 1 ).
Furthermore, the linear case is a sandbox that can be mathematically analyzed and can generate insights for the nonlinear case as well.
2.2 DPGrad: Efﬁcient gradient based method for linear features
For the case of linear features, we propose an efﬁcient algorithm which we term DPGrad (pseudocode in Algorithm 1), which is an efﬁcient gradient based method and provably learns the representation while avoids catastrophic forgetting.
Towards stating the result, we make a few technical assumptions.
Assumption 2.4 (Distribution assumption) .
For any i 2 [ k ] , we assume D
i has zero means and it is in isotropic position, that is, E x ⇠ D i [ x ]
= ~ 0 and E x ⇠ D i [ xx > ] = I .
Remark 2.5.
This assumption is largely for convenience.
In fact, one can replace the isotropic condition with a general bounded covariance assumption, our algorithm still can work with extra preprocessing step, and the sample complexity scales with the condition number of covariance matrix.
Assumption 2.6 (Range assumption) .
For any i 2 [ k ] , w i has bounded norm, i.e., k w i k 2  D .
Assumption 2.7 (Signal assumption) .
For any i 2 [ k ] , let W i
=
span ( w 1 , . .
. , w i ) , W i, ?
be the space perpendicular to W i and P W i , P W i, ?
be the projection operator.
We assume either w i belongs to
W i  1 or it has non-negligible component orthogonal to W i  1 , i.e., k P W i  1 , ?
w i k
2 2 { 0 } [ [1 /D, D ] .
Assumption 2.8 (Bit complexity assumption) .
Each coordinate of w i is a multiple of ⌫ > 0 .
Remark 2.9.
The Range and Signal assumptions are standard in the statistical learning literature.
The former ensures an upper bound on k w i k 2 and the later ensures that a new task provides enough “signal” for new features.
They are used to set up learning rate and number of gradient iterations.
Remark 2.10.
The Bit complexity assumption states that w i can be described with a ﬁnite number of bits, and is mostly for convenience — namely so we can argue we exactly recover w i —which makes calculations involving projections of features learned in the past cleaner.
Since the number of gradient iterations only depends logarithmically on ⌫ , one can relax the bit complexity restriction to only approximately recovering the ground truth features up to a polynomially small (in d, k, D, ✏ ) precision.
Our argument can still go through at the cost of some additional error analysis.
The main result is then as follows:
Theorem 2.11 (Continual learning with linear features) .
Let k, d, r 2 N , r ⌧ k, d , ✏ 2 (0 , 1 / 2) .
When the features are a linear function over the input data, under Assumption 2.4 and Assumption 2.62.8, with high probability, DPGrad provably achieves good loss and avoids catastrophic forgetting.
In particular, during the execution of i -th environment, DPGrad always ensures
[Equation]
and at the end of i -th environment, DPGrad ensures
[Equation]
2.3 Fundamental obstructions for non-linear features
The continual learning setup with non-linear features turns out to be much more difﬁcult — even without computational constraints.
Our result rules out the existence of a proper continual learner, even when all environment distributions are uniform and the representation function is realizable by a two-layer convolutional neural network.
Theorem 2.12 (Barrier for Continual learning with non-linear feature) .
Let k, r  2 , d  3 .
There exists a class of non-linear feature mappings and a sequence of environments, such that there is no (proper) continual learning algorithm that can guarantee to achieve less than 1 1000 -error over all environments with probability at least 1 / 2 ,under the feature extraction formalization of Deﬁnition 2.2.
Related work
Continual learning in practice The study of continual learning (or lifelong learning) dates back to the work of [36] and it receives a surge of research interest over recent years [15, 19, 11, 5, 14, 31, 34, 17, 29, 39, 18].
A central challenge in the ﬁeld is to avoid catastrophic forgetting [24, 23], which the work of [15] observed happened for gradient-based training of neural networks.
While there is a large amount of empirical work, we’ll brieﬂy summarize the dominant approaches.
The regularization based approach alleviates catastrophic forgetting by posing constraints on the update of the neural weights.
The elastic weight consolidation (EWC) approach [19] adds weighted ` 2 regularization to the objective function that penalizes the movement of neural weights.
The orthogonal gradient descent (OGD) algorithm from [11, 5] enforces the gradient update being orthogonal to update direction (by viewing the gradients as a high dimensional vector).
The memory replay approach restores data from previous tasks and alleviates catastrophic forgetting by rehearsing in the later tasks.
[31] introduces experience replay to continual learning.
[14] trains a deep generative model (a.k.a.
GAN) to simulate past dataset for future use.
The dynamic architecture approach dynamically adjusts the neural network architecture to incorporate new knowledge and avoid forgetting.
The progressive neural network [34] blocks changes to the existing network and expands the architecture by allocating a new subnet to be trained with the new information.
We refer the interested reader to more complete surveys [25, 7].
Continual learning in theory In comparison to the vast empirical literature, theoretical works are comparatively few.
The work of [6] characterize the memory requirement of continual learning, when the environment identity is unknown.
The works [35, 27, 1, 4] provide sample complexity guarantees on lifelong learning.
Their approaches can be categorized roughly into the duplicate and ﬁne-tuning paradigm: The algorithm maintains a weighted combination over a family of representation functions and the focus is on the sample complexity guarantee.
By contrast, we focus on the feature extraction paradigm and learn linear prompts on top of a single representation function.
Both the duplicate-andﬁne-tuning and the feature extraction paradigm have been extensively investigated in the literature, detailed discussions can be found at [7] and we provide a brief comparison.
From an algorithmic perspective, learning a weighted combination over a family of representation functions (i.e. the duplicate and ﬁne-tuning) is much easier, as one can always initiates a new representation function for a new task.
The algorithmic convenience allows previous literature focus more on the generalization and sample complexity guarantee, culminating with the recent work of [4].
We note again that learning a single representation function and task speciﬁc linear prompts is much more challenging, but has practical beneﬁts, e.g. memory efﬁciency.
For example, in the applications of NLP, the basic representation function (e.g. BERT [9]) is already overparameterized and contains billions of parameters.
It is then formidable to maintain a large amount of the basic models and learn a linear combination over them.
We mention several more works that are morally related in Appendix A.
Continual learning with linear feature
We restate our main result for linear feature mapping.
Theorem 2.11 (Continual learning with linear features) .
Let k, d, r 2 N , r ⌧ k, d , ✏ 2 (0 , 1 / 2) .
When the features are a linear function over the input data, under Assumption 2.4 and Assumption 2.62.8, with high probability, DPGrad provably achieves good loss and avoids catastrophic forgetting.
In particular, during the execution of i -th environment, DPGrad always ensures
[Equation]
and at the end of i -th environment, DPGrad ensures
[Equation]
4.1 Algorithm
A complete and formal description of DPGrad is presented in Algorithm 1.
DPGrad simultaneously updates the matrix of features U , as well as the linear classiﬁer v i using gradient descent—with the restriction that the update of U only occurs along directions that are orthogonal to the column and row span of the previous feature matrix.
Intuitively, one wishes the projection guarantees that existing features that have been learned are not erased or interfered by new environments.
Due to the quadratic nature of the loss, and the appearance of “cross-terms”, this turns out to require both column and row orthogonality, and interestingly deviates from the practically common OGD method [11, 5].
In more detail, at the beginning of the i -th ( i 2 [ k ] ) environment, DPGrad adds Gaussian noise to the feature matrix U and the linear classiﬁer v i , to generate a good initialization for U and v i .
Subsequently, we perform gradient descent to both the feature mapping matrix U and linear classiﬁer v i —except U is only updated along orthogonal directions w.r.t.
the column span and the row span.
At the end of each environment, DPGrad has a post-processing step to recover the ground truth w i by rounding each entry of Uv i to the nearest multiple of ⌫ , 3 and then update the column and row span if the orthogonal component is non-negligible.
The reason for the later step is that we only need to preserve row space when encountering new features.
Parameters
We use  to denote the initialization scale, ⌘ to denote the learning rate, and T to denote the number of iterations for each task.
These are all polynomially small parameters, whose scaling is roughly D, d, k ⌧   1 ⌧ ⌘  1 < T .
Notation We write [ n ] = { 1 , 2 , . . . , n } , [ n 1 : n 2 ] =
{ n 1 , . . . , n 2 } .
We use rand ( n 1 , n 2 ) 2 R n 1 ⇥ nto denote a size n 1 ⇥ n 2 matrix whose entries are draw from random Gaussian N (0 , 1) .
For each i 2 [ k ] , t 2 [0 : T ] , denote U i,t to be the feature matrix in the t -th iteration of the i -th environment (after performing the gradient update), denote v i,t similarly.
DPGrad includes a projection step at the end of i -th environment, we use U i, end to denote the feature matrix after this projection.
We use
W i (resp.
V i ) to denote the column (resp. row) space maintained at the end of i -th environment.
Let
W ?
✓ R n be the subspace orthogonal to W and deﬁne V ?
similarly.
Let P W , P V , P W ? , P V ?
be the projection onto W , V , W ? , V ? separately.
1: W ; , V ; , U .
U 2 R d ⇥ r 2:  e
O ( ✏ d 2 kD 4 ) , ⌘
O (  3 k 2 D 5 ) , T
O ( D ⌘ log Dkd ✏⌫ )
+ O ( D ⌘ log k  ) 3: for i = 1 , . . .
, k do 4:
U init  · P W ? rand ( d, r ) P V ? , v i  · rand ( r ) .
U init 2 R d ⇥ r , v i 2 R r 5:
U
U + U init 6: for t = 1 , . . .
, T do 7: r U
E ( x,y )
⇠ D i [ x ( x > Uv i  y ) v > i ] , r
v i
E ( x,y )
⇠ D i [ U > x ( x > Uv i  y )] 8:
U =
U  ⌘ P W ?
r U P V ?
9: v i
= v i  ⌘ r
v i 10: end for 11: end for 12: b w i
Round ⌫ ( Uv i ) .
Round to the nearest multiple of ⌫ , b w i 2 R d 13: if k P W ?
b w i k 2  1 /D
then W span ( W [ b w i ) , V span ( V [ v i ) 14: U
P W UP V
4.2 Analysis
We sketch the analysis of DPGrad and prove Theorem 2.11.
Due to space limitation, the detailed proof is deferred to Appendix B.
The proof proceeds in the following four steps:
The ﬁrst step, presented in Section 4.2.1, reduces continual learning to a problem of continual matrix factorization and it allows us to focus on a more algebraically friendly objective function.
We then present some basic linear-algebraic facts to decompose the feature mapping matrix U , its gradient, and the loss into orthogonal components.
The orthogonality of gradient update allows us to decouple the process of leveraging the existing features and the process of learning a new feature , as reﬂected in the loss terms and gradient update rules.
See Section 4.2.2 for details.
In Section 4.2.3, we zoom into one single environment, and prove DPGrad provably converges to a global optimum, assuming the feature matrix U from previous environment is well conditioned.
This step contains the major bulk of our analysis: The objective function of continual matrix factorization is non-convex, and no regularization or spectral initialization used.
(We cannot re-initialize, lest we destroy progress from prior environments.)
Finally, in Section 4.2.4, we inductively prove that DPGrad converges and the feature matrix is always well-conditioned.
This wraps up the entire proof.
4.2.1 Reduction
We ﬁrst recall the formal statement of the problem of continual matrix factorization.
Deﬁnition 4.1 (Continual matrix factorization) .
Let d, k, r 2 N , r ⌧ d, k , ✏ > 0 .
Let W =
[ w 1 , . . . ,
w k ]
= U ?
( V ? )
> 2 R d ⇥ k , where U ?
2 R d ⇥ r , V ? 2 R k ⇥ r .
In an continual matrix factorization problem, the algorithm receives w i 2 R d in the i -th step, and it is required to maintain a matrix
U 2 R d ⇥ r and output a vector
v
i 2 R r such that
[Equation]
and
[Equation]
The key observation is that running DPGrad on the original continual learning objective (2) implicitly optimizes the continual matrix factorization objective (3) (Lemma 4.2).
Moreover, an ✏ -approximate solution of continual matrix factorization is also an ✏ -approximate solution of continual learning (Lemma 4.3).
Lemma 4.2 (Gradient equivalence) .
Under Assumption 2.4, for any i 2 [ k ] , the gradient update of objective (2) equals the gradient update of objective (3) .
Lemma 4.3 (Objective equivalence) .
For any w 1 , . . .
, w k 2 R d , U 2 R d ⇥ r and v 1 , . .
, v k 2 R r , suppose
[Equation]
[Equation]
 ✏ holds for all i 2 [ k ] , then L ( U, v i ) =
12 E ( x,y )
⇠ D i ( x > Uv i  [Equation]
Combining the above observations, it sufﬁces to analyse DPGrad for continual matrix factorization and prove Eq.
(3) and Eq.
4.2.2 Decomposition
We ﬁrst provide some basic linear algebraic facts about orthogonal decompositions.
For any i 2 [ k ] , we decompose U i , v i , w i along W i  1 , W i  1 , ? , V i  1 and V i  1 , ? .
Let w i =
w i,A + w i,B where w i,A 2 W i  1 and w i,B 2 W i  1 , ? .
Note this decomposition is unique.
We focus on the case that k w i,B k 2 2 [1 /D, D ] in the following statements, and the case of k w i,B k 2 = 0 carries over easily.
(These are the only two cases, per Assumption 2.7).
Similarly, let U i
=
U i,A
+ U i,B , where each column of U i,
A lies W i  1 and each column of w i,B lies in W i  1 , ? .
(Note, again, U i,A and U i,B are unique.)
We further write U i,B = w i,B x > i
+ U
i, 2 , where the columns of U i, 2 lie in W i  1 , ?
\{ w i,B } .
Finally, denote v i
=
v i, 1 + v i, 2 with v
i, 1 2 V i  1 and v i, 2 2 V i  1 , ? .
We summarize decompositions mentioned above, with a few additional observations, in the lemma below:
Lemma 4.4 (Orthogonal decomposition) .
For any i 2 [ k ] and any t 2 [0 : T ] , there exists an unique decomposition of U i,t , w i and v i,t of the form
U i,t
=
U
i,A, 0 + U i,B,t ,
[Equation]
Here we use column ( A ) , row ( A ) to denote the column and row space of matrix A , and column ( A )
W if the column space of A is a subspace of W .
Since U i,A,t remains unchanged for t =
[0 : T ] , we abbreviate it as U i,A hereafter.
We next provide the exact gradient update of each component under loss function b
L ( U i , v i ) = 12 k U i v i  w i k 22 and orthogonal projection.
Lemma 4.5 (Gradient formula) .
For any i 2 [ k ] , the gradient update (after projection) obeys the relations (1) r x i ( b L )
= v i, 2 ( x > i v i, 2  1) ; (2) r U 2 ,i ( b L ) =
U i, 2 v i, 2 v > i, 2 ; (3) r
v i
[Equation]
[Equation]
U > i,A U i,
A v i, 1  U >
i,A w i,A and (4) r v i, 2 (
[Equation]
We perform a similar decomposition to the loss function.
Lemma 4.6 (Loss formula) .
For any i 2 [ k ] , t 2 [ T ] , we have
[Equation]
Decoupling existing features from “new” features We now offer some intuitive explanation for the decomposition.
The ﬁrst loss term in Eq.
(5) quantiﬁes the error with already learned features.
That is, the matrix U i,A stores existing features that have been learned, and it remains unchanged during the execution of the i -th environment; it remains to optimize v i, 1 ,t such that U i,
A v
i, 1 ,t matches w i,A .
The second and last loss term quantify the loss on a new feature, where w i,B is the new feature component, and the matrix U i, 2 ,t can be thought of as random noise.
Intuitively, one should hope x > i,t v i, 2 ,t = 1 and this matches the new component of w i,B .
At the same time, one hopes U i, 2 ,t would disappear, or at least, k U i, 2 ,t v i, 2 ,t k 2 !
when t ! 1 .
4.2.3 Convergence
For a ﬁxed environment, we prove w.h.p.
DPGrad converges and the loss approaches to zero, given the initial feature mapping matrix
U i,A is well conditioned.
Lemma 4.7.
For any i 2 [ k ] , suppose U i,A satisﬁes
[Equation]
where  min ( U i,A ) and  max ( U i,A ) denote the minimum and maximum non-zero singular value of matrix
U i,A .
After T
= O ( D ⌘ log Dkd ✏⌫ )
+ O ( D ⌘ log k  ) iterations, with probability
at least 1  O (1 /k ) , the loss b
L ( U
[Equation]
/Dnk .
Outline of the proof
DPGrad ensures existing features are preserved and it only optimizes the linear classiﬁer, hence a linear convergence rate can be easily derived for the ﬁrst loss term, given the feature matrix is well-conditioned.
The key part is controlling the terms that capture learning with new features, i.e., the second and last loss term, where both the feature mapping U i,B and linear prompt v i get updated.
In this case, the objective is non-convex and non-smooth.
Our analysis draws inspiration from the recent work of [40], and divides the optimization process into two stages.
We prove DPGrad ﬁrst approaches to a nice initialization position with high probability, and then show linear convergence.
To be concrete, in the ﬁrst stage, we prove (1) x > i,t v i, 2 ,t moves closer to 1 , and (2) k x i,
t  k w i,B k 2 v i, 2 ,t k 2 ⇡ 0 .
That is, the second loss term of Eq.
(5) decreases to a small constant while the pairs x i,t , v i, 2 ,t remain balanced and roughly equal up to scaling.
Meanwhile, we note U i, 2 ,t is non-increasing, though the last loss term could still increase because k v i, 2 ,t k 2 increases.
In the second stage, we prove by induction that k U > i, 2 ,t v i, 2 ,t k 2 and | x > i,t v i, 2 ,t  1 | decay with a linear rate (hence converging to a global optimal), and k x i,t  k w i,B k 2 v i, 2 ,t k 2 ⇡ 0 .
4.2.4 Induction
Lemma 4.7 proves rapid convergence of DPGrad for one single environment.
To extend the argument to the whole sequence of environments, we need to ensure (1) the feature matrix is always wellconditioned and (2) catastrophic forgetting does not happen.
For (1), we need to analyse the limiting point of DPGrad (there are inﬁnitely many optimal solutions to Eq.
(3)), make sure it is well-balance and orthogonal to previous row/column space.
For (2), we make use of the orthogonality of DPGrad .
Proof Sketch of Theorem 2.11.
Thanks to the reduction established in Section 4.2.1, it sufﬁces to prove Eq.
(3) and Eq.
For each environment i ( i 2 [ k ] ), we inductively prove (1) DPGrad achieves good accuracy on the current environment,
i.e., k U
i,T v i  w i k 2  ✏⌫ ; (2) The feature matrix U
i remains well conditioned, i.e.
1 2 p D   min ( U i, end )   max ( U i, end )  2 p D and (3)
The algorithm does not suffer from catastrophic forgetting, i.e., k U i,t v j  w i k 2  ✏ for any j < i , t 2 [ T ] .
The ﬁrst claim is already implied by Lemma 4.7.
For the second claim, one ﬁrst shows DPGrad exactly recovers w i by taking w i =
b w i
=
Round ⌫ ( U i,T v i ) .
When w i,B = 0 , one can prove the feature matrix does not change, i.e, U i, end =
U i  1 , end ; when w i,B 2 [1 /D, D ] , then one can show
U i, end ⇡ U
i, end + k v i, 2 ,T k 22 w i,B v > i, 2 ,T , as w i,B ? column ( U i  1 , end ) , v i, 2 ,T ?
row ( U i  1 , end ) and k k v i, 2 ,T k 22 w B v > i, 2 ,T k  O ( p D ) , the feature matrix U remains well-conditioned.
The last claim can be derived from the orthogonality.
This wraps up the proof of Theorem 2.11.
Lower bound for non-linear features
We next consider continual learning under a non-linear feature mapping.
Learning with non-linear features turns out to be much more difﬁcult, and our main result is to rule out the possibility of a (proper) continual learner.
We restate the formal statement.
The detailed proof are deferred to
Appendix C.
Theorem 2.12 (Barrier for Continual learning with non-linear feature) .
Let k, r  2 , d  3 .
There exists a class of non-linear feature mappings and a sequence of environments, such that there is no (proper) continual learning algorithm that can guarantee to achieve less than 1 1000 -error over all environments with probability at least 1 / 2 ,under the feature extraction formalization of Deﬁnition 2.2.
Our lower bound is constructed on a simple family of two-layer convolutional neural network with quadratic activation functions.
The input distribution is assumed to be uniform and the target function is a polynomial over the input.
The ﬁrst environment is constructed such that multiple global optimum exist (hence the optimization task is under-constrained).
However, if a wrong optimum solution is picked, when the second environment is revealed, the non-linearity makes it impossible to switch back-and-forth.
Proof Sketch.
It sufﬁces to take k = 2 , d = 3 , r =
and we sketch the construction here.
For both environments, we assume the input data are drawn uniformly at random from B 3 (0 , 1) , where
B 3 (0 , 1) denotes the unit ball in R 3 centered at origin.
The hypothesis class H consists of all two-layer convolutional neural network with a single kernel of size 2 and the quadratic activation function.
That is, the representation function is parameterized by w 2 R 2 and takes the form of
R w ( x ) =
( h w, x 1:2 i 2 , h w, x 2:3 i 2 ) 2 R 2 , where x 2 R 3 , x i : j 2 R j  i +1 is a vector consists of the i -th entry to the j -th entry of x .
The hard sequence of environments are drawn from the following distribution: (1) The objective function f 1 of the ﬁrst environment is f 1 ( x ) = x 22 ; (2) The objective function f 2 of the second environment equals f 2 ( x ) = x 23 with probability 1 / 2 , and equals f 2 ( x ) =
x 21 with probability 1 / 2 .
Note the continual learning task is realizable and one can prove no (proper) continual learning algorithm can guarantee to achieve less than 1 / 1000 -error on both environments with probability at least 1 / 2 .
Though our lower bound instance uses a polynomial activation function, this assumption is not essential – in Appendix C, we prove similar lower bounds with a ReLU activation function.
Experiments
For linear feature functions, we perform simulations on a synthetic dataset to verify the practicality of DPGrad and compare its performance with vanilla SGD and Orthogonal gradient descent (OGD), a close practical cousin of our algorithm.
In our simulations, we set d = 100 , r = 20 , k = 500 and the ground truth U ? , V ? is drawn from Gaussian.
The input data are sampled from N (0 , I d ) and we draw 1000 samples for each task.
Additional details about the setup can be found in Appendix D.
The results are presented at Figure 1.
It indicates the (1) practicality of DPGrad and (2) DPGrad signiﬁcantly outperforms the vanilla SGD and OGD (of course, DPGrad is designed for this kind of data).
The population loss is measured at the end and the it equals k Uv i  w i k 2 for each task i .
The average error of DPGrad is 0 .
001 , the average error of OGD is 83 .
59 , the average error of SGD is 5 .
Moreover, in Appendix E, we provide additional experimental results on two popular benchmarks,
Rotated MNIST and Permuted MNIST.
Since DPGrad is designed speciﬁcally for linear regression, we provide two variants of DPGrad (without provable guarantees on their performance, of course)— one is a modiﬁcation suitable for multi-class classiﬁcation, the other is a modiﬁcation suitable for non-linear featurizers.
Detailed numbers and ﬁgures can be found in Appendix E.
In brief, both algorithms alleviate catastrophic forgetting and perform much better than vanilla SGD.
Furthermore, the performance of both is much more stable than OGD and the accuracy remains at a high level across tasks.
Conclusion
In this paper, we initiate a study of continual learning through the feature extraction lens , proposing an efﬁcient gradient based algorithm, DPGrad , for the linear case, and a fundamental impossibility result in the general case.
Our work leaves several interesting future directions.
First, it would be interesting to generalize DPGrad to non-linear feature mappings (perhaps even without provable guarantees) and conduct an empirical study of its performance.
Second, our impossibility result does not rule out an improper continual learner, and in general, one can always maintain a task speciﬁc representation function and achieve good performance over all environments.
It would be thus interesting to investigate what are the fundamental memory-accuracy trade-offs.
Acknowledgement
BP acknowledges support of the NSF via CCF-1909756, CCF2007443, CCF-2134105, CCF-1703925, IIS-1838154, CCF-2106429 and CCF-2107187.
AR acknowledges support of the NSF via IIS2211907, the CMU/PwC DT&I Center, and an Amazon Research Award.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]