Continual learning: a feature extraction formalization, an efﬁcient algorithm, and barriers
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Continual learning is an emerging paradigm in machine learning, wherein a modelis exposed in an online fashion to data from multiple different distributions (i.e.environments), and is expected to adapt to the distribution change.
Precisely, thegoal is to perform well in the new environment, while simultaneously retaining theperformance on the previous environments (i.e. avoid “catastrophic forgetting”).
While this setup has enjoyed a lot of attention in the applied community, therehasn’t be theoretical work that even formalizes the desired guarantees.
In thispaper, we propose a framework for continual learning through the framework offeature extraction—namely, one in which features, as well as a classiﬁer, are beingtrained with each environment.
When the features are linear, we design an efﬁcientgradient-based algorithm DPGrad , that is guaranteed to perform well on the currentenvironment, as well as avoid catastrophic forgetting.
In the general case, when thefeatures are non-linear, we show such an algorithm cannot exist, whether efﬁcientor not.
Introduction
In the last few years, there has been an increasingly large focus in the modern machine learningcommunity on settings which go beyond iid data .
This has resulted in the proliferation of newconcepts and settings such as out-of-distribution generalization [15], domain generalization [3],multi-task learning [39], continual learning [23] and etc.
Continual learning , which is the focus ofthis paper, concerns learning through a sequence of environments, with the hope of retaining oldknowledge while adapting to new environments.
Unfortunately, despite a lot of interest in the applied community—as evidenced by a multitude of
NeurIPS and ICML workshops [24, 11, 28]—approaches with formal theoretical guarantees are fewand far between.
The main reason, similar encountered as its cousin ﬁelds like out-of-distributiongeneralization or multi-tasks learning, usually come with some “intuitive” desiderata — but no formaldeﬁnitions.
What’s worse, it’s often times clear that without strong data assumptions—the problem iswoefully ill-deﬁned.
The intuitive desiderata the continual learning community has settled on is that the setting involvescases where an algorithm is exposed (in an online fashion) to data sequentially coming from differentdistributions (typically called “environments”, inspired from a robot/agent interacting with differentenvironments).
Moreover, the goal is to keep the size of the model being trained ﬁxed, and makesure the model performs well on the current environment while simultaneously maintaining a goodperformance in the previously seen environments.
In continual learning parlance, this is termed“resistance to catastrophic forgetting”.
It is clear that some of the above desiderata are shared with well-studied learning theory settings(e.g.
online learning, lifelong learning), while some aspects differ.
For example, in online learning, we don’t care about catastrophic forgetting (or we only do so in some averaged sense); in lifelonglearning, it’s not necessary to keep the size of the model ﬁxed.
It is also clear that absent someassumptions on the data and the model being trained, these desiderata cannot possibly be satisﬁed:why would there even exist a model of some ﬁxed size that performs well on both past environments,and current ones — let alone one that gets updated in an online fashion.
A feature-extraction formalization of continual learning:
Our paper formalizes a setting forcontinual learning through the lens of feature extraction : the model maintains a ﬁxed number of(trainable) features, as well as a linear classiﬁer on top of said features.
The features are updatedfor every new environment, with the objective that the features are such that a good linear classiﬁerexists for the new environment, while the previously trained linear classiﬁers (on the updated features)are still good for the past environments.
The reason the linear classiﬁers from previous rounds arenot allowed to be updated is storage efﬁciency: in order to tune the prompts, one needs to storethe training data from previous tasks, this would bring a storage overhead and potentially privacyconcerns.
This is a very common approach in practice—examples of this are systems involving largeamounts of data of a streaming nature (e.g. Google searches, Youtube, a robotic agent interactingwith a continual stream of environments), and it be would prohibitive to store it for later ﬁne tuning.
The number of features is kept ﬁxed for the same reason: if we are to expand with new features forevery new environment, the model size (and hence storage requirements) would grow.
We prove two main results for our setting.1.
When the features are a linear function of the input data, and a good set of features exist, wedesign an efﬁcient algorithm , named doubly projected gradient descent, or DPGrad , thathas a good accuracy on all environments, and resists catastrophic forgetting.
Our algorithm,while being novel, bears some resemblance to a class of projection-based algorithms used inpractice [10, 5] – we hope the theoretical analysis can shed insight onto large scale continuallearning.
Experiments on synthetic dataset are performed to verify the practicality of ouralgorithm.2.
When the features are allowed to be a non-linear function of the input, we show that continuallearning is not possible—in general.
Namely, we construct an instance for which even if agood set of features exists, the online nature of the setting, as well as the fact that the linearclassiﬁers for past environments are not allowed to be updated, makes it possible for thealgorithm to “commit” to linear classiﬁers, such that either catastrophic forgetting, or poorperformance on the current environment has to occur.
Our results
2.1 Setup
In a continual learning problem, the learner has sequential access to k environments.
In the i -th( i 2 [ k ] ) environment, the data is drawn i.i.d.
from the underlying distribution D i over R d ⇥ R ,denoted as ( x, y )
⇠ D i , where x 2 R d is the input and y 2 R is the label.
Motivated by the empiricalsuccess of representation learning [2, 8], we formulate the continual learning problem through thefeature extraction view: The learner is required to learn a common feature mapping (also knownas representation function)
R : R d !
R r that maps the input data x 2 R d to a low dimensionalrepresentation R ( x ) 2 R r ( r ⌧ d ), along with a sequence of task-dependent linear classiﬁers (alsoknown as linear prompts) v 1 , . .
, v k 2 R r on top of the representation.
Precisely, the prediction ofthe i -th environment is made by f ( x ) =
h v i , R ( x ) i .
As this is the ﬁrst-cut study, we focus on the realizable and the proper learning setting.
1 That is, weassume the existence of a feature mapping R in the function class H (which is known in advance)and a sequence of linear predictor v 1 , . .
, v k such that for any i 2 [ k ] and any data ( x, y )
⇠ D i ,y =
h
v i , R ( x ) i (realizable).
The learner is required to output a function R that belongs to thehypothesis class H (i.e. the learner is proper).
Remark 2.1 (Known environment identity) .
Our model requires the knowledge of environmentidentity at test time, and thus can be classiﬁed into the category of incremental task learning.
Wenote there is also empirical research focusing on unknown environment identity, which would be aninteresting direction for future work (See Section 7).
The guarantee that we wish our learning algorithm to obtain is as follows:
Deﬁnition 2.2 (Goal of Continual Learning) .
Let d, k, r 2 N , r ⌧ d, k , ✏ 2 (0 , 1 / 2) .
Let H be afunction class in which the feature mappings from R d to R r lie.
The continual learning probleminvolves k environments D 1 , . . .
, D k .
We assume there exists a function R ?
2 H and a sequence 92 of linear classiﬁers v ? 1 , . .
. , v ?
k 2 R r such that for any ( x, y )
⇠ D i ( i 2 [ k ] ), the label satisﬁes 93 y
= h v ?
i , R ?
( x ) i .
The continual learner has sequential access to environments D 1 , . . .
, D k , as well as access toarbitrarily many samples per environment 2 .
The goal is to learn a representation function R 2 Hand a sequence of linear prompts v 1 , . .
, v k 2 R r that achieve a good accuracy on the current taskand do not suffer from catastrophic forgetting.
Formally, in the i -th environment ( i 2 [ k ] ), the learneroptimizes the feature mapping R and the linear classiﬁer v i (without changing v 1 , . .
. , v i  1 ) andaims to satisfy
• Avoid catastrophic forgetting : During the execution of the
i th
task
, the algorithm guarantees that
[Equation]
for all j = 1 , . . .
[Equation]
i
[Equation]
• Good accuracy on the current task: At the end of i -th task, the algorithm guarantees that
[Equation]
For linear feature mapping, the representation function can be written in a linear form R ( x ) =
U > xfor some U 2 R d ⇥ r , and it implies the i -th environment is generated by a linear model.
That is,deﬁning w i =
Uv i 2 R d , one can write y
=
h
v i , U > x i
=
h w i , x i .
Remark 2.3 (The beneﬁt of continual learning with linear feature) .
Note, for linear features, it’sin principle possible to just learn a sequence of linear classiﬁers w 1 , . . .
, w k 2 R d separately—without learning a low-dimensional featurizer.
Choosing an r -dimensional featurizer confers memoryefﬁciency ( O ( kr + dr ) vs. O ( dk ) ) and sample efﬁciency ( O ( r ) vs. O ( d ) samples per task in theasymptotic regime k ! 1 ).
Furthermore, the linear case is a sandbox that can be mathematicallyanalyzed and can generate insights for the nonlinear case as well.
2.2 DPGrad: Efﬁcient gradient based method for linear features
For the case of linear features, we propose an efﬁcient algorithm which we term DPGrad (pseudocodein Algorithm 1), which is an efﬁcient gradient based method and provably learns the representationwhile avoids catastrophic forgetting.
Towards stating the result, we make a few technical assumptions.
Assumption 2.4 (Distribution assumption) .
For any i 2 [ k ] , we assume D
i has zero means and it isin isotropic position, that is, E x ⇠ D i [ x ]
= ~ 0 and E x ⇠ D i [ xx > ] = I .
Remark 2.5.
This assumption is largely for convenience.
In fact, one can replace the isotropiccondition with a general bounded covariance assumption, our algorithm still can work with extrapreprocessing step, and the sample complexity scales with the condition number of covariance matrix.
Assumption 2.6 (Range assumption) .
For any i 2 [ k ] , w i has bounded norm, i.e., k w i k 2  D .
Assumption 2.7 (Signal assumption) .
For any i 2 [ k ] , let W i
=
span ( w 1 , . .
. , w i ) , W i, ?
be thespace perpendicular to W i and P W i , P W i, ?
be the projection operator.
We assume either w i belongs to
W i  1 or it has non-negligible component orthogonal to W i  1 , i.e., k P W i  1 , ?
w i k
2 2 { 0 } [ [1 /D, D ] .
Assumption 2.8 (Bit complexity assumption) .
Each coordinate of w i is a multiple of ⌫ > 0 .
Remark 2.9.
The Range and Signal assumptions are standard in the statistical learning literature.
The former ensures an upper bound on k w i k 2 and the later ensures that a new task provides enough“signal” for new features.
They are used to set up learning rate and number of gradient iterations.
Remark 2.10.
The Bit complexity assumption states that w i can be described with a ﬁnite numberof bits, and is mostly for convenience — namely so we can argue we exactly recover w i —which makes calculations involving projections of features learned in the past cleaner.
Since the number ofgradient iterations only depends logarithmically on ⌫ , one can relax the bit complexity restriction toonly approximately recovering the ground truth features up to a polynomially small (in d, k, D, ✏ )precision.
Our argument can still go through at the cost of some additional error analysis.
The main result is then as follows:
Theorem 2.11 (Continual learning with linear features) .
Let k, d, r 2 N , r ⌧ k, d , ✏ 2 (0 , 1 / 2) .
When the features are a linear function over the input data, under Assumption 2.4 and Assumption 2.62.8, with high probability, DPGrad provably achieves good loss and avoids catastrophic forgetting.
In particular, during the execution of i -th environment, DPGrad always ensures
[Equation]
for all
[Equation]
and at the end of i -th environment, DPGrad ensures
[Equation]
2.3 Fundamental obstructions for non-linear features
The continual learning setup with non-linear features turns out to be much more difﬁcult — evenwithout computational constraints.
Our result rules out the existence of a proper continual learner,even when all environment distributions are uniform and the representation function is realizable by atwo-layer convolutional neural network.
Theorem 2.12 (Barrier for Continual learning with non-linear feature) .
Let k, r  2 , d  3 .
Thereexists a class of non-linear feature mappings and a sequence of environments, such that there is no(proper) continual learning algorithm that can guarantee to achieve less than 1 1000 -error over allenvironments with probability at least 1 / 2 .
Related work
Continual learning in practice The study of continual learning (or lifelong learning) dates backto the work of [34] and it receives a surge of research interest over recent years [14, 17, 10, 5, 13, 29,32, 16, 27, 37].
A central challenge in the ﬁeld is to avoid catastrophic forgetting [22, 21], whichthe work of [14] observed happened for gradient-based training of neural networks.
We refer theinterested reader to more complete surveys [23, 6].
Continual learning in theory In comparison to the vast empirical literature, theoretical works arecomparatively few.
The works [33, 25, 1, 4] provide theoretical guarantees on lifelong learning.
Theirapproaches can be categorized roughly into the duplicate and ﬁne-tuning paradigm:
The algorithmmaintains a weighted combination over a family of representation functions and the focus is onthe sample complexity guarantee.
By contrast, we focus on the feature extraction paradigm andlearn linear prompts on top of a single representation function.
Both the duplicate-and-ﬁne-tuningand the feature extraction paradigm have been extensively investigated in the literature, detaileddiscussions can be found at [6] and we provide a brief comparison.
From an algorithmic perspective,learning a weighted combination over a family of representation functions (i.e. the duplicate andﬁne-tuning) is much easier, as one can always initiates a new representation function for a new task.
The algorithmic convenience allows previous literature focus more on the generalization and samplecomplexity guarantee, culminating with the recent work of [4].
We note again that learning a singlerepresentation function and task speciﬁc linear prompts is much more challenging, but has practicalbeneﬁts, e.g. memory efﬁciency.
For example, in the applications of NLP, the basic representationfunction (e.g. BERT [8]) is already overparameterized and contains billions of parameters.
It is thenformidable to maintain a large amount of the basic models and learn a linear combination over them.
We mention several more works that are morally related in Appendix A.
Continual learning with linear feature
We restate our main result for linear feature mapping.
Theorem 2.11 (Continual learning with linear features) .
Let k, d, r 2 N , r ⌧ k, d , ✏ 2 (0 , 1 / 2) .
When the features are a linear function over the input data, under Assumption 2.4 and Assumption 2.62.8, with high probability, DPGrad provably achieves good loss and avoids catastrophic forgetting.
In particular, during the execution of i -th environment, DPGrad always ensures
[Equation]
and at the end of i -th environment, DPGrad ensures
[Equation]
4.1 Algorithm
A complete and formal description of DPGrad is presented in Algorithm 1.
DPGrad simultaneouslyupdates the matrix of features U , as well as the linear classiﬁer v i using gradient descent—with therestriction that the update of U only occurs along directions that are orthogonal to the column androw span of the previous feature matrix.
Intuitively, one wishes the projection guarantees that existingfeatures that have been learned are not erased or interfered by new environments.
Due to the quadraticnature of the loss, and the appearance of “cross-terms”, this turns out to require both column and roworthogonality, and interestingly deviates from the practically common OGD method [10, 5].
In more detail, at the beginning of the i -th ( i 2 [ k ] ) environment, DPGrad adds Gaussian noiseto the feature matrix U and the linear classiﬁer v i , to generate a good initialization for U and v i .
Subsequently, we perform gradient descent to both the feature mapping matrix U and linear classiﬁerv i —except U is only updated along orthogonal directions w.r.t.
the column span and the row span.
At the end of each environment, DPGrad has a post-processing step to recover the ground truth w i byrounding each entry of Uv i to the nearest multiple of ⌫ , 3 and then update the column and row spanif the orthogonal component is non-negligible.
The reason for the later step is that we only need topreserve row space when encountering new features.
Algorithm 1: W ; , V ; , U .
U 2 R d ⇥ r 2:  e
O ( ✏ d 2 kD 4 ) , ⌘ O (  3 k 2 D 5 ) , T O ( D ⌘ log Dkd ✏⌫ )
+ O ( D ⌘ log k  ) 3: for i = 1 , . . .
, k do 4:
U init  · P W ? rand ( d, r ) P V ? , v i  · rand ( r ) .
U init 2 R d ⇥ r , v i 2 R r 5:
U
U + U init 6: for t = 1 , . . .
, T do 7: r U
E ( x,y ) ⇠ D i [ x ( x > Uv i  y ) v > i ] , r v i E ( x,y )
⇠ D i [ U > x ( x > Uv i  y )] 8: U = U  ⌘ P W ? r U P V ?
9: v i = v i  ⌘ r
v i 10: end for 11: end for 12: b w i
Round ⌫ ( Uv i ) .
Round to the nearest multiple of ⌫ , b w i 2 R d 13: if k P W ?
b w i k 2  1 /D
then W span ( W [ b w i ) , V span ( V [ v i ) 14: U
P W UP V
Parameters
We use  to denote the initialization scale, ⌘ to denote the learning rate, and T todenote the number of iterations for each task.
These are all polynomially small parameters, whosescaling is roughly D, d, k ⌧   1 ⌧ ⌘  1 < T .
Notation We write [ n ] = { 1 , 2 , . . . , n } , [ n 1 : n 2 ] =
{ n 1 , . . . , n 2 } .
We use rand ( n 1 , n 2 ) 2 R n 1 ⇥ n 2to denote a size n 1 ⇥ n 2 matrix whose entries are draw from random Gaussian N (0 , 1) .
For eachi 2 [ k ] , t 2 [0 : T ] , denote U i,t to be the feature matrix in the t -th iteration of the i -th environment(after performing the gradient update), denote v i,t similarly.
DPGrad includes a projection step atthe end of i -th environment, we use U i, end to denote the feature matrix after this projection.
We use
W i (resp.
V i ) to denote the column (resp. row) space maintained at the end of i -th environment.
Let
W ?
✓ R n be the subspace orthogonal to W and deﬁne V ?
similarly.
Let P W , P V , P W ? , P V ?
be the 206 projection onto W , V , W ? , V ? separately.
4.2 Analysis
We sketch the analysis of DPGrad and prove Theorem 2.11.
Due to space limitation, the detailedproof is deferred to Appendix B.
The proof proceeds in the following four steps:1.
The ﬁrst step, presented in Section 4.2.1, reduces continual learning to a problem of continualmatrix factorization and it allows us to focus on a more algebraically friendly objectivefunction.2.
We then present some basic linear-algebraic facts to decompose the feature mapping matrix
U , its gradient, and the loss into orthogonal components.
The orthogonality of gradientupdate allows us to decouple the process of leveraging the existing features and the processof learning a new feature , as reﬂected in the loss terms and gradient update rules.
See
Section 4.2.2 for details.3.
In Section 4.2.3, we zoom into one single environment, and prove DPGrad provably converges to a global optimum, assuming the feature matrix U from previous environment iswell conditioned.
This step contains the major bulk of our analysis: The objective function ofcontinual matrix factorization is non-convex, and no regularization or spectral initializationused.
(We cannot re-initialize, lest we destroy progress from prior environments.)4.
Finally, in Section 4.2.4, we inductively prove that DPGrad converges and the feature matrixis always well-conditioned.
This wraps up the entire proof.
4.2.1 Reduction
We ﬁrst recall the formal statement of the problem of continual matrix factorization.
Deﬁnition 4.1 (Continual matrix factorization) .
Let d, k, r 2 N , r ⌧ d, k , ✏ > 0 .
Let W =[ w 1 , . . .
, w k ]
=
U ?
( V ? )
> 2 R d ⇥ k , where U ?
2 R d ⇥ r , V ? 2 R k ⇥ r .
In an continual matrixfactorization problem, the algorithm receives w i 2 R d in the i -th step, and it is required to maintaina matrix
U 2 R d ⇥ r and output a vector
v
i 2 R r such that
[Equation]
and
[Equation]
The key observation is that running DPGrad on the original continual learning objective (2) implicitlyoptimizes the continual matrix factorization objective (3) (Lemma 4.2).
Moreover, an ✏ -approximatesolution of continual matrix factorization is also an ✏ -approximate solution of continual learning(Lemma 4.3).
Lemma 4.2 (Gradient equivalence) .
Under Assumption 2.4, for any i 2 [ k ] , the gradient update ofobjective (2) equals the gradient update of objective (3) .
Lemma 4.3 (Objective equivalence) .
For any w 1 , . . .
, w k 2 R d , U 2 R d ⇥ r and v 1 , . .
, v k 2 R r ,suppose b
L ( U, v
[Equation]
) = 12 k Uv i  w i k 22  ✏ holds for all i 2 [ k ] , then L ( U, v i ) =
12 E ( x,y )
⇠ D i ( x > Uv i y )
2  ✏ .
Combining the above observations, it sufﬁces to analyse DPGrad for continual matrix factorizationand prove Eq.
(3) and Eq.
4.2.2 Decomposition
We ﬁrst provide some basic linear algebraic facts about orthogonal decompositions.
For any i 2 [ k ] ,we decompose U i , v i , w i along W i  1 , W i  1 , ? , V i  1 and V i  1 , ? .
Let w i =
w i,A + w i,B where w i,A 2 W i  1 and w i,B 2 W i  1 , ? .
Note this decomposition is 247 unique.
We focus on the case that k w i,B k 2 2 [1 /D, D ] in the following statements, and the case ofk w i,B k 2 = 0 carries over easily.
(These are the only two cases, per Assumption 2.7).
Similarly, let 249 U
i
=
U i,A
+ U i,B , where each column of U i,
A lies W i  1 and each column of w i,B lies in W i  1 , ?
.(Note,
again, U i,A and U i,B are unique.)
We further write U i,B = w i,B x > i
+ U
i, 2 , where the columnsof U i, 2 lie in W i  1 , ?
\{ w i,B } .
Finally, denote v i
=
v i, 1 + v i, 2 with v
i, 1 2 V i  1 and v i, 2 2 V i  1 , ? .
We summarize decompositions mentioned above, with a few additional observations, in the lemmabelow:
Lemma 4.4 (Orthogonal decomposition) .
For any i 2 [ k ] and any t 2 [0 : T ] , there exists an uniquedecomposition of U i,t , w i and v i,t of the form U i,t =
U i,A
[Equation]
+ U i,B,t , column ( U i,A
[Equation]
, column
[Equation]
U
[Equation]
Here we use column ( A ) , row ( A ) to denote the column and row space of matrix A , and column ( A )
W if the column space of A is a subspace of W .
Since U i,A,t remains unchanged for t =
[0 : T ] , we abbreviate it as U i,A hereafter.
We next providethe exact gradient update of each component under loss function b
L ( U i , v i ) = 12 k U i v i  w i k 22 and 260 orthogonal projection.
Lemma 4.5 (Gradient formula) .
For any i 2 [ k ] , the gradient update (after projection) obeysthe relations (1) r x i ( b L )
=
v i, 2 ( x > i v i, 2  1) ; (2) r U 2 ,i ( b
L )
= U i, 2 v i, 2 v > i, 2 ; (3) r v i, [Equation] ) =
U > i,A U i,
A v i, 1  U >
i,A w i,A and (4) r v i,
[Equation]
We perform a similar decomposition to the loss function.
Lemma 4.6 (Loss formula) .
For any i 2 [ k ] , t 2 [ T ] , we have
[Equation]
i,t
[Equation] i, [Equation] Decoupling existing features from “new” features We now offer some intuitive explanation forthe decomposition.
The ﬁrst loss term in Eq.
(5) quantiﬁes the error with already learned features.
That is, the matrix U i,A stores existing features that have been learned, and it remains unchangedduring the execution of the i -th environment; it remains to optimize v i, 1 ,t such that U i,
A v
i, 1 ,t matches 270 w i,A .
The second and last loss term quantify the loss on a new feature, where w i,B is the new featurecomponent, and the matrix U i, 2 ,t can be thought of as random noise.
Intuitively, one should hopex > i,t v i, 2 ,t =
and this matches the new component of w i,B .
At the same time, one hopes U i, 2 ,t woulddisappear, or at least, k U i, 2 ,t v i, 2 ,t k 2 !
when t ! 1 .
4.2.3 Convergence
For a ﬁxed environment, we prove w.h.p.
DPGrad converges and the loss approaches to zero, giventhe initial feature mapping matrix U i,A is well conditioned.
Lemma 4.7.
For any i 2 [ k ] , suppose U i,A satisﬁes 1 2 p D   min ( U i,A )   max ( U i,A )  2 p D ,where  min ( U i,A ) and  max ( U i,A ) denote the minimum and maximum non-zero singular value ofmatrix U i,A .
After T
= O ( D ⌘ log Dkd ✏⌫ )
+ O ( D ⌘ log k  ) iterations, with probability
at least 1  O (1 /k ) ,the loss b
L ( U i , v i )  ✏⌫ /Dnk .
Outline of the proof
DPGrad ensures existing features are preserved and it only optimizes thelinear classiﬁer, hence a linear convergence rate can be easily derived for the ﬁrst loss term, given thefeature matrix is well-conditioned (Lemma 4.8).
The key part is controlling the terms that capturelearning with new features, i.e., the second and last loss term, where both the feature mapping U i,Band linear prompt v i get updated.
In this case, the objective is non-convex and non-smooth.
Ouranalysis draws inspiration from the recent work of [38], and divides the optimization process intotwo stages.
We prove DPGrad ﬁrst approaches to a nice initialization position with high probability,and then show linear convergence.
To be concrete, in the ﬁrst stage, we prove (1) x > i,t v i, 2 ,t moves closer to 1 , and (2) k x i,t k w i,B k 2 v i, 2 ,t k 2 ⇡ 0 (Lemma 4.9).
That is, the second loss term of Eq.
(5) decreases to a small constant while the pairs x i,t , v i, 2 ,t remain balanced and roughly equal up to scaling.
Meanwhile, wenote U i, 2 ,t is non-increasing, though the last loss term could still increase because k v i, 2 ,t k 2 increases.
In the second stage, we prove by induction that k U > i, 2 ,t v i, 2 ,t k 2 and | x > i,t v i, 2 ,t  1 | decay with a linearrate (hence converging to a global optimal), and k x i,t  k w i,B k 2 v i, 2 ,t k 2 ⇡ 0 (Lemma 4.10). First, we prove linear convergence for the ﬁrst loss term. Lemma 4.8 (Fast learning on existing features) . For any i 2 [ k ] and t 2 [ T ] , we have
[Equation]
We next focus on the second and last loss terms.
{+One can show that x > i,t v i, 2 ,t moves to 1 whilekk w i,B k 2 x i,t  v i, 2 ,t k 2 remains small in the ﬁrst T 
 = O ( D ⌘ log k  ) iterations. Lemma 4.9. With probability at least 1  O (1 /k ) over the random initialization, there exists+}
{+T 
+}
= O ( D ⌘ log k  ) , such that for any t  T 1 , one has (1) kk w i,B k 2 x i,t  v i, 2 ,t k 2  O ( r  log( k/  )) ;(2) x > i,t v i, 2 ,t < 0 .
when t < T 1 and 0 .
9 < x > i,T 1 v i, 2 ,T 1 < 1 ; (3) U > i, 2 ,t U i, 2 ,t  U > i, 2 ,
0 U i, 2 , 0 .
A linear convergence of the second and the last loss terms can be shown, after the ﬁrst T 1 iterations.
Lemma 4.10.
{+Let T 
+}
= O ( D ⌘ log( kdD ✏⌫ )) .
After T = T 1 + T 2 iterations, we have (1) | x > i,T v i, 2 ,T  1 | ✏⌫ /kdD ; (2) k U > i, 2 ,T U > i, 2 ,T v i, 2 ,T k 2  ✏⌫ .
Combining Lemma 4.6, Lemma 4.8 – 4.10, one can conclude the proof of Lemma 4.7.
4.2.4 Induction
Lemma 4.7 proves rapid convergence of DPGrad for one single environment.
To extend the argumentto the whole sequence of environments, we need to ensure (1) the feature matrix is always wellconditioned and (2) catastrophic forgetting does not happen.
For (1), we need to analyse the limitingpoint of DPGrad (there are inﬁnitely many optimal solutions to Eq.
(3)), make sure it is well-balanceand orthogonal to previous row/column space.
For (2), we make use of the orthogonality of DPGrad .
Proof Sketch of Theorem 2.11.
Thanks to the reduction established in Section 4.2.1, it sufﬁces toprove Eq.
(3) and Eq.
For each environment i ( i 2 [ k ] ), we inductively prove (1) DPGradachieves good accuracy on the current environment, i.e., k U
i,T v i  w i k 2  ✏⌫ ; (2)
The featurematrix U i remains well conditioned, i.e.
1 2 p D   min ( U i, end )   max ( U i, end )  2 p D and (3) Thealgorithm does not suffer from catastrophic forgetting, i.e., k U i,t v j  w i k 2  ✏ for any j < i , t 2 [ T ] .
The ﬁrst claim is already implied by Lemma 4.7.
For the second claim, one ﬁrst shows DPGradexactly recovers w i by taking w i =
b [Equation]
i =
Round ⌫ ( U i,T v i ) .
When w i,B = 0 , one can prove thefeature matrix does not change, i.e, U i, end =
U i  1 , end ; when w i,B 2 [1 /D, D ] , then one can show
U i, end ⇡ U
i, end + k v i, 2 ,T k 22 w i,B v > i, 2 ,T , as w i,B ? column ( U i  1 , end ) , v i, 2 ,T ?
row ( U i  1 , end ) andk k
v i, 2 ,T k 22 w B v > i, 2 ,T k  O ( p D ) , the feature matrix U remains well-conditioned.
The last claim canbe derived from the orthogonality.
This wraps up the proof of Theorem 2.11.
Lower bound for non-linear features
We next consider continual learning under a non-linear feature mapping.
Learning with non-linearfeatures turns out to be much more difﬁcult, and our main result is to rule out the possibility ofa (proper) continual learner.
We restate the formal statement.
The detailed proof are deferred to
Appendix C.
Theorem 2.12 (Barrier for Continual learning with non-linear feature) .
Let k, r  2 , d  3 .
Thereexists a class of non-linear feature mappings and a sequence of environments, such that there is no(proper) continual learning algorithm that can guarantee to achieve less than 1 1000 -error over allenvironments with probability at least 1 / 2 .
Our lower bound is constructed on a simple family of two-layer convolutional neural network withquadratic activation functions.
The input distribution is assumed to be uniform and the target functionis a polynomial over the input.
The ﬁrst environment is constructed such that multiple global optimum exist (hence the optimization task is under-constrained).
However, if a wrong optimum solution ispicked, when the second environment is revealed, the non-linearity makes it impossible to switchback-and-forth.
Proof Sketch.
It sufﬁces to take k = 2 , n = 3 , d =
and we sketch the construction here.
Forboth environments, we assume the input data are drawn uniformly at random from B 3 (0 , 1) , where
B 3 (0 , 1) denotes the unit ball in R 3 centered at origin.
The hypothesis class H consists of alltwo-layer convolutional neural network with a single kernel of size 2 and the quadratic activationfunction.
That is, the representation function is parameterized by w 2 R 2 and takes the form of
R w ( x ) =
( h w, x 1:2 i 2 , h w, x 2:3 i 2 ) 2 R 2 , where x 2 R 3 , x i : j 2 R j  i +1 is a vector consists of the 343 i -th entry to the j -th entry of x .
The hard sequence of environments are drawn from the following distribution: (1) The objectivefunction f 1 of the ﬁrst environment is f 1 ( x ) = x 22 ; (2) The objective function f 2 of the secondenvironment equals f 2 ( x ) =
x
23 with probability 1 / 2 , and equals f 2 ( x ) = x 21 with probability 1 / 2 .
Note the continual learning task is realizable and one can prove no (proper) continual learningalgorithm can guarantee to achieve less than 1 / 1000 -error on both environments with probability atleast 1 / 2 .
Simulations
For linear feature functions, we perform simulations on a synthetic dataset to verify the practicalityof DPGrad and compare its performance with vanilla SGD and Orthogonal gradient descent (OGD),a close practical cousin of our algorithm.
In our simulations, we set d = 100 , r = 20 , k = 500 andthe ground truth U ? , V ? is drawn from Gaussian.
The input data are sampled from N (0 , I d ) and wedraw 1000 samples for each task.
Additional details about the setup can be found in Appendix D.
Data is synthetically generated with d = 100 , r = 20 , k = 500 and the ground truth U ?
, V ? is drawn from Gaussian.
Additional details about the setup can be found in Appendix D.
The results are presented at Figure 1.
It indicates the (1) practicality of DPGrad and (2) DPGradsigniﬁcantly outperforms the vanilla SGD and OGD (of course, DPGrad is designed for this kind ofdata).
The population loss is measured at the end and the it equals k Uv i  w i k 2 for each task i .
Theaverage error of DPGrad is 0 .
001 , the average error of OGD is 83 .
59 , the average error of SGD is
Conclusion
In this paper, we initiate a study of continual learning through the feature extraction lens , proposingan efﬁcient gradient based algorithm, DPGrad , for the linear case, and a fundamental impossibilityresult in the general case.
Our work leaves several interesting future directions.
First, it would beinteresting to generalize DPGrad to non-linear feature mappings (perhaps even without provableguarantees) and conduct an empirical study of its performance.
Second, our impossibility resultdoes not rule out an improper continual learner, and in general, one can always maintain a taskspeciﬁc representation function and achieve good performance over all environments.
It would bethus interesting to investigate what are the fundamental memory-accuracy trade-offs.
Finally, ourformulation assumes the task identity is known at test time; generalizing it to unknown task identityis direction for further work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] We only run simulations and the error gap is pretty large and robust.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board ) approvals, if applicable
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
[N/A]
[Table]