L EARNING CURVES FOR CONTINUAL LEARNING IN NEURAL NETWORKS : S ELF - KNOWLEDGE TRANSFER AND FORGETTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequential training from task to task is becoming one of the major objects in deep learning applications such as continual learning and transfer learning.
Nevertheless, it remains unclear under what conditions the trained model’s performance improves or deteriorates.
To deepen our understanding of sequential training, this study provides a theoretical analysis of generalization performance in a solvable case of continual learning.
We consider neural networks in the neural tangent kernel (NTK) regime that continually learn target functions from task to task, and investigate the generalization by using an established statistical mechanical analysis of kernel ridge-less regression.
We ﬁrst show characteristic transitions from positive to negative transfer.
More similar targets above a speciﬁc critical value can achieve positive knowledge transfer for the subsequent task while catastrophic forgetting occurs even with very similar targets.
We also reveal that even for continual learning of the same target function, the trained model shows some transfer and forgetting depending on the sample size of each task.
We can guarantee that the generalization error monotonically decreases from task to task for equal sample sizes while unbalanced sample sizes deteriorate the generalization.
We respectively refer to these improvement and deterioration as self -knowledge transfer and forgetting, and empirically conﬁrm them in realistic training of deep neural networks.
I NTRODUCTION
As deep learning develops for a single task, it enables us to work on more complicated learning frameworks where the model is sequentially trained on multiple tasks, e.g., transfer learning, curriculum learning, and continual learning.
Continual learning deals with the situation in which the learning machine cannot access previous data due to memory constraints or privacy reasons.
It has attracted much attention due to the demand on applications, and fundamental understanding and algorithms are being explored (Hadsell et al., 2020).
One well-known phenomenon is catastrophic forgetting; when a network is trained between different tasks, naive training cannot maintain performance on the previous task (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017).
It remains unclear in most cases under what conditions a trained model’s performance improves or deteriorates in continual learning.
Understanding its generalization performance is still limited (Pentina & Lampert, 2014; Bennani et al., 2020; Lee et al., 2021).
For single-task training, however, many empirical and theoretical studies have succeeded in characterizing generalization performance in over-parameterized neural networks and given quantitative evaluation on sample-size dependencies, e.g., double descent (Nakkiran et al., 2020).
For further development, it will be helpful to extend the analyses on single-task training to sequential training on multiple tasks and give theoretical backing.
To deepen our understanding of continual learning, this study shows a theoretical analysis of its generalization error in the neural tangent kernel (NTK) regime.
In more detail, by extending a statistical mechanical analysis of kernel ridge-less regression on single-task training (Bordelon et al., 2020) to continual learning, we investigate the learning curve, i.e., the dependence of generalization error on sample size or number of tasks. We focus on the continual learning with explicit task boundaries.
Suppose that the model continually learns data generated by two similar teacher functions, which we call target functions, from one task to another.
All input samples are generated from the same distribution in an i.i.d. manner, and each task has output samples (labels) generated by its own target function.
In the NTK regime, the training of the network becomes independent among heads, and learning is formulated as a regression of each head on a corresponding target function.
Our main contributions are summarized as follows.
First, we revealed characteristic transitions from negative to positive transfer depending on the target similarity.
More similar targets above a speciﬁc critical value can achieve positive knowledge transfer (i.e., better prediction on the subsequent task than training without the ﬁrst task).
Compared to this, backward transfer (i.e., prediction on the previous task) has a large critical value, and subtle dissimilarity between targets causes negative transfer.
The error can rapidly increase, which clariﬁes that catastrophic forgetting is literally catastrophic (Section 4.1).
Second, we considered continual learning of the same target function.
Even for the continual learning in iterative tasks, the trained model’s performance improves or deteriorates in a non-monotonic way.
This depends on the sample size of each task; for equal sample sizes, we can guarantee that the generalization error monotonically decreases from task to task (Section 4.2 for two tasks & Section 5 for more tasks).
Unbalanced sample sizes, however, deteriorates generalization (Section 4.3).
We refer to these improvement and deterioration of generalization as self-knowledge transfer and forgetting, respectively.
Finally, we empirically conﬁrmed that self-knowledge transfer and forgetting actually appear in continual learning of multi-layer perceptron (MLP) and ResNet-18 (Section 5.1).
Thus, this study sheds light on fundamental understanding and the universal behavior of continual learning in over-parameterized learning machines.
R ELATED WORK
Method of analysis.
As an analysis tool, we use the replica method originally developed for statistical mechanics.
Statistical mechanical analysis enables us typical-case evaluation, that is, the average evaluation over data samples or parameter conﬁgurations.
It sometimes provides us with novel insight into what the worst-case evaluation has not captured (Abbaras et al., 2020; Spigler et al., 2020).
The replica method for kernel methods has been developed in Dietrich et al.
(1999), and recently in Bordelon et al.
Canatar et al.
These recent works showed excellent agreement between theory and experiments on NTK regression, which enables us to quantitatively understand sample-size dependencies including implicit spectral bias, double descent, and multiple descent.
Continual learning.
Continual learning dynamics in the NTK regime that we investigate was ﬁrst formulated by Bennani et al.
Doan et al.
(2021), though the evaluation of generalization remains unclear.
They derived an upper bound of generalization via the Rademacher complexity, but it includes naive summation over single tasks and seems conservative.
In contrast, our typical-case evaluation enables us to ﬁnd such rich behaviors as negative/positive transfer and self-knowledge transfer/forgetting we newly ﬁnd.
The continual learning in the NTK regime belongs to so-called single-head setting (Farquhar & Gal, 2018), and it allows the model to revisit the previous classes (target functions) in subsequent tasks.
This is complementary to earlier studies on incremental learning of new classes and its catastrophic forgetting (Ramasesh et al., 2020; Lee et al., 2021), where each task includes different classes and does not allow the revisit.
Note that the basic concept of continual learning is not limited to incremental learning but allows the revisit (McCloskey &
Cohen, 1989; Kirkpatrick et al., 2017).
Under the limited data acquisition or resources of memory, we often need to train the same model with the same targets (but different samples) from task to task.
Therefore, the setting allowing the revisit seems reasonable.
P RELIMINARIES 3. N EURAL TANGENT KERNEL REGIME
We summarize conventional settings of the NTK regime (Jacot et al., 2018; Lee et al., 2019).
Let us consider a fully connected neural network f = u L given by
[Equation]
[Equation]
where we deﬁne weight matrices W l ∈ R
M l × M l − 1 , bias terms b l ∈ R M l , and their variances σ 2 w and σ 2 b .
We set random Gaussian initialization W l,ij , b l,i ∼ N (0 , 1) and focus on the mean squared error loss:
L ( θ ) =
(cid:80)
Nµ =1 (cid:107)
y µ − f ( x µ ) (cid:107) 2 , where the training samples { x µ , y µ } N µ =1 are composed of inputs x µ ∈ R D normalized by (cid:107)
x µ (cid:107) 2 = 1 and labels y µ ∈ R C .
The set of all parameters is denoted as θ , and the number of training samples is N .
Assume the inﬁnite-width limit for hidden layers ( M l → ∞ ), ﬁnite sample size, and depth compared with the width.
The gradient descent dynamics with a certain learning rate then converges to a global minimum sufﬁciently close to the random initialization.
This is known as the NTK regime, and the trained model is explicitly obtained as
[Equation]
We denote the NTK matrix as Θ , arbitrary input samples (including test samples) as x (cid:48) , and the set of training samples as X .
The indices of f mean 0 for the model at initialization and c for the head of the network.
Entries of NTK Θ( x (cid:48) , x ) are deﬁned by ∇ θ f 0 ( x (cid:48) ) ∇ θ f 0 ( x ) (cid:62) .
We write Θ( X ) as an abbreviation for Θ( X, X ) .
The trained network is equivalent to a linearized model around random initialization (Lee et al., 2019), that is, f ( c ) =
f ( c )
+ ∇ θ f ( c )
∆ θ with
[Equation]
While over-parameterized models have many global minima, the NTK regime’s dynamics implicitly select the above θ , which corresponds to the L2 min-norm solution.
Usually, we ignore f 0 by taking the average over random initialization.
The trained model (2) is then equivalent to the kernel ridge-less regression (KRR).
NTK regime also holds in various architectures including ResNets and CNNs (Yang & Littwin, 2021), and the difference only appears in the NTK matrix.
Although we focus on the fully connected network in synthetic experiments, the following NTK formulation of continual learning and our theoretical results hold in any architecture under the NTK regime.
Continual learning.
We denote the set of training samples in the n -th task as ( X n , y n ) ( n = 1 , 2 , ..., K ), a model trained in a sequential manner from task 1 to task n as f n and its parameters as θ n .
That is, we train the network initialized at θ n − 1 for the n -th task and obtain f n .
Assume the number of tasks K is ﬁnite.
The trained model within the NTK regime is then given as follows (Bennani et al., 2020;
Doan et al., 2021):
[Equation]
We omit the index c because each head is updated independently.
The model f n completely ﬁts the n -th task, i.e., y n = f n ( X n ) .
The main purpose of this study is to analyze the generalization performance of the sequentially trained model (4).
At each task, the model has an inductive bias of
KRR in the function space and L2 min-norm solution in the parameter space.
The next task uses this inductive bias as the initialization of training.
The problem is whether this inductive bias helps improve the generalization in the subsequent tasks.
Remark (independence between different heads).
For a more accurate understanding of the continual learning in the NTK regime, it may be helpful to remark on the heads’ independence, which previous studies did not explicitly mention.
As in Eq.
(2), all heads share the same NTK, and f ( c ) depends only on the label of its class y ( c ) .
While the parameter update (3) includes information of all classes, the c -th head can access only the information of the c -th class 1 .
For example, suppose that the ( n − 1) -th task includes all classes except 1 , i.e., { 2 , ..., C } , but the n -th task includes only the class 1 .
Then, the model update at the n -th task, i.e., f 1 n − f 1 n − 1 , becomes
[Equation]
because ∇ θ f ( c ) 0 ∇ θ f ( c (cid:48) )
(cid:62)= 0 ( c (cid:54) =
c (cid:48) ) in the inﬁnite-width limit (Jacot et al., 2018; Yang, 2019).
Thus, we can deal with each head independently and analyze the generalization by setting C =without loss of generality.
This indicates that in the NTK regime, interaction between different heads do not cause knowledge transfer and forgetting.
One may wonder if there are any non-trivial knowledge transfer and forgetting in such a regime.
Contrary to such intuition, we reveal that when the subsequent task revisits previous classes (targets), the generalization shows interesting increase and decrease.
3. L EARNING CURVE ON SINGLE - TASK TRAINING
To evaluate the generalization performance of (4), we extend the following theory to continual learning.
Bordelon et al.
(2020) obtained an analytical expression of the generalization for NTK regression on a single task (2) as follows.
Assume that training samples are generated in an i.i.d.
manner ( x µ ∼ p ( x ) ) and that labels are generated using a square integrable target function ¯ f : ¯ f ( x ) = ∞ (cid:88)
i =
¯
w i ψ i ( x ) , y µ = ¯ f ( x µ ) + ε µ ( µ = 1 , ..., N ) , (6) where ¯ w i are constant coefﬁcients and ε represents Gaussian noise with (cid:104)
ε µ ε ν (cid:105)
=
δ µν σ 2 .
We deﬁne ψ
i ( x ) := √ η i φ i ( x ) with basis functions φ i ( x ) given by Mercer’s decomposition: [Equation] [Equation] =
[Equation] [Equation]
Here, η i denotes NTK’s eigenvalue and we assume the ﬁnite trance of NTK (cid:80)
i η i < ∞ .
We set η 0 = 0 in the main text to avoid complicated notations.
We can numerically compute eigenvalues by using the Gauss-Gegenbauer quadrature.
Generalization error is expressed by
E 1 := (cid:68)(cid:82) dxp ( x ) (cid:0) ¯ f ( x ) − f ∗ ( x ) (cid:1) 2 (cid:69) D where f ∗ is a trained model and (cid:104)· · · (cid:105) D is the average over training samples.
Bordelon et al.
(2020) derived a typical-case evaluation of the generalization error by using the replica method: for a sufﬁciently large N , we have asymptotically
[Equation]
Although the replica method takes a large sample size N , Bordelon et al.
Canatar et al.
(2021) reported that the analytical expression (8) coincides well with empirical results even for small N .
The constants κ and γ are deﬁned as follows and characterize the increase and decrease of E 1 :
[Equation]
The κ is a positive solution of the ﬁrst equation and obtained by numerical computation.
The γ satisﬁes 0 < γ < 1 by deﬁnition.
For N = αD l ( α > 0 , l ∈ N , D
(cid:29) 1 ), we can analytically solve it and obtain more detailed evaluation.
For example, a positive κ decreases to zero as the sample size increases and E 1 also decreases for σ 2 = 0 .
For σ 2 > 0 , the generalization error shows multiple descent depending on the decay of eigenvalue spectra.
Since multiple descent is not a main topic of this paper, we brieﬂy summarize it in Section A.5 of the Supplementary Materials.
L EARNING C URVES BETWEEN TWO TASKS
In this section, we analyze continual learning in the NTK regime (4) between two tasks ( K = 2 ).
One can also regard this setting as transfer learning.
We sequentially train the model from task A to task B, and each one has a target function deﬁned by
[Equation]
The target functions are dependent on each other and belong to the reproducing kernel Hilbert space (RKHS).
By denoting the RKHS by H , one can interpret the target similarity ρ as the inner product (cid:104) ¯ f A , ¯ f B (cid:105) H / ( (cid:107) ¯ f A (cid:107) H (cid:107) ¯ f B (cid:107) H )
= ρ .
These targets have dual representation such as ¯ f ( x ) =
(cid:80)
i α i Θ( x (cid:48)
i , x ) with i.i.d.
Gaussian variables
α i (Bordelon et al., 2020), as summarized in Section
E.1.
We generate training samples by y µA =
¯ f A ( x µA ) + ε µA ( µ [Equation] 1 , ..., N A ) and y µB =
¯ f B ( x µB ) +
ε µB ( µ = 1 , ..., N B ), although we focus on the noise-less case ( σ = 0 ) in this section.
Input samples x µ A and x µ B are i.i.d.
and generated by the same distribution p ( x ) .
We can measure the generalization error in two ways: generalization error on subsequent task B and that on previous task A:
[Equation]
where we write f 2 as f A → B to emphasize the sequential training from A to B.
The notation E backA → B ( ρ ) is referred to as backward transfer (Lopez-Paz & Ranzato, 2017).
Large negative backward transfer is known as catastrophic forgetting.
We take the average (cid:104)· · · (cid:105) over training samples of two tasks {D A , D B } , and target coefﬁcients ¯ w .
In fact, we can set ¯ w as constants, and it is unnecessary to take the average.
To avoid complicated notation, we take the average in the main text.
We then obtain the following result.
Theorem 1.
Using the replica method under sufﬁciently large N A and N B , for σ = 0 , we have
[Equation]
where we deﬁne q A,i =
κ A / ( κ A + N A
η
i ) , q B,i = κ B / ( κ B + N B η i ) , E B,i =
q 2 B,i η 2 i / (1 − γ B ) and
F γ B ( a, b ) =
b ( a − 2) + b 2 (1 − a ) / (1 − γ B ) .
The constants κ A and γ A ( κ B and γ B , respectively) are given by setting N = N A ( N B ) in (9).
The detailed derivation is given in Section A.
Technically speaking, we use the following lemma:
Lemma 2.
Denote the trained model (2) on single task A as f A = (cid:80) i w ∗ A,i ψ i , and deﬁne the following cost function: E = (cid:10)(cid:80)
i φ i ( w ∗ A,i − u i ) 2 (cid:11)
D A for arbitrary constants φ i and u i .
Using the replica method under a sufﬁciently large N A , we have
E =
[Equation]
where E A denotes generalization error E 1 on single task A.
For example, we can see that E A is a special case of E with φ i
= η i and u = ¯ w A , and that E A → B ( ρ ) is reduced to u = ¯ w B and φ i =
q 2 B,i η 2 i / (1 − γ B ) after certain calculation.
Spectral Bias.
The generalization errors obtained in Theorem 1 are given by the summation over spectral modes like the single-task case.
The E B,i corresponds to the i -th mode of generalization error (8) on single task
B. The study on the single task (Bordelon et al., 2020) revealed that as the sample size increases, the modes of large eigenvalues decreases ﬁrst.
This is known as spectral bias and clariﬁes the inductive bias of the NTK regression.
In more detail, they showed that when D is sufﬁciently large and N B = αD l ( α > 0 , l ∈ N ), the error of each mode is asymptotically given by
E B,i =
( i < l ) and η 2 i ( i > l ) .
We see that the spectral bias also holds in E A → B because it is a weighted sum over E B,i .
In contrast, E backA → B includes a constant term 2(1 − ρ ) η 2 i .
This constant term causes catastrophic forgetting, as we show later.
4. T RANSITION FROM NEGATIVE TO POSITIVE TRANSFER
We now look at more detailed behaviors of generalization errors obtained in Theorem 1.
We ﬁrst discuss the role of the target similarity ρ for improving generalization.
Figure 1(a) shows the comparison of the generalization between single-task training and sequential training.
Solid lines show theory, and markers show experimental results of trained neural networks in the NTK regime.
We trained the model (1) with ReLU activation, L = 3 , and M l = 4 , 000 by using the gradient descent over 50 trials.
More detailed settings of our experiments are summarized in Section E.2.
Because we set N A = N B = 100 , we have E A = E B .
The point is that both E A → B ( ρ ) and
E backA → B ( ρ ) are lower than E A (= E B ) for large ρ .
This means that the sequential training degrades generalization if the targets are dissimilar, that is, negative transfer.
In particular, E backA → B ( ρ ) rapidly deteriorates for the dissimilarity of targets.
Note that both E A → B and E backA → B are linear functions of ρ .
Figure 1(a) indicates that the latter has a large slope.
We can gain quantitative insight into the critical value of ρ for the negative transfer as follows.
Knowledge transfer.
The following asymptotic equation gives us the critical value for E A → B :
[Equation]
[Figure]
A straightforward algebra leads to this (Section A.3).
In the context of transfer learning, it is reasonable that the target domain has a limited sample size compared to the ﬁrst task.
For ρ > 1 / 2 , we have E A → B < E B , that is, previous task
A contributes to improving the generalization performance on subsequent task B (i.e., positive transfer).
For ρ < 1 / 2 , however, negative transfer appears.
The following sufﬁcient condition for the negative transfer is also noteworthy.
By evaluating E A → B >
E B , we can prove that for any N A and N B , the negative transfer always appears for
[Equation]
This is just a sufﬁcient condition and may be loose.
For example, we asymptotically have the critical target similarity ρ = 1 / 2 > ρ ∗ for N A (cid:29) N B .
Nevertheless, this sufﬁcient condition is attractive in the sense that it clariﬁes the unavoidable negative transfer for the small target similarity.
Backward transfer.
The E backA → B (0) includes the constant term (cid:80) i
η 2 i independent of sample sizes.
Note that q A and q B decrease to zero for large sample sizes (Bordelon et al., 2020), and we have
E backA → B ( ρ ) ∼ (cid:80) i
η 2 i (1 − ρ ) .
In contrast, E A converges to zero for a large N A .
Therefore, the intersection between E backA → B ( ρ ) and E A reach ρ = 1 as N A and N B increase.
This means that when we have a sufﬁcient number of training samples, negative backward transfer ( E backA → B ( ρ ) > E A ) occurs even for very similar targets.
Figure 1(a) conﬁrms this result, and Figure 6 in Section A.6 shows more detailed learning curves of backward transfer.
Catastrophic forgetting seems literally ”catastrophic” in the sense that the backward transfer rapidly deteriorates by the subtle target dissimilarity.
4. S ELF - KNOWLEDGE TRANSFER
We have shown that target similarity is a key factor for knowledge transfer.
We reveal that the sample size is another key factor.
To clarify the role of sample sizes, we focus on the same target function ( ρ = 1) in the following analysis.
We refer to the knowledge transfer in this case as self-knowledge transfer to emphasize the network learning the same function by the same head.
As is the same in ρ < 1 , the knowledge obtained in the previous task is transferred as the network’s initialization for the subsequent training and determines the eventual performance.
Positive transfer by equal sample sizes.
We ﬁnd that positive transfer is guaranteed under equal sample sizes, that is, N A = N B .
To characterize the advantage of the sequential training, we compare it with a model average: ( f A + f B ) / 2 , where f A ( f B ) means the model obtained by a single-task training on A (B).
Note that since the model is a linear function of the parameter in the NTK regime, this average is equivalent to that of the trained parameters: ( θ A + θ B ) / 2 .
After straightforward calculation in Section D, the generalization error of the model average is given by
[Equation]
Sequential training and model average include information of both tasks A and B; thus, it is interesting to compare it with E A → B .
We ﬁnd
Proposition 3.
For ρ = 1 and any N A = N B ,
E A → B (1) < E ave < E A = E B .
The derivation is given in Section D.
This proposition clariﬁes the superiority of sequential training over single-task training and even the average.
The ﬁrst task contributes to the improvement on the second task; thus, we have positive transfer.
Negative transfer by unbalanced sample sizes.
While equal sample size leads to positive transfer, the following unbalanced sample sizes cause the negative transfer of self-knowledge:
[Equation]
The derivation is based on Jensen’s inequality (Section A.3).
While E A → B and E B asymptotically decrease to zero for the large N B , their ratio remains ﬁnite.
Because 0 < γ A < 1 , E A → B (1) > E B .
It indicates that the small sample size of task A leads to a bad initialization of subsequent training and makes the training on task B hard to ﬁnd a better solution.
Figure 1(b) summarizes the learning curves which depend on sample sizes in a highly non-linear way.
Solid lines show theory, and markers show the results of NTK regression over 100 trials.
The ﬁgure shows excellent agreement between theory and experiments.
Although we have complicated transitions from positive to negative transfer, our theoretical analyses capture the basic characteristics of the learning curves.
For self-knowledge transfer ( ρ = 1 ), we can achieve positive transfer at
N A /N
B = 1 , as shown in Proposition 3, and for large N A /N B , as shown in (15).
In contrast, negative transfer appears for small N A /N B , as shown in (19).
For ρ < 1 , large N A /N
B produces positive transfer for ρ < 1 / 2 , as shown in (15).
If we set a relatively large σ > 0 , the learning curve may become much more complicated due to multiple descent.
Figure 5 in Section A.5 conﬁrms the case in which multiple descent appears in E A → B .
The curve shape is generally characterized by the interaction among target similarity, self-knowledge transfer depending on sample size, and multiple descent caused by the noise.
4. S ELF - KNOWLEDGE FORGETTING
We have shown in Section 4.1 that backward transfer is likely to cause catastrophic forgetting for ρ < 1 .
We show that even for ρ = 1 , sequential training causes forgetting.
That is, the training on task B degrades generalization even though both tasks A and B learn the same target.
We have E backA → B (1) =
E A → B (1) by deﬁnition, and Proposition 3 tells us that E A → B < E A .
Therefore, no forgetting appears for equal sample sizes.
In contrast, we have
E A → B (1) /E A ∼ 1 / (1 − γ B ) for N A (cid:29) N B .
One can obtain this asymptotic equation in the same manner as (19) since E A → B (1) is a symmetric function in terms of indices A and B. Combining (20) with (15), we have E A <
E A → B (1) (cid:28) E B .
Sequential training is better than using only task B, but training only on the ﬁrst task is the best.
One can say that the model forgot the target by task B despite learning the same target.
We call this self-knowledge forgetting.
In the context of continual learning, many studies have investigated the catastrophic forgetting caused by different heads (i.e., in the situation of incremental learning).
Our results suggest that even the sequential training on the same task and the same head shows such forgetting.
Intuitively speaking, the self-knowledge forgetting is caused by the limited sample size of task
B. Note that we have
E A → B (1) = (cid:80) i q 2 A,i E B,i / (1 − γ A ) .
The generalization error of single-task training on task B ( E B,i ) takes a large value for a small N B and this causes the deterioration of E A → B as well.
Figure 2 conﬁrms the self-knowledge forgetting in NTK regression.
We set N A = N B as the red line and N B = 100 as the yellow line.
The dashed line shows the point where forgetting appears.
L EARNING CURVES OF MANY TASKS
We can generalize the sequential training between two tasks to that of more tasks.
For simplicity, let us focus on the self-knowledge transfer ( ρ = 1 ) and equal sample sizes.
Applying Lemma 2 recursively from task to task, we can evaluate generalization deﬁned by E n = (cid:10)(cid:82) dxp ( x )( ¯ f ( x ) − f n ( x )) 2 (cid:11) D .
Theorem 4.
Assume that (i) ( X n , y n ) ( n = 1 , ..., K ) are given by the same distribution X n ∼ P
( X )
and target y n = (cid:80) i ¯
w i ψ ( X n ) + ε n .
(ii) sample sizes are equal: N n = N .
For n = 1 , ..., K ,
[Equation]
where q i = κ/ ( κ + η i N ) , ˜ q i =
η i
q 2 i and diag ( q 2 ) denotes a diagonal matrix whose entries are q 2 i .
The noise term R n is a positive constant.
In the noise-less case ( σ = 0 ), the learning curve shows monotonic decrease: E n +1 ≤ E n .
If all eigenvalues are positive, we have
[Equation]
See Section B for details of derivation.
The learning curve (i.e., generalization error to the number of tasks) monotonically decreases for the noise-less case.
the monotonic decrease comes from λ max ( Q ) < 1 .
This result means that the self-knowledge is transferred and accumulated from task to task and contributes in improving generalization.
It also ensures that no self-knowledge forgetting appears.
We can also show that R n converges to a positive constant term for a large n and the contribution of noise remains as a constant.
KRR-like expression.
The main purpose of this work was to address the generalization of the continually trained model f n .
As a side remark, we show another expression of f n : [Θ (
[Equation]
(cid:48)
[Equation]
[Equation] (cid:48) [Equation] [Equation]

[Equation]
O
[Equation]
O Θ [Equation] X 1 ) Θ ( X 2 ) ... O Θ ( X n , X 1 ) · · · Θ ( X n )   − 1   y 1 .
y n   .
This easily comes from comparing the update (4) with a formula for the inversion of triangular block matrices (Section C).
One can see this expression as an approximation of KRR, where the upper triangular block of NTK is set to zero.
Usual modiﬁcation of KRR is diagonal, e.g., L2 regularization and block approximation, and it seems that the generalization error of this type of model has never been explored.
Our result revealed that this model provides non-trivial sample size dependencies such as self-knowledge transfer and forgetting.
5. E XPERIMENTS
Although Theorem 4 guarantees the monotonic learning curve for equal sizes, unbalanced sample sizes should cause a non-monotonic learning curve.
We empirically conﬁrmed this below.
Synthetic data.
First, we empirically conﬁrm E n on synthetic data (10).
Figure 3(a1) conﬁrms the claim of Theorem 4 that the generalization error monotonically decreases for σ = 0 as the task number increases.
Dashed lines are theoretical values calculated using the theorem, and points with error bars were numerically obtained by f n (4) over 100 trials.
For σ > 0 , the decrease was suppressed.
We set σ 2 = { 0 , 10 − 5 , 10 − 4 , 10 − 3 } and N i = 100 .
Figure 3(a2) shows self-knowledge forgetting.
When the ﬁrst task has a large sample size, the generalization error by the second task can increase for small subsequent sample sizes N i .
For smaller N i , there was a tendency for the error to keep increasing and taking higher errors than that of the ﬁrst task during several tasks.
In practice, one may face a situation where the model is initialized by the ﬁrst task training on many samples and then trained in a continual learning manner under a memory constraint.
The ﬁgure suggests that if the number of subsequent tasks is limited, we need only the training on the ﬁrst task.
If we have a sufﬁciently large number of tasks, generalization eventually improves.
MLP on MNIST / ResNet-18 on CIFAR-10.
We mainly focus on the theoretical analysis in the
NTK regime, but it will be interesting to investigate whether our results also hold in more practical settings of deep learning.
We trained MLP (fully-connected neural networks with 4 hidden layers) and ResNet-18 with stochastic gradient descent (SGD) and cross-entropy loss.
We set the number of epochs sufﬁcient for the training error to converge to zero for each task.
We conﬁrmed that they show qualitatively similar results as in the NTK regime.
We randomly divided the dataset into tasks without overlap of training samples.
Figure 3(b1,c1) shows the monotonic decrease for an equal sample size and that the noise suppressed the decrease.
We set N i = 500 and generated the noise by the label corruption with a corruption probability { 0 , 0 .
2 , ..., 0 . 8 } (Zhang et al., 2017).
The vertical axis means Error = 1 − ( Test accuracy [%])/100.
Figure 3(b2,c2) shows that unbalanced sample sizes caused the non-monotonic learning curve, similar to NTK regression. [Figure]
D ISCUSSION
We provided novel quantitative insight into knowledge transfer and forgetting in continual learning.
Even in the NTK regime, where the model is simple and linearized, learning curves show rich and non-monotonic behaviors depending on both target similarity and sample size.
Even in training on the same target, we revealed that self-knowledge transfer is successful or forgetting appears depending on the balance of sample sizes.
This result indicates that the performance of continual learning is more complicated than we thought, but we can still ﬁnd some universal laws behind it.
There are other research directions to be explored.
While we focused on reporting novel phenomena on transfer and forgetting, it is also important to develop efﬁcient algorithms to achieve better performance.
To mitigate catastrophic forgetting, previous studies proposed several strategies such as regularization, parameter isolation, and experience replay (Mirzadeh et al., 2020).
Evaluating such strategies with theoretical backing would be helpful for further development and improve the design of continual learning.
For example, orthogonal projection methods modify gradient directions (Doan et al., 2021) and replay methods allow the reuse of the previous samples.
We conjecture that these could be analyzed by extending our calculations in a relatively straightforward way.
It would also be interesting to investigate richer but complicated situations required in practice, such as streaming of non-i.i.d.
data and distribution shift (Aljundi et al., 2019).
The current work and other theoretical works in continual learning or transfer learning basically assume the same input distribution between different tasks (Lee et al., 2021; Tripuraneni et al., 2020).
Extending these theories to the case with a input distribution shift will be essential for some applications including domain incremental learning.
Our analysis may also be extended to different topics with a similar setting as continual learning.
For example, self-distillation uses the outputs of the model for training in the subsequent phase and plays an interesting role of regularization (Mobahi et al., 2020).
While our study provides universal results, which do not depend on speciﬁc eigenvalue spectra, it is interesting to investigate individual cases depending on architecture.
Studies on NTK eigenvalues have made remarkable progress, covering shallow and deep ReLU neural networks (Geifman et al., 2020; Chen & Xu, 2020), skip connections (Belfer et al., 2021), and CNNs (Favero et al., 2021).
We expect that our analysis and ﬁndings will serve as a foundation for further understanding and development on theory and experiments of continual learning.
R EPRODUCIBILITY S TATEMENT
The main contributions of this work are theoretical claims, and we clearly explained their assumptions and settings in the main text.
Complete proofs of the claims are given in the Supplementary Materials.
For experimental results of training deep neural networks, we used only already-known models and algorithms implemented in Pytorch.
All of the detailed settings, including learning procedures and hyperparameters, are clearly explained in Section E.
Supplementary Materials A S EQUENTIAL TRAINING BETWEEN TWO TASKS A. N OTATIONS
The kernel ridge regression is given by
[Equation]
where N is the sample size and H is the reproducing kernel Hilbert space (RKHS) induced by the neural tangent kernel Θ .
We assume that the input samples x µ are generated from a probabilistic distribution p ( x ) and that the NTK has ﬁnite trace, i.e., (cid:82) dxp ( x )Θ( x, x ) < ∞ .
Mercer’s decomposition of Θ is expressed by
[Equation]
In other words, we have Θ( x (cid:48) , x ) = (cid:80) ∞ i =
η i φ i ( x (cid:48) )
φ i ( x ) .
Note that a function belonging to RKHS is given by
[Equation]
with (cid:107) f (cid:107) 2 H = (cid:80) ∞
i =
a 2 i /η i < ∞ .
Then, using the orthonormal bases { φ i } ∞ i =0 , the solution of the regression is given by
[Equation]
where ψ i ( x ) := √ η i φ i ( x ) .
Each column of Ψ is given by φ ( x µ ) ( µ = 1 , ..., N ).
We focus on the
NTK regime and take the ridge-less limit ( λ → 0 ).
The sequentially trained model (4) between tasks A and B is written as
[Equation]
The model f A is trained on single task A and represented by
[Equation]
with
[Equation]
Eq.
(S.9 ) is the objective function equivalent to (S.1 ) because (cid:104)
f A , f A (
cid:105) H
= (cid:107) w A (cid:107) 22 .
Similarly, we can represent f A → B by the series expansion with the bases of RKHS.
Note that the right-hand side of (S.5 ) is equivalent to the kernel ridge-less regression on task B’s input samples and labels y B − f A ( B ) .
We have
[Equation]
10) with
[Equation]
A. P ROOF OF T HEOREM 1. A.2. K NOWLEDGE TRANSFER E A → B
First, we take the average over training samples of task B conditioned by task A, that is,
[Equation]
where the set of task B’s training samples is denoted by D B .
We have
[Equation]
Note that the objective function H ( w B , w ∗ A ) is transformed to
[Equation]
Comparing (S.14 ) and (S.16 ), one can see that the average over task B conditioned by task A is equivalent to single-task training with the target function ( ¯ w B − w ∗ A ) (cid:62) ψ ( x ) .
Therefore, by using (8), we immediately obtain
[Equation]
Next, we take the average of (S.18 ) over task A.
Only ¯ w ∗ A depends on the task A and is determined by the single-task training (S.8 ).
This corresponds to Lemma 2 with u = ¯ w B and φ i =
η i q 2 B,i / (1 − γ B ) .
We obtain
[Equation]
Although this is a general result that holds for any ¯ w A and ¯ w B , it is a bit complicated and seems not easy to give an intuitive explanation.
Let us take the average over
[Equation]
[Equation]
The generalization error is then simpliﬁed to
[Equation]
S.22) where we used (cid:104) w 2 A,i (cid:105) w = (cid:104) w 2 B,i (cid:105) w = η i and (cid:104) w A
i , w B,i (cid:105)
w = ρ .
A.2. B ACKWARD TRANSFER
Backward transfer is measured by the prediction on the previous task A:
[Equation]
First
[Equation]
we take the average over task B,
[Equation]
This corresponds to Lemma 2 with the replacement φ i ← η i and u ← ¯ w A − w ∗ A .
Recall that the objective function of NTK regression was given by ( S. 16) .
The target ¯ w A in Lemma 2 is replaced as ¯ w A ← ¯ w B − w ∗ A .
We then have
[Equation]
Next, we take the average over D A .
Since the ﬁrst and third terms of (S.27 ) are independent of
D A , we need to evaluate only the second term.
The second term corresponds to Lemma 2 with φ i =
η i q 2 B,i / (1 − γ B ) and u i = ¯ w B,i − (1 − γ B ) /q
B,i ( ¯ w B,i − ¯ w A,i ) .
We obtain
E
[Equation]
+  
[Equation] [Equation]
(cid:88)
[Equation]
)   q
[Equation]
B
[Equation]
(cid:35)
η
[Equation]
1 − γ
[Equation]
+ σ(cid:32)
− γ A 1 − γ B
N A κ 2 A (cid:88) i
η 2 i q 2 A
[Equation]
i q 2 B
[Equation]
i + γ
[Equation]
1 − γ B (cid:33) .
(S.28)
Finally, by taking the average over (S.20 ), we have
[Equation]
A.2.
L EMMA
Lemma 2.
Suppose training on single task A, the target of which is given by ¯ f ( x )
= (cid:80) i ¯ w A,i ψ
i ( x ) , and denote the trained model (2) as f ∗ = (cid:80) i w ∗ A,i ψ i .
Deﬁne the following cost function:
[Equation]
for arbitrary constants φ i and u i .
Using the replica method under a sufﬁciently large N A , we have
[Equation]
Proof.
The process of derivation is similar to Canatar et al.
, but we have additional constants Φ and c .
They cause several differences in the detailed form of equations.
Deﬁne
[Equation]
) where E ( w A ) = (cid:10)(cid:80) i φ i ( w A,i − u i )
2 (cid:11) D A .
We omit the index A in the following.
We have
[Equation]
To evaluate (cid:104) log Z (cid:105) , we use the replica method (a.k.a.
replica trick):
[Equation]
The point of the replica method is that we ﬁrst calculate Z n for n ∈ N then take the limit of n to zero by treating it as a real number.
In addition, we calculate the average (cid:104) Z n (cid:105) D under a replica symmetric ansätz and a Gaussian approximation by following the calculation procedure of the previous works (Dietrich et al., 1999; Bordelon et al., 2020; Canatar et al., 2021).
We have
[Equation]
where we deﬁne dW n = (cid:81) na =1 dw a and Φ is a diagonal matrix whose diagonal entries given by φ i .
We take the shift of w a → w a + ¯ w .
Then,
[Equation]
¯ w − u ) .
[Equation]
First, let us calculate (cid:104) Q (cid:105) { x µ ,ε µ } .
This term is exactly the same as appeared in the previous works (Bordelon et al., 2020; Canatar et al., 2021).
To describe notations, we overview their derivation.
Deﬁne q a = w a (cid:62) ψ ( x ) + ε , which are called order parameters.
We approximate the probability distribution of q a by a multivariate Gaussian:
[Equation]
with
[Equation]
where Σ ab = σδ ab , (cid:104)· · · (cid:105) x denotes the average over p ( x ) and Λ is a diagonal matrix whose entries are η i .
We have √ η 0 w a 0 in (S.41 ) since η 0 corresponds to the constant shift φ 0 ( x ) = 1 .
Training samples are i.i.d.
and we omitted the index µ .
We then have
[Equation]
where I denotes the identity matrix.
Deﬁne conjugate variables { ˆ µ a , ˆ C ab } by the following identity:
[Equation]
where C denotes an uninteresting constant and we took the conjugate variables on imaginary axes.
Next, we perform the integral over W n .
Eq.
(S.37 ) becomes
[Equation]
where d Ω ab = dµ a d ˆ µ a dC ab
d ˆ C ab and deﬁne
[Equation]
We can represent (cid:82) dW n exp( − G S ) by
[Equation]
where x i ∈ R n denotes a vector [ w 1 i , ..., w ai , ..., w ni ] (cid:62) and i is the index of kernel’s eigenvalue mode ( i = 0 , 1 , ... ) .
We deﬁned
[Equation]
where I n is an n × n identity matrix and 1 n is an n -dimensional vector whose all entries are 1 .
The
G S term includes φ and c that are speciﬁc to our study.
When φ = η and u = ¯ w , it is reduced to previous works (Bordelon et al., 2020; Canatar et al., 2021).
Taking the integral over { x i } , we have
[Equation]
That is,
[Equation]
R EPLICA SYMMETRY AND SADDLE - POINT METHOD
Next
[Equation]
we carry out the integral (S.47 ) by the saddle-point method.
Assume the replica symmetry:
[Equation]
The following three terms are the same as in the previous works:
[Equation]
where 11 (cid:62) denotes a matrix whose all entries are 1.
Regarding the leading term of order n ( n → 0 ),
[Equation]
Furthermore.
we have
[Equation]
After straightforward algebra, the leading terms become
[Equation]
with
[Equation]
Substituting these leading terms into (S.45 )
[Equation]
we obtain (cid:104) Z n (cid:105) D
= C (cid:82) dθ exp( − nNS ( θ )) with
[Equation]
where θ denotes a set of variables { r, ˆ r, c, ˆ c, µ, ˆ µ } .
Here, we take N (cid:29) 1 and use the saddle-point method.
We calculate saddle-point equations ∂S ( θ ∗ ) /∂θ
=
and obtain
[Equation] [Equation]
From (S.67 ) and (S.68 ),
[Equation]
From (S.70 ) and (S.71 ), we also have
[Equation]
Substituting (S.64 ) and (S.72 ) into (S.73 ), We obtain the implicit function for κ :
[Equation]
S.
[Equation]
Using κ and W , the saddle-point equations become
[Equation]
Substituting back these quantities into (S.65 ), we obtain
[Equation]
Recall (cid:104) Z n (cid:105) D ≈ exp( − nNS ( θ ∗ )) .
We have (cid:104)
[Equation]
(S
[Equation]
G ENERALIZATION ERROR
We evaluate
[Equation]
Note that W, g, κ and ¯ k depend on J .
At the point of J = 0 ,
[Equation] [Equation]
[Equation]
Substituting (S.83 -S.86 ) into (S.82 ), we obtain
E = (cid:88) i =
[Equation]
87) where ξ 0 is
[Equation]
with W 0 = κ + Nη 0 .
When η 0 = 0 , we have ξ 0 = 0 .
Note that the additional term ξ 0 is not speciﬁc to our case but also appeared in the previous work (Canatar et al., 2021).
We have ξ
= O (1 /η 0 ) for a large η 0 and ξ 0 is often negligibly small.
A. E QUATIONS FOR UNDERSTANDING NEGATIVE TRANSFER A.3. D ERIVATION OF EQUATION
First, let us evaluate the term including q A,i = κ
A / ( κ A + N A η i ) .
Note that κ A is a monotonically decreasing function of N A because
[Equation]
which comes from the implicit function theorem.
Let us write κ A at a ﬁnite N A
= c as κ A ( c ) .
We then have
[Equation]
for N A > c .
Next, we evaluate
[Equation]
The second term is negligibly small for a sufﬁciently large N A > c because
[Equation]
Note that we have 1 − γ = κ (cid:80) i =
η i / ( κ + η i N ) 2 .
The third term is (cid:88)
[Equation]
where we used q B,i ≤ 1 .
It decreases to zero as N A increases.
Thus, we have E A → B ( ρ ) ∼ 2(1 − ρ ) E B .
A.3. D ERIVATION OF EQUATION
[Equation]
where the second line comes from Jensen’s inequality.
Using (S.90 ), we have
[Equation]
96) for N B ≥ c .
Since we assumed the ﬁnite trance of NTK (i.e., (cid:80) i η i < ∞ ), the left-hand side of (S.96 ) is ﬁnite and converges to zero for a sufﬁciently large N B .
Therefore, we have
[Equation]
In contrast, E A → B /E B ≤ 1 / (1 − γ A ) since q A ≤ 1 .
Thus, the ratio is sandwiched by 1 / (1 − γ A ) .
A.3. S UFFICIENT CONDITION FOR THE NEGATIVE TRANSFER
Let us denote the i -th mode of E A → B as E A → B,i and that of E B as E B,i .
From E A → B,i > E B,i , we have
[Equation]
[Equation]
where f ( q ) is a quadratic function and takes the minimum at q ∗
= (1 − ρ )(1 − γ ) .
The condition f ( q ) > 0 holds if and only if
[Equation]
− 2 γρ + γ > 0 .
(S.99)
Solving this, we ﬁnd ρ < √ γ/ (1 + √ γ ) .
A. P ROOF OF P ROPOSITION 3.
For N A = N B , we have q A,i = q B,i = q i .
Then,
[Equation]
and the generalization error of single-task training is given by
[Equation]
[Equation]
Model average is obtained in Section D and we have E ave =
(1 − γ/ 2) E < E B because 0 < γ < 1 .
Thus, we only need to evaluate the relation between E ave and E A → B :
[Equation]
where we deﬁned
[Equation]
and used q i = 1 − a i N and γ = N (cid:80) i a 2 i .
We can provide a lower bound of F by using the following cubic function: [Equation] We have 0 ≤ a i ≤ 1 /N by deﬁnition. Let us consider a lower bound of G by using its tangent line at a i
= t/N (0 ≤ t ≤ 1) , that is, Nt (4 − 3 t ) a i − 2 t 2 (1 − t ) . Deﬁne [Equation] [Equation]
t )) .
We have H (0) =
2 t 2 (1 − t ) ≥ 0 .
Since H (1 /N )
{+= − 2( t − 1) 2 ( t − 1 / 2) , we need t ≤ 1 / 2 to guarantee H ( a i ) ≥ 
+}
for all 0 ≤ a i ≤ 1 /N .
Here, note that H is a cubic function of a i and has two ﬁxed points a i = t/N and (4 − 3 t ) / (3 N ) .
We can see that for t ≤ 1 / 2 , H has the local minimum at a i = t/N and
[Equation]
{+Therefore, we have H ( a i ) ≥ 
+}
for all 0 ≤ a i ≤ 1 /N when the ﬁxed constant t satisﬁes t ≤ 1 / 2 .
Thus, we have the lower bound of G :
[Equation]
[Equation]
Using this lower bound, we have
[Equation] [Equation]
) where we used γ = N (cid:80) i a 2 i and (cid:80) i a i = 1 .
By setting t =
γ/ 2 , we obtain
F
[Equation]
γ [Equation]
[Equation] [Equation] = 0 . (S.109)
Therefore, we have E ave >
E A → B .
We overview the multiple descent of E 1 investigated in Canatar et al.
It appears for σ > 0 .
Let us set N B = αD l ( α > 0 , l ∈ N , D
(cid:29) 1 ), which is called the l -th learning stage.
We have E B,i =
( i < l ) , E B,l ( α ) and
η 2 i ( i > l ) .
E B,l ( α ) is a function of α , and becomes a one-peak curve depending on the noise and the decay of kernel’s eigenvalue spectrum.
Because each learning stage has a one-peak curve, the whole learning curve shows multiple descents as the sample size increases.
Roughly speaking, the appearance of the multiple descent is controlled by γ.
The γ is determined by the kernel’s spectrum and sample size N .
For example, previous studies showed that if we assume the l -th learning stage, γ is a non-monotonic function of α , the maximum of which is γ = [Equation] / ( (cid:112) [Equation]
) 2 for a constant ¯ [Equation] where ¯ η is a normalized eigenvalue of the kernel (Canatar et al., 2021) This tells us that γ repeat the increase and decrease depending on the increase of the learning stage.
Roughly speaking, this non-monotonic change of γ leads to the multiple descent.
In sequential training, E B,l ( α ) can similarly cause multiple descent as in single tasks because E A → B is a weighted summation over E B,i .
Figure 5 is an example in which noise causes multiple decreases and increases depending on the sample sizes.
We set ρ = 1 and obtained the theoretical values by (S.22 ).
While E A → B (1) is a symmetric function of indices A and B for σ = 0 , it is not for σ > 0 .
Depending on N A and N B , the learning ”surface” becomes much more complicated than the learning curve of a single task.
A. A DDITIONAL EXPERIMENT ON BACKWARD TRANSFER
Figure 6 shows the learning curves of backward transfer.
Solid lines show theory, and markers show the mean and interquartile range of NTK regression over 100 trials.
The ﬁgure shows excellent agreement between theory and experiments.
As is shown in Section 4.1, subtle target dissimilarity ( ρ < 1 ) causes negative backward transfer (i.e., catastrophic forgetting). For a large N A , E backA → B ( ρ < 1) approaches a non-zero constant while E A approaches zero. Thus, we have E backA → B > E A even for the target similarity close to 1. When sample sizes are unbalanced ( N A (cid:29) N B ), the learning curve shows negative transfer even for ρ = 1 .
This is the self-knowledge forgetting revealed in (20).
The ratio E back A → B (1) /E
A takes a constant larger than 1, that is, 1 / (1 − γ B ) .
[Figure]
We changed N A and set N B = 10 2 .
We set D = 20 and other experimental details are the same as in Figure 1(b).
B P ROOF OF T HEOREM 4. B. L EARNING CURVE OF MANY TASKS
Eq.
(4) is written as
[Equation]
Each term of this summation is equivalent to the kernel ridge-less regression on input samples X k and labels y k − f k − 1 ( X k ) .
Therefore, we can represent f n by
[Equation]
with the minimization of the objective function H :
[Equation]
[Equation]
(S
[Equation]
The generalization error is
[Equation]
S
[Equation]
where D = {D 1 , ..., D n } .
Since the data samples are independent of each other among different tasks, we can apply Lemmasequentially from w n +1 to w 1 .
First, we take the average over the ( n + 1) -th task, that is,
[Equation]
This corresponds to Lemma 2 with φ = η and u = ¯ w A = ¯ w − (cid:80) nk
=1 w ∗ k .
We have
[Equation]
with
[Equation]
Here, κ n , γ n and q n,k are determined by N n .
Next, we take the average over the n -th task:
[Equation]
This corresponds to Lemma 2 with φ = φ 1 and u = ¯ w A
= ¯ w − (cid:80) n − 1 k
=1 w ∗ k .
We obtain
[Equation]
with
[Equation]
Similarly, we can take the averages from the ( n − 1) -th task to the ﬁrst task, and obtain
[Equation]
) for m = 1 , ..., n , and
[Equation]
They are general results for any N n .
By setting N n = N for all n , we can obtain a simpler formulation.
In a vector representation, φ n +1 is explicitly written as
[Equation]
Finally, by taking the average over ¯ w i ∼ N (0 , η i ) which is the one-dimensional case of (S.20 ), we have
[Equation]
with
[Equation]
B.
M ONOTONIC DECREASE OF E n
For σ 2 = 0 , we have
[Equation]
Note that Q is a positive semi-deﬁnite matrix.
If the maximum eigenvalue is no greater than 1, E n is monotonically non-increasing with n .
We denote the inﬁnite norm as (cid:107) A (cid:107) ∞ =
max i (cid:80) j | A ij | .
Because the inﬁnite norm bounds the eigenvalues, we have
[Equation]
where G ( η ) is monotonically decreasing for η ≥ 0 and G (0)
= 1 .
Therefore, the largest eigenvalue λ 1 ( Q ) is upper-bounded by 1 , and E n becomes monotonically decreasing.
In particular.
if η i >for all i , we have λ 1 ( Q ) < 1 and obtain E n +1 < E n .
For λ 1 ( Q ) < 1 , it is also easy to check that the series (S.130 ) in R n +1 converges to a constant for large n .
C D ERIVATION OF KRR- LIKE EXPRESSION
KRR-like expression comes from the inverse formula of a block triangular matrix.
Let us write (23) as
[Equation]
where Z n is a block triangular matrix recursively deﬁned by
Z n = (cid:20)
[Equation]
O
[Equation]
n,
[Equation]
( n
[Equation]
[Equation]
[Equation]
S.135) Then, (S.134 ) is transformed to [
[Equation]
[Equation]
[Equation]
Thus, one can see that the KRR-like expression is equivalent to our continually trained model.
D D ERIVATION OF MODEL AVERAGE
For clarity, we set σ = 0 , N A = N B , and ρ = 1 (that is, ¯ w A = ¯ w B ).
Generalization error of a model average is expressed by
[Equation]
We can evaluate E ave in a similar way to E A → B .
First, take the average over D B .
We deﬁne
[Equation]
This average corresponds to Lemma 2 with replacement from the target task A to B, φ ← η and u ← 2 ¯ w B − w A .
Then, we have
[Equation]
Next, we take the average over D A .
The ﬁrst term of (S.144 ) corresponds to Lemma 2 with replacement φ ← η and u ← (1 + q B,i ) ¯ w B .
Then, we get
[Equation]
For N A = N B , we have q A = q B , γ A = γ B , and E A = E B .
In addition, since we focused on ¯ w A = ¯ w B , we have
[Equation]
E E XPERIMENTAL SETTINGS E. D UAL REPRESENTATION
In dual representation, the targets (10) are given by
[Equation]
for a sufﬁciently large P (cid:48) .
We sampled x (cid:48) from the same input distribution p ( x ) and set P (cid:48)
= 10in all experiments.
It is easy to check that targets (10) and (S.150 ) are asymptotically equivalent for a large P (cid:48) .
Because Θ( x (cid:48) , x ) = (cid:80) i ψ i ( x (cid:48) )
ψ i ( x ) , we have (cid:80) P (cid:48)
j α A,j Θ( x (cid:48)
j , x ) = (cid:80) i (cid:80) P
[Equation]
j ( α A,j ψ i ( x (cid:48) j ))
ψ i ( x ) ∼ (cid:80) i ¯ w i ψ i ( x ) , where ¯ w i is sampled from N (0 , η i ) .
E. S UMMARIES ON EXPERIMENTAL SETUP
Figure 1(a).
We trained the deep neural network (1) with ReLU, L = 3 , and M l = 4 , 000 by the gradient descent.
We set a learning rate 0 .
5 and trained the network during 10,000 steps over 50 trials with different random seeds.
The target is given by (S.150 ) with C = 1 , D = 10 , and Gaussian input samples x i ∼ N (0 , 1) .
We initialized the network with ( σ 2 w , σ 2 b ) =
(2 , 0) , set N A = N B = 100 , and calculated the generalization error over 4 , 000 samples.
Each marker shows the mean and interquartile range over the trials.
To calculate the theoretical curves, we need eigenvalues η i of the NTK.
We numerically compute them by the Gauss-Gegenbauer quadrature following the instruction of Bordelon et al. Its implementation is also given by Canatar et al.
Since the input samples are deﬁned on a hyper-sphere S d − 1 and the NTK is a dot product kernel (i.e., Θ( x (cid:48) , x ) can be represented by Θ( x (cid:48)(cid:62) x ) ), the NTK can be decomposed into Gegenbauer polynomials:
[Equation]
where { Q i } are the Gegenbauer polynomials and N ( d, i ) are constants depending d and i .
Since the
Gegenbauer polynomials are orthogonal polynomials, we have [Equation] for a certain constant c and dτ ( z ) = (1 − z [Equation]
[Equation]
/ 2 dz .
The Gauss-Gegenbaur quadrature approximates this integral by
[Equation] where w j are constant weights and z j are the r roots of Q r ( z ) . The deﬁnitions of constants ( N ( d, i ) , c and w j ) are given in Section 8 of Bordelon et al. and we set r = 1000 as is the same in this previous work.
We computed η i ( i = 1 , .., 1000 ) and substituted them to the analytical expressions.
Figure the results of NTK regression (2) for the single task and (4) for sequential training.
We set N B = 4 , 000 , D = 50 and other settings were the same as in Figure 1(a) regression D = settings are the same a
show
the means and error bars over 100 trials.
Other settings were the same as in Figure 1(a).
(MLP)
We trained the deep neural network (1) with ReLU, L = 5 , and M l = 512 by SGD with a learning rate 0.001, momentum 0.9, and mini-batch size 32 over 10 trials with Pytorch seeds 1-10.
We set the number of epochs to 150 and scaled the learning rate × 1 / 10 at the 100-th epoch, conﬁrming that the training accuracy reached 1 for each task.
(ResNet-18) We trained ResNet-18 by SGD with a learning rate 0.01, momentum 0. and mini-batch size 128 over 10 trials with Pytorch seeds 1-10.
We set the number of epochs to 150 and scaled the learning rate × 1 / 10 at the 100-th epoch, conﬁrming that the training accuracy reached 1 for each task.
Note that the purpose of these experiments was not to achieve performance comparable to state-ofthe-art but to conﬁrm self-knowledge transfer and forgetting.
Therefore, we did not apply any data augmentation or regularization method such as weight decay and dropout.