Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
Anonymous Authors
Abstract
In continual learning settings, deep neural networks are prone to catastrophic forgetting.
Orthogonal Gradient Descent (Farajtabar et al., 2019) achieves state-of-the-art results in practice for continual learning, although no theoretical guarantees have been proven yet.
We derive the ﬁrst generalisation guarantees for the algorithm
OGD for continual learning, for overparameterized neural networks.
We ﬁnd that OGD is only provably robust to catastrophic forgetting across a single task.
We propose OGD+, prove that it is robust to catastrophic forgetting across an arbitrary number of tasks, and that it veriﬁes tighter generalisation bounds.
The experiments show that OGD+ outperforms OGD on settings with long range memory dependencies, even though the models are not overparameterized.
Also, we derive a closed form expression of the learned models through tasks, as a recursive kernel regression relation, which captures the transferability of knowledge through tasks.
Finally, we quantify theoretically the impact of task ordering on the generalisation error, which highlights the importance of the curriculum for lifelong learning.
1. Introduction
Continual learning is a setting in which an agent is exposed to multiples tasks sequentially (Kirkpatrick et al., 2016).
The core challenge lies in the ability of the agent to learn the new tasks while retaining the knowledge acquired from previous tasks.
Too much plasticity will lead to catastrophic forgetting, which means the degradation of the ability of the agent to perform the past tasks (McCloskey & Cohen 1989,
Ratcliff 1990, Goodfellow et al. 2014).
On the other hand, too much stability will hinder the agent from adapting to new tasks.
Recent works on the Neural Tangent Kernel (Jacot et al., 2018) and on the convergence of Stochastic Gradient Descent for overparameterized neural networks (Arora et al., 2019) have unlocked powerful tools to analyze the training dynamics of over-parameterized neural networks.
We leverage these theoretical ﬁndings in order to to prove guarantees on the convergence and the generalisation of the algorithm,
Orthogonal Gradient Descent for Continual Learning (Farajtabar et al., 2019).
Our contributions are summarized as follows:
We provide closed form expressions of the functions learned across tasks.
We ﬁnd that they can be expressed as a linear combination of kernel regressors, over the previously seen tasks.
The relationship also captures task similarity and the transferability of knowledge across tasks through the NTK (Sec. 3, Theorem 1).
We prove the ﬁrst generalisation bound for continual learning with OGD, to our knowledge.
We derive bounds for within-task and outside-task generalisation.
We ﬁnd that generalisation through time depends on a task similarity with respect to the NTK, which we quantify rigorously (Sec. 4, Theorem 2).
We prove that OGD is robust to forgetting with respect to the previous task only (Sec. 4, Lemma 1).
We build-up on this insight to propose OGD+ (Sec. 5, Alg. 1), an extension of OGD, which we prove robust to catastrophic forgetting across an arbitrary number of tasks (Sec. 5, Lemma 3).
We also prove tighter generalisation bounds than OGD (Sec. 5, Theorem 3).
As a side result, we ﬁnd that Lemma 2 also quantiﬁes the impact of the learning curriculum on the generalisation error.
We ﬁnd that NTK task dissimilarity impacts negatively generalisation, and that an ordering of tasks that minimises dissimilarity between neigh - bouring tasks leads to a tighter generalisation bound.
(Sec. 2, Lemma 2).
Even though the analysis relies on the assumption that the model is overparametrised, the analysis leads to prac- tical insights to develop OGD+.
Experiments in nonoverparametrised settings on the MNIST and CIFARbenchmarks show that OGD+ outperforms OGD on settings with long range memory dependencies (Sec. 6).
2. Preliminaries
Notation We use bold-faced characters for vectors and matrices.
We use (cid:107)·(cid:107) to denote the Euclidian norm of a vector or the spectral norm of a matrix, and (cid:107) · (cid:107) F to denote the Frobenius norm of a matrix.
We use (cid:104)· , ·(cid:105) for the Euclidian dot product, and (cid:104)· , ·(cid:105)H the dot product in the Hilbert space H .
We index the the task ID by τ .
The ≤ operator if used with matrices, corresponds to the partial ordering over symetric matrices.
We denote N the set of natural numbers, R the space of real numbers and N (cid:63) for the set N (cid:114) { 0 } .
We use ⊕ to refer to the direct sum over Euclidian spaces.
2.1. Continual Learning
Continual learning considers a series of tasks {T 1 , T 2 , . . .
} , where each task can be viewed as a separate supervised learning problem.
Similarly to online learning, data from each task is revealed only once.
The goal of continual learning is to model each task accurately with a single model.
The challenge is to achieve a good performance on the new tasks, while retaining knowledge from the previous tasks (Nguyen et al., 2018).
We assume the data from each task T τ , τ ∈ N (cid:63) , is drawn from a distribution D τ .
Individual samples are denoted ( x τ,i , y τ,i ) , where i ∈ [ n τ ] .
Also, we only consider the binary classiﬁcation setting for the sake of simplicity: x τ,i ∈
R d and y τ,i ∈ {− 1 , +1 } .
We note that it does not restrict the scope of the analysis, which can be easily extended to multiclass settings.
2.2. OGD for Continual Learning
Let T T the current task, where T ∈ N (cid:63) .
For all i ∈ [ n T ] , let v T,i = ∇ θ f (cid:63) T − 1 ( x T − 1 ,i ) , which is the Jacobian of task
T T .
We deﬁne E τ = vec( { v τ,i , i ∈
[ n τ ] } ) , which is the subspace induced by the Jacobian.
The idea behind OGD is to update the weights along the projection of the gradient on the orthogonal space induced by the Jacobians over the previous tasks E 1 ⊕ . . .
⊕ E τ − 1 .
The update rule for the task T T is as follows (Farajtabar et al., 2019):
[Equation]
The intuition behind OGD is to “preserve the previously acquired knowledge by maintaining a space consisting of the gradient directions of the neural networks predictions on previous tasks” (Farajtabar et al., 2019).
Throughout the paper, we only consider the OGD-GTL variant which stores the gradient with respect to the ground truth logit.
To prevent over-ﬁtting and guarantee the uniqueness of the global minimum in the Neural Tangent Kernel (NTK) regime, we apply a ridge regularization with a parameter λ ∈ R + .
For a task T τ , we write the corresponding loss as follows:
[Equation]
2.3. Generalisation for Continual Learning
We deﬁne within-task generalisation as the ability of the agent to acquire new knowlege and outside-task generalisation as its ability to preserve the acquired knowledge.
Deﬁnition 1 (Within-task and outside-task generalisation) Consider a loss function l : R × R
→ R .
The population loss over the distribution D , and the empirical loss over n samples D = { ( x i , y i ) , i ∈ [ n ] } from the same distribution D are deﬁned as:
[Equation]
Let T 1 , . . .
T T a sequence of tasks, and D 1 , . . .
D T their corresponding distributions.
Let f (cid:63) 1 , . . .
f (cid:63)T the trained models at each task.
Let τ ∈
[ T ] ﬁxed.
Wedeﬁne:
• within-task generalisation of the task T τ as L D τ ( f (cid:63)τ ) .
• outside-task generalisation of the task T τ with respect to a task T τ (cid:48) , where τ (cid:48) <
τ as L D τ ( f (cid:63)τ (cid:48) ) .
In practice, several works also tracked these metrics in their experiments (Kirkpatrick et al. 2016, Farajtabar et al. 2019).
2.4. Neural Tangent Kernel
In their seminal paper, Jacot et al.
(2018) established the connection between deep networks and kernel methods by introducing the Neural Tangent Kernel (NTK).
They showed that at the inﬁnite width limit, the kernel remains constant throughout training.
Lee et al.
(2019) also showed that a network evolves as a linear model in the inﬁnite width limit when trained on certain losses under gradient descent.
Throughout our analysis, we make the assumption that the neural network is overparameterized, and consider the linear approximation of the neural network around its initialisation:
[Equation]
3. Convergence of OGD for Continual Learning
In this section, we derive a closed form expression for the learned models across tasks.
We ﬁnd a recursive kernel ridge regression relationship between the models across tasks.
The result is presented in Theorem 1, a stepping stone towards proving the generalisation bound for OGD in
Sec. 4.
3.1. Convergence Theorem
Now, we state the main result of this section:
Theorem 1 (Convergence of SGD and OGD for Continual Learning) Let T 1 , . .
, T T be a sequence of tasks.
Fix a learning rate sequence ( η τ ) τ ∈
[ T ] .
If, for all τ , the learning rate satisﬁes
[Equation]
then for all τ , w τ ( t ) converges linearly to a limit solution w (cid:63)τ such that
[Equation]
where
[Equation]
φ τ ( x ) for SGD , T τ φ τ ( x ) for OGD .
and { T τ , τ ∈
[ T ] } are proxy matrices for the analysis.
The theorem describes how the model f
(cid:63)τ evolves across tasks.
The theorem is recursive because the learning is incremental.
For a given task T τ , f (cid:63)τ − 1 ( x ) is the knowledge acquired by the agent up to the task T τ − 1 .
At this stage, the model only ﬁts the residual ˜y τ
=
y τ − y τ − 1 → τ , which complements the knowledge acquired through previous tasks.
This residual is also a proxy for task similarity.
If the tasks are identical, the residual is equal to zero.
The knowledge increment is captured by the term: k τ ( x , X τ ) T ( k τ ( X τ , X τ ) + λ 2 τ I ) − 1 ˜y τ .
Finally, task similarity is computed with respect to the most recent feature map (cid:101) φ τ , and k τ is the NTK with respect to the feature map (cid:101) φ τ .
Corollary
The recursive relation from Theorem 1 can also be written as a linear combination of kernel regressors as follows:
[Equation]
where
[Equation]
Proof Sketch: We prove Theorem 1 by induction.
We rewrite the loss function as a regression on the residual ˜y τ instead of y τ .
Then, we rewrite the optimisation objective as an unconstrained strongly convex optimisation problem.
Finally, we compute the unique solution in a closed form.
The full proof is presented in App. A.1.
3.2. Distance from Initialisation
As described in Sec. 3.1, ˜y τ is a residual.
It is equal to zero if the model f (cid:63)τ − 1 makes perfect predictions on the next task T τ .
The more the next task T τ is different, the further the neural network needs to move from its previous state in order to ﬁt it.
Corollary 2 tracks the distance from initialisation as a function of task similarity.
Corollary 2 For SGD, and for OGD under the additional assumption that { T τ , τ ∈ [ T ] } are orthonormal,
[Equation]
where
[Equation]
The proof is presented in App.
A.2.
The orthonormality assumption is not restrictive, since the set { T τ , τ ∈ [ T ] } is only a proxy for the analysis; indeed we can choose any convenient basis to work with.
Remark 1 Corollary 2 can be applied to get a similar result to Theorem 3 by Liu et al.
In this remark, we consider mostly their notations.
Their theorem states that under some conditions, for 2-layer neural networks with a RELU activation function, with probability no less than 1 − δ over random initialisation,
[Equation]
where, in their work:
[Equation]
Note that H ∞ P is a Gram matrix, which also corresponds to the NTK of the neural network they consider.
We see an analogy with our result, where we work directly with the NTK, with no assumptions on the neural network.
One important observation is that, to our knowledge, since there are no guarantees for the invertibility of our Gram matrix, we add a ridge regularisation to work with a regularised matrix, which is then invertible.
In our setting, by considering λ → 0 , and with the additional assumption of invertibility of H τ, 0 ,which is valid in the two-layer overparametrised
RELU neural network considered in the setting of Liu et al.
(2019), we can recover a similar approximation.
4. Generalisation of OGD for Continual Learning
In this section, we study the generalisation properties of
OGD.
First, we prove that OGD is robust to catastrophic forgetting with respect to the previous task (Lemma 1).
Then, we present the the main generalisation theorem for OGD (Thm. 2).
The theorem provides several insights on the relation between task similarity and generalisation.
Finally, we present how the Rademacher complexity relates to task similarity across a large number of tasks (Lemma 2).
The lemma states that the more dissimilar tasks are, the larger the class of functions explored by the neural network, with high probability.
This result highlights the importance of the curriculum for Continual Learning.
4.1. Memorisation property of OGD
The key to obtaining tight generalisation bounds for OGD is Lemma 1.
Lemma 1 (Memorisation Property of OGD)
Given a task T τ , for all x τ,i ∈ D τ , a sample from the training data of the task T τ , it holds that
[Equation]
As motivated by Farajtabar et al.
2019, the orthogonality of the gradient updates aims to preserve the acquired knowledge, by not altering the weights along relevant dimensions when learning new tasks.
Lemma 1 states that the training error on the previous task is unchanged, when training with
OGD.
However, there are no guarantees that the knowledge from the tasks before the previous task is preserved.
The proof of Lemma 1 is presented in App.
B.1.
4.2. Generalisation of OGD for Continual Learning
Now, we state the main generalisation theorem for OGD, which provides within-task and outside-task generalisation bounds.
Theorem 2 (Generalisation of OGD for Continual
Learning) Let {T 1 , . . .
T T } be a sequence of tasks.
Let be {D 1 , . . .
, D T } the respective distributions over R d × {− 1 , 1 } .
Let { ( x τ,i , y τ,i ) , i ∈
[ n t ] , τ ∈ [ T ] } be i.i.d.
samples from D τ , τ ∈ [ T ] .
Denote X τ = ( x τ, 1 , . .
. , x τ,n τ ) ,
y τ = ( y τ, 1 , . . .
, y τ,n τ ) .
Consider the kernel ridge regression solution f (cid:63)T .
Suppose that the kernel matrices satisfy
Tr( k τ ( X τ , X τ )) =
O ( n τ ) , then, for any loss function (cid:96) : R × R → [0 , c ] that is cLipchitz in the ﬁrst argument, with probability at least 1 − δ , for within-task generalisation ( T T ),
[Equation]
for outside-task generalisation ( T T − 1 ),
[Equation]
and for outside-task generalisation ( T τ , τ < T − 1 ),
[Equation]
where
[Equation]
The intution behind Theorem 2 is as follows:
• Within-task generalisation: The generalisation error on the most recent task leverages the information learned during training on the previous tasks.
The bound is tighter compared to learning from scratch, since it depends on the residual ˜y τ , which is a proxy for Forward Transfer.
Therefore it captures the transferability of knowledge across tasks.
• Outside-task generalisation
( T T − 1 ): The generalisation bound increases only with respect to the Rademacher complexity when training with OGD.
The tightness of this bound for OGD is due to Lemma 1.
This lemma is valid for OGD and not for SGD, which implies that tighter generalisation is guaranteed compared to SGD.
• Outside-task generalisation ( T τ , τ ≤ T − 2 ):
The upper bound depends on the similarity between the outside task and the latest task; the more dissimilar the subsequent tasks are, the more the upper bound diverges from the initial upper bound.
This bound captures catastrophic forgetting as a function of the tasks dissimilarity.
This bound is the same for OGD and SGD.
These bounds share some similarities with the bounds derived by Arora et al.
(2019), Liu et al.
(2019) and Hu et al.
(2019), where in these works, the bounds were derived for supervised learning settings, and in some cases for two-layer
RELU neural networks.
Similarly, the bounds depend on the
Gram matrix of the data, with the feature map corresponding to the NTK.
Proof Sketch:
The proof is presented in App.
B.1.
One challenge is that the function class is the set of linear combinations of kernel regressors (Theorem 1).
We state Lemmato bound the Rademacher complexity of this function class.
Then we derive bounds for the training error for each case in Theorem 2.
The ﬁrst case is straightforward.
For the second case, we use Lemma 1, then derive a similar proof to the ﬁrst case.
The third case presents some additional technical challenges.
In order to derive the upper bounds, we draw a strong inspiration from Hu et al.
and leverage several of their proof techniques and mathematical tools.
4.3. Distance from Initialisation through Tasks
Now, we state Lemma 2, which tracks the Rademacher complexity through tasks.
Lemma 2 Keeping the same notations and setting as Theorem 2, the Rademacher Complexity can be bounded as follows:
[Equation]
The intuition behind Lemma 2 is that the upper bound on the Rademacher complexity increases when the tasks are dissimilar.
The NTK task similarity between two subsequent tasks τ and τ +1 is ˜y Tτ ( k t ( X τ , X τ )+ λ 2 I ) − 1 ˜y τ .
The knowledge from the previous tasks is implicitly encoded in the kernel k τ , which is based on the feature map φ τ .
This feature map encodes the acquired knowledge.
As an edge case, if two successive tasks are identical, the residual ˜y τ = y τ − y τ − 1 → τ = 0 , therefore, the upper bound on the Rademacher complexity does not increase.
Implications for Curriculum Learning We also observe that the upper bound depends on the task ordering, which may provide a theoretical explanation on the importance of learning with a curriculum (Bengio et al., 2009).
In the following, we present an edge case which provided an intuition on how the bound captures the importance of the order.
Consider two dissimilar tasks T 1 and T 2 .
A sequence of tasks alternating between T 1 and T 2 will lead to a large upper bound, as explained in the ﬁrst paragraph.
While, a sequence of tasks concatenating two sequences of T 1 then
T 2 will lead to a lower upper bound.
Proof Sketch:
The proof techniques for Lemma 2 are exactly the same as the ones for Theorem 2.
The full proof is presented in Sec.
B.1.6.
5. OGD+: Learning without Forgetting, provably
In the previous section, we demonstrated the limits of OGD, in terms of robustness to catastrophic forgetting on the long run.
Now, we present OGD+, an extension of OGD, which we prove robust to catastrophic forgetting, across an arbitrary number of tasks (Lemma 3).
Then, we prove tighter generalisation bounds compared to OGD (Theorem 3).
5.1. The OGD+ Algorithm
Algorithm 1 presents the OGD+ algorithm, we highlight the differences with OGD in red.
The main difference is that
OGD+ stores the feature maps with respect to the samples from previous tasks, in addition to the feature maps with respect to the samples from the current task, as opposed to OGD.
This small change unlocks the proof of Lemmagiven below, which implies tighter bounds for Theorem 3.
The idea behind OGD+ comes from the convergence Theorem (Sec. 3, Thm. 1).
After training on a task T τ , the learned model is a linear combination of the previous models.
For a given sample x from a task T k where k < τ , in order to keep the training error identical, the weights need to be updated along the directions that are orthogonal to all the subsequent feature maps of x .
OGD only considers the feature map of the source task of the sample.
Storing all the feature maps implies that the learned model back from task T k , can be recovered even after training on an arbitrary number of tasks.
In order to compute the feature maps with respect to the previous samples, OGD+ saves these samples in a dedicated memory, we call this storage the samples memory .
This memory comes in addition to the orthonormal feature maps memory .
The only role of the samples memory is to compute the feature maps.
While the proofs below are under the assumption that the memory size is inﬁnite, in the experiments, we keep a limited size for both memories.
S and k ∈ [1 , c ] s.t. y
k
= 1 do u ← ∇ f τ ( x ; w ) − (cid:80) v ∈ S J proj v ( ∇ f τ ( x ; w ))
S J ← S J (cid:83) { u } end
Sample D ⊂ D τ ;
Update S D ← S D (cid:83) D end
5.2. Memorisation Property of OGD+
The key to obtaining tight generalisation bounds for OGD+ is the Lemma 3 below.
It states that the training error across all previous tasks is unchanged, when training with OGD+.
Lemma 3 (Memorisation Property of OGD+)
Given a task T τ , for all x k,i ∈ D k , a sample from the training data of a previous task, it holds that:
[Equation]
The full proof of Lemma 3 is presented in App .
C.1.
5.3. Generalisation Guarantees for OGD+
Now, we state the generalisation theorem for OGD+, which provides tighter generalisation bounds in comparison with
Theorem 2, for OGD.
Theorem 3 (Generalisation of OGD+ for Continual
Learning)
Under the same conditions as Theorem 2, for
OGD+ [Equation] it holds that, for all tasks T τ , within-task and outside-task generalisation error can be bounded as follows
[Equation]
where
[Equation]
1 → τ , y τ − 1 → τ = f (cid:63)τ − 1 ( X τ ) .
The generalisation bounds of Theorem 3 are tighter than the generalisation bounds for OGD.
The tightness of the bounds is a consequence of Lemma 3.
The term that corresponds to the Rademacher complexity is unchanged, while the term that bounds the training error is tighter.
It is also tighter than a standard supervised learning bound, because it captures the transferability of knowledge across tasks through the residual ˜y τ , as opposed to the Supervised Learning only bounds, which would depend on y τ instead.
Proof Sketch The full proof is presented in App.
C.2.
The proof is based on Lemma 3 and Lemma 4.
The proof techniques are the same as the ones for Theorem 2.
6. Experiments
We perform experiments on the continual learning benchmarks Permuted MNIST, Rotated MNIST, Split MNIST (Farajtabar et al., 2019), and Split CIFAR-100 benchmark (Chaudhry et al., 2019).
In order to assess the robustness to catastrophic forgetting over long tasks sequences, we increase the length of the tasks streams from 5 to 15 for the MNIST benchmarks, and consider all the 20 tasks for the
Split CIFAR-100 benchmark. Baselines We consider SGD and OGD as baselines for comparison.
SGD draws a lower bound over the performance as it doesn’t explicitly retain information in Continual Learning settings. OGD is an important baseline, as the analysis shows that OGD+ is more robust to Catastrophic Forgetting than OGD in theory and that OGD achieves stateof-art results on Continual Learning benchmarks.
Setup We keep the same neural network architecture and mostly similar hyperparameters as Farajtabar et al. 2019.
We use a single-head MLP for Permuted MNIST and Rotated
MNIST, a multi-head MLP for Split MNIST and a multihead LeNet (Lecun et al., 1998) for Split CIFAR-100.
We keep a small learning rate of 10 − 2 in order to preserve the locality assumption.
Since the tasks streams we consider are large, we set the memory size to 1.000.
Reproducibility details are presented in App. E.1. 6.1. Permuted MNIST
Permuted MNIST (Goodfellow et al., 2014) consists of a series of MNIST supervised learning tasks, where the pixels of each task are permuted with respect to a ﬁxed permutation.
The permutations are sampled independently in order for the tasks to be equally hard.
Therefore, the benchmark is a proxy for the effetiveness of the algorithms to remember samples from the previous tasks without any without knowledge transfer between the tasks.
The results are reported in Table 1 and Table 8 (App. E.4.1). We ﬁnd that OGD+ signiﬁcantly outperforms OGD and SGD on the oldest tasks. While OGD either outperforms or is equivalent to OGD+ on the most recent tasks.
[Table]
[Table]
5± [Table] 6.2. Rotated MNIST Rotated MNIST (Farajtabar et al., 2019) consists of a series of MNIST classiﬁcation tasks, where the images are rotated with respect to a ﬁxed angle, monotonically.
We increment the rotation angle by 5 degrees at each new task.
The results are reported in Table 2 and Table 9 (App. E.4.2) [Table] We ﬁnd that OGD+ signiﬁcantly outperforms OGD and SGD on the oldest tasks
[Table] and that OGD and OGD+ become equivalent the more recent the tasks are. [Table] 6.3. Split MNIST
Split MNIST (Zenke et al., 2017) consists of ﬁve binary classiﬁcation tasks built from MNIST.
The labels are split into the sets 0/1, 2/3, 4/5, 6/7 and 8/9.
The results are reported in Table 3. We see that OGD+ outperforms OGD and SGD on the oldest task, OGD and OGD+ are equivalent on the two subsequent tasks, then SGD, OGD and OGD+ are almost equivalent on the remaining most recent tasks. [Table] 6.4. Split CIFAR-
Split CIFAR-100 (Chaudhry et al., 2019) is constructed by splitting the original CIFAR-100 dataset (Krizhevsky, 2009) into 20 disjoint subsets, where each subset is formed by sampling without replacement 5 classes out of 100.
The results are reported in Table 4 and Table 10 (App. E.4.3).
We ﬁnd that OGD+ outperforms OGD on average on the oldest tasks.
However, the performance difference is not statistically signiﬁcant overall.
One probable reason is that for the Split CIFAR-100 benchmark, it is likely that independent tasks would share multiple features, therefore the forgetting component is less isolated by the benchmark compared to the benchmarks presented in the previous sections.
Therefore, even over long tasks sequences, relevant features may occur multiple times in intermediate tasks, in which case, OGD+ may not have a signiﬁcant edge over OGD.
In order to verify this hypothesis, we track the dynamics of
SGD, OGD and OGD+ on the 8 ﬁrst tasks (Fig 1, App. E.2).
We observe that backward transfer occurs frequently during training, as there are performance improvement spikes on old tasks even though the model is not trained on them speciﬁcally. Also, as opposed to the previous benchmarks, the performance margin with SGD is much smaller. [Table] [Table]
[Table] Overall, we ﬁnd that OGD and OGD+ outperform SGD on all benchmarks. OGD+ outperforms OGD when the task’s relevant features’ occurrence is the furthest away in the past. Otherwise, OGD and OGD+ are mostly equivalent.
These results concur with Lemma 6 which states that OGD is robust to catastrophic forgetting up to a single task ahead, and Lemma 3, which states that OGD+ is robust to catastrophic forgetting across any number of tasks. Also, two probable reasons OGD+ is not perfectly prone to catastrophic forgetting in the experiments are the memory limit and the non-overparameterization of the neural network. We discuss more in detail the limits of OGD+ in Sec.
D. 7. Related works
Continual Learning Approaches to Continual Learning can be categorised into: regularization methods, memory based methods, and dynamic architectural methods.
We refer the reader to the survey (Parisi et al., 2019) for an extensive overview on the existing methods.
The idea behind memory-based methods is to store data from previous tasks in a buffer of ﬁxed size, which can then be reused during training on the current task (Chaudhry et al.
2019, Van de
Ven & Tolias 2018).
Dynamic architectural methods rely on growing architectures which keep the past knowledge ﬁxed and store new knowledge in new components, such as new nodes, layers ...
(Lee et al.
2018, Schwarz et al. 2018)
The idea behind regularization methods is to regularize the objective in order to preserve the knowledge acquired from the previous tasks (Kirkpatrick et al. 2016, Aljundi et al.
2018, Farajtabar et al.
2019, Zenke et al. 2017).
Catastrophic Forgetting refers to the tendency of agents to ”forget” the previous tasks over the course of training.
It has proven to be a challenging problem, several heuristics were developed in order to characterise it (Ans & Rousset 1997, Ans & Rousset 2000, Goodfellow et al. 2014, French 1999, McCloskey & Cohen 1989, Robins 1995, Nguyen et al. 2019).
Several works have also recently started studying Transfer Learning from a theoretical perspective (Yu et al. 2020, Liu et al.
2019, Nguyen et al. 2019, Achille et al. 2019).
Deep Learning Theory Recent works have started to provide explanations about the mechanics of overparametrised
Neural Networks.
In their seminal work, Du et al.
(2018) prove that Gradient Descent on multilayer overparametrised
RELU neural networks achieve zero training error at the limit.
These works have unlocked the analysis of several properties of Deep Neural Networks, in the context of various applications, such as Transfer Learning (Liu et al., 2019), Noisy Supervision (Hu et al., 2019), Reinforcement
Learning (Wang et al., 2020) ...
Another line of works provide closed form expressions of the training dynamics of overparameterized neural networks, leveraging tools from statistical physics (Goldt et al.
2019, Goldt et al. 2020).
Statistical Learning Theory Alquier et al.
(2017) deﬁne a compound regret for lifelong learning, as the regret with respect to the oracle who would have known the best common representation g for all tasks in advance.
Another line of works addresses lifelong learning and meta-learning from a statistical learning theory perspective, they deﬁne and provide regret bounds for lifelong learning (Alquier et al., 2017), or study the sample complexity and convergence of meta-learning algorithms (Denevi et al. 2018a, Du et al.
2020, Ji et al.
2020, Saunshi et al.
2020, Denevi et al. 2018b).
8. Conclusion
We presented several theoretical results on the convergence and generalisation properties of OGD and OGD+ for Continual Learning, then showed that these results are applicable to practical settings.
We also found theoretical connections to Transfer Learning and Curriculum Learning.
Firstly, as opposed to OGD, the memory requirement of
OGD+ scales quadratically with the number of samples.
Finding a way to prioritize the feature maps to store is critical to scale to larger tasks and model architectures.
Another important direction for future investigation is a theoretical analysis of the other Continual Learning training methods and catastrophic forgetting heuristics, such as the ones presented by Nguyen et al.
Even for asymptotic cases, the analysis may provide insights on their properties, limits and directions of improvement.
Also, it would be interesting to investigate the applicability of the theory to neighbouring ﬁelds such as meta-learning, multi-task learning ...
We hope this analysis provides new keys to investigate these directions.
A. Missing proofs of section 3 - Convergence A.1. Proof of TheoremOrthogonal Gradient Descent Proof
We prove the Theorem 1 by induction.
Our induction hypothesis H τ is the following :
H τ : For all k ≤ τ , Theorem 1 holds.
First, we prove that H 1 holds.
The proof is straightforward.
For the ﬁrst task, since there were no previous tasks E 1 = ∅ .
Therefore, OGD on this task is equivalent to SGD.
Therefore, it is equivalent to minimising the following objective, where τ = 1 :
[Equation]
The objective is quadratic and the Hessian is positive deﬁnite, therefore the minimum exists and is unique :
[Equation]
For τ = 1 , since there are no previous tasks ˜y τ = y τ .
Therefore :
[Equation]
Which completes the proof of H 1 .
Let τ ∈ N (cid:63) , assume H τ is true, we show H τ +
On the task τ + 1 , we can write the loss L τ +1 as :
[Equation]
We recall that the optimisation problem at time ( τ + 1) :
[Equation]
Let T τ
+1 ∈ R d × ( d − K τ +1 ) and ˜w τ +1 ∈ R
d − K τ +1 such as :
[Equation]
We rewrite the objective by plugging in the variables we just deﬁned.
The two objectives are equivalent :
[Equation]
For clarity, we deﬁne Z τ +1 R n τ +1 ×
d − K τ +1 )
as :
[Equation]
By plugging in Z τ +1 , we rewrite the objective as :
[Equation]
The optimisation objective is quadratic, unconstrainted, with a positive deﬁnite hessian.
Therefore, an optimum exists and is unique :
[Equation]
We recover the expression of the optimum in the original space :
[Equation]
We deﬁne the kernel k τ
+1 :
R d × R d → R as :
[Equation]
) =
φ τ ( x ) T T
{+τ +1 T Tτ +
+}
φ τ ( x (cid:48) ) for all x , x
[Equation]
d
Now we rewrite w (cid:63)τ +1 − w (cid:63)τ : w
[Equation]
1 − w (cid:63)τ =
T τ +1 Z Tτ +
[Equation]
k τ +
[Equation]
X τ , X τ ) + λ
[Equation]
I ) − 1 ˜y τ +
Finally, we recover a closed form expression for f (cid:63)τ +1 :
First, we use the induction hypothesis H τ :
[Equation]
At this stage, we have proven H t +1 .
We conclude.
Stochastic Gradient Descent
The proof is exactly the same as the proof for Orthogonal Gradient Descent, except that there are no equalities constraints.
A.2. Proof of the CorollaryOrthogonal Gradient Descent Proof
In the proof of Theorem 1 (App. A.1), we proved that :
[Equation]
Therefore :
[Equation]
λ 2 τ +1 I ) − 1 ˜y τ +
Stochastic Gradient Descent
The proof is exactly the same as for Orthogonal Gradient Descent.
B. Missing proofs of section 4 - Generalisation B.1. Proof of TheoremB.1.1. N OTATIONS
We recall that :
[Equation]
We deﬁne :
[Equation]
where :
α
[Equation]
Then :
[Equation]
Reminder on RKHS norm Let k a kernel, and H the reproducing kernel Hilbert space (RKHS) corresponding to the kernel k .
Recall that the RKHS norm of a function f ( x ) =
α T k ( x , X ) is :
[Equation]
Reminder on Generalization and Rademacher Complexity Consider a loss function l : R × R → R .
The population loss over the distribution D , and the empirical loss over n samples D = { ( x i , y i ) , i ∈ [ n ] } from the same distribution D are deﬁned as :
[Equation]
Theorem 4 Suppose the loss function is bounded in [0 , c ] and is ρ − Lipchitz in the ﬁrst argument.
Then, with probability at least 1 − δ over sample S of size n :
[Equation]
B.1.2. B OUNDING THE R ADEMACHER C OMPLEXITY
Lemma 4 (Rademacher Complexity of a linear combination of kernels) Let k t :
X × X → R , t ∈
[ T ] kernels such that :
[Equation]
To every kernel k t , we associate a feature map φ t : X → H t , where H t is a Hilbert space with inner product (cid:104)· , ·(cid:105)H t , and for all x , x (cid:48) ∈ X , k
[Equation] x [Equation] t
We deﬁne F as follows :
[Equation]
Let X 1 , ..., X n be random elements of X .
Then for the class F , we have :
[Equation]
Proof
Let f ∈ F , and let x ∈ X :
[Equation]
( x
[Equation]
=
[Equation]
=
[Equation] x [Equation] For all t ∈ [ T ] , we associate a feature map φ t : X → H t ∀ x , x (cid:48)
∈
X
[Equation]
Therefore :
[Equation]
On the other hand [Equation] the following holds ∀ t ∈
[ T ] :
[Equation]
Therefore :
[Equation]
Now
[Equation]
we derive an upper bound of the Rademacher complexity of F :
[Equation]
The last inequality is obtained by applying the upper bound from Lemma 22 in (Bartlett & Mendelson, 2003), on each function f t
B
1.3. B OUNDING
(cid:13)(cid:13)(cid:13) ˜ f
(cid:63) τ (cid:13)(cid:13)(cid:13) H τ :
Lemma 5 Let H τ the Hilbert space associated to the kernel k τ .
We recall that : ˜ f
[Equation]
α
[Equation]
= ( k τ
[Equation]
X
[Equation]
, X
[Equation]
) + λ
[Equation]
I )
[Equation]
˜y τ
Then :
[Equation]
H
[Equation]
Proof (cid:13) (cid:13) (cid:13) ˜
[Equation]
H
[Equation]
=
[Equation]
= y Tτ ( k τ ( X [Equation] , X
[Equation]
) + λ 2 I ) − 1 k [Equation] ( X [Equation]
, X
[Equation]
( k
[Equation]
( X
[Equation]
, X
[Equation]
) + λ 2 I ) −
[Equation] ˜y [Equation] Since ( k τ ( X , X ) + λ 2 I ) − 1 ≤ k τ ( X , X ) − 1 , we get : (cid:13)(cid:13)(cid:13) ˜
f (cid:63) τ (cid:13)(cid:13)(cid:13)
[Equation]
B.1.4. P ROOF OF L EMMA
The intuition behind the proof is : since the gradient updates were performed orthogonally to the feature maps of the training data of the source task, the parameters in this space are unchanged, while the remaining space, which was changed, is orthogonal to these features maps, therefore, the inference is the same and the training error remanis the same as at the end of training on the source task.
Proof
[Equation]
Since we the training on task T T is perfomed with OGD, we have :
[Equation]
Since ∇ w f ( w (cid:63)T − 1 , x ) ∈ E T − 1 by deﬁnition, it follows that :
[Equation]
B.1.5. B OUNDING THE TRAINING ERROR
Lemma
The training errors on the source and target tasks can be bounded as follows :
Let T ∈ N ﬁxed.
Then, for all τ ∈ [ T ]
[Equation]
[Equation] [Equation]
For all τ ∈ [ T − 2]
[Equation]
Task T T Proof
We start from the deﬁnition of the training error :
[Equation]
i =
[Equation]
The expression is very similar to the previous norm, we can derive the same analysis as above to derive the following bound :
[Equation]
Therefore :
[Equation]
Task T T −Proof
We start with the deﬁnition of the training error
[Equation]
then applying Lemma 1 :
[Equation]
Since :
[Equation]
We get :
[Equation]
Therefore :
[Equation]
Task T 1 , ..
[Equation]
, T T
[Equation]
Proof
Let τ ∈
[ T − 2] ﬁxed.
We recall that :
[Equation]
Then :
[Equation]
T
[Equation]
We can upper bound A similarly to the previous paragraphs, therefore we get :
[Equation]
Now
[Equation]
we upper bound B
[Equation]
Let k ∈
[ τ + 1 , T ] :
[Equation]
[Table]
[Equation]
We conclude by plugging back the upper bounds of A and B
[Equation]
Therefore :
[Equation]
B.1.6. B OUNDING THE R ADEMACHER C OMPLEXITY Proof
The proof strategy is exactly the same as sec.
B.1.
We generalize the previous proof, by applying the same lemmas.
[Equation]
It holds that :
[Equation]
where, for all τ ∈ [ T ] :
[Equation]
We made the assumption that for all τ ∈ [ T ] tr
( k τ ( X τ , X τ ))
=
O ( n τ ) , also, by setting (cid:15) = 1 :
[Equation]
B.1.7. P ROOF OF THE T HEOREM
Proof
With probability 1 − δ we have :
[Equation]
We get Theorem 2 by replacing into L S
τ ( f (cid:63) T ) using the inequalities from Lemma
C. Missing proofs of section 5 - OGD+ : Learning without forgetting C.1. Memorisation property of OGD+ - Proof Proof
In the proof of Theorem 1, App. A.1, we showed that, for T τ a ﬁxed task:
[Equation]
We rewrite the recursive relation into a sum:
[Equation]
We observe that, for all k ∈ [ T ] :
[Equation]
On the other hand, for OGD+, given a sample x from D τ , for all k (cid:48) ∈ [ τ + 1 , T ] :
[Equation]
Therefore, for all k (cid:48) ∈
[ k + 1 , τ ] :
[Equation]
Therefore :
[Equation]
We conclude.
C.2. OGD+ Generalisation Theorem - Proof
Proof
The proof is very similar to the proof of Theorem
[Equation]
Let T τ a given task and T ∈ N (cid:63) ﬁxed
We start from the following result in Appendix B.1.
[Equation]
We apply Lemma 3 for tasks T τ and T T :
[Equation]
Therefore :
[Equation]
We replace into the ﬁrst inequality :
[Equation]
We recall the folowwing result from the proof of Theorem 2 in App.
B.1 : n
[Equation]
By replacing into the previous inequality, we conclude the proof of Theorem 3.
D. Complementary discussion
Imperfect robustness to Catastrophic Forgetting Two probable reasons OGD+ is not perfectly prone to catastrophic forgetting in the experiments are the memory limit and the non-overparameterization of the neural network.
As in Lemma 3, we assumed that the memory is unlimited and that the neural network is overparameterized.
We run additional experiments to verify these hypotheses. Table 6 (App. E.3.1) shows that the test accuracy increases uniformly with overparameterization, in which case the linear approximation is more accurate. Table 7 (App. E.3.2) shows that the test accuracy also increases uniformly with the memory size. OGD and OGD+ on the short run
One probable reason OGD+ doens’t outperform OGD on the short run is that OGD+ performs a uniform sampling across samples from all past tasks, with the same memory budget as OGD.
Also the memory requirements of OGD+ increase quadratically with tasks, We expect an information loss with respect to the most recent tasks, since the corresponding storage is used by OGD+ to “remember” older tasks, while OGD uses the equivalent storage for the most recent task.
E. Experiments : E.1. Reproducibility E.1.1. C ODE DETAILS
We forked the initial repository from https://github.com/GMvandeVen/continual-learning , then implemented the OGD and OGD+ algorithms.
This initial repository is related to Van de Ven & Tolias (2018) and Van de Ven &
Tolias (2019). E.1.2. H YPERPARAMETERS We use the same architecture and mostly the same hyperparameters as Farajtabar et al.
We also keep a small learning rate, in order to preserve the locality assumption of OGD, and in order to verify the conditions of the theorems.
For the MNIST benchmarks, the neural network is a three-layer MLP with 256 hidden units in two layers, each layer uses RELU activation function.
The model has either 2 logits or 10 logit outputs, which do not use any activation function.
For the Split CIFAR-100 benchmark, the neural network is a multi-head LeNet (Lecun et al., 1998) network with Batch
Normalisation (Ioffe & Szegedy, 2015) and 200 hidden units for the penultimate layer.
The optimiser is either SGD, OGD or OGD+ and the loss is Softmax cross-entropy.
We report the hyperparameters in detail in Table 5. Hyperparameter
MNIST CIFAREpochs
Architecture
MLP
LeNet Hidden dimension Learning [Table]
E.1.3. E XPERIMENT SETUP
We run each experiment 5 times, the seeds set is the same across all experiments sets.
We report the mean and standard deviation of the measurements.
The test error is measured every 50 mini-batch interval.
Memory : In practice, we split the memories uniformly across tasks. Also, we construct S from S D by sampling uniformly without replacement.
Finally, for the memory reduction step, we truncate the last elements of the storage to free-up the space for the next task’s data.
Multi-head models : For the dataset streams Split MNIST and Split MNIST, we consider multi-headed neural networks.
We only store the feature maps with respect to the shared weights, therefore the projection step is not performed for the heads’ weights.
[Figure] E.3. Complementary experiments - OGD+ E.3.1. OGD+ O VERPARAMETERIZATION In order to assess the hypothesis we made in Sec.
D, which states that one reason OGD+ is not perfectly prone to catastrophic forgetting as stated in Thm.
3 is the non-overparameterisation, we track the test accuracy with respect to the MLP hidden layer size, which is a proxy for overparameterisation.
We run the experiments on the Permuted MNIST using OGD+ and vary the hidden size from 100 to 400.
The results are presented in Table 6.
We measure the accuracy of the models through time after being trained on all 15 tasks.
We see that the test accuracy of OGD+ increases uniformly with the hidden size.
We also observe that overparameterisation has a signiﬁcant impact on the test accuracy the older the task is.
The average accuracy margin between the hidden sizes 100 and 400 is 7 .
7% for the ﬁrst task, while it is only 0 .
6% for the latest task.
[Table] E.3.2. OGD+ M EMORY SIZE :
In order to assess the hypothesis we made in Sec.
D
[Table]
which states that one reason OGD+ is not perfectly prone to catastrophic forgetting as stated in Thm.
3 is the limited memory, we track the test accuracy with respect to the memory size.
We run the experiments on the Permuted MNIST using OGD+ and vary the memory size from 1.000 to 2 .000.
The results are presented in Table 7.
We measure the accuracy of the models through time after being trained on all 15 tasks.
We see that the mean test accuracy of OGD+ increases uniformly with the memory size.
The conﬁdence intervals are not tight enough in most cases to conclude that the test accuracy increases uniformly with the memory size.
[Table]
E.4. Complementary tables - Generalisation E.4.1. P ERMUTED MNIST
We present the results of the Split MNIST experiments in Table 8.
We measure the accuracy of the models through time after being trained on all 15 tasks.
We can see that OGD+ outperforms OGD and SGD on the 6 initial tasks
[Table]
and that OGD outperforms OGD+ on the subsequent tasks.
[Table]
E.4.2. R OTATED MNIST
We present the results of the Rotated MNIST experiments in Table 8.
We measure the accuracy of the models through time after being trained on all 15 tasks.
We observe that OGD+ outperforms OGD and SGD on the 8 initial tasks
[Table]
and OGD+ and
OGD are equivalent in terms of robustness to catastrophic forgetting on the tasks 9 to 11.
[Table]
E.4.3. S PLIT CIFAR-
We present the results of the Split CIFAR-100 experiments in Table 8.
We measure the accuracy of the models through time after being trained on all 20 tasks, each task consists of a classes subset of size 5 sampled without repetition from theclasses of CIFAR-100.
We see that on average , OGD+ outperforms OGD on the long task sequences.
The conﬁdence intervals are not tight enough to conclude on the relative performance of OGD and OGD+ on the Split CIFAR-100 benchmark.
Also, OGD and OGD+ outperform SGD on all tasks and are equivalent to SGD on the latest task.
This result is intuitive as the last task is only a supervised leanring problem which does not involve Continual Learning.
[Table]