G ENERALISATION G UARANTEES FOR C ONTINUAL L EARNING WITH O RTHOGONAL G RADIENT D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
In Continual Learning settings, deep neural networks are prone to Catastrophic
Forgetting.
Orthogonal Gradient Descent was proposed to tackle the challenge.
However, no theoretical guarantees have been proven yet.
We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime.
This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity.
In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the ﬁrst generalisation bound for SGD and OGD for Continual Learning.
Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.
I NTRODUCTION
Continual Learning is a setting in which an agent is exposed to multiples tasks sequentially (Kirkpatrick et al., 2016).
The core challenge lies in the ability of the agent to learn the new tasks while retaining the knowledge acquired from previous tasks.
Too much plasticity (Nguyen et al., 2018) will lead to catastrophic forgetting, which means the degradation of the ability of the agent to perform the past tasks (McCloskey & Cohen 1989, Ratcliff 1990, Goodfellow et al. 2014).
On the other hand, too much stability will hinder the agent from adapting to new tasks.
While there is a large literature on Continual Learning (Parisi et al., 2019), few works have addressed the problem from a theoretical perspective.
Recently, Jacot et al.
(2018) established the connection between overparameterized neural networks and kernel methods by introducing the Neural Tangent
Kernel (NTK).
They showed that at the inﬁnite width limit, the kernel remains constant throughout training.
Lee et al.
(2019) also showed that, in the inﬁnite width limit or Neural Tangent Kernel (NTK) regime, a network evolves as a linear model when trained on certain losses under gradient descent.
In addition to these ﬁndings, recent works on the convergence of Stochastic Gradient Descent for overparameterized neural networks (Arora et al., 2019) have unlocked multiple mathematical tools to study the training dynamics of over-parameterized neural networks.
We leverage these theoretical ﬁndings in order to to propose a theoretical framework for Continual
Learning in the NTK regime then prove convergence and generalisation properties for the algorithm
Orthogonal Gradient Descent for Continual Learning (Farajtabar et al., 2019).
Our contributions are summarized as follows
We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel (NTK) regime.
This framework frames Continual Learning as a recursive kernel regression and comprises proxies for Transfer Learning, generalisation, tasks similarity and Curriculum Learning.
(Thm. 1, Lem. 1 and Thm. 3).
In this framework, we prove that OGD is robust to forgetting with respect to an arbitrary number of tasks under an inﬁnite memory (Sec . 5, Thm. 2).
We prove the ﬁrst generalisation bound for Continual Learning with SGD and OGD.
We ﬁnd that generalisation through tasks depends on a task similarity with respect to the NTK.
(Sec. 5, Theorem 3) 4.
We study the limits of this framework in practical settings, in which the Neural Tangent Kernel may vary.
We ﬁnd that the variation of the Neural Tangent Kernel impacts negatively
R ELATED WORKS
Continual Learning addresses the Catastrophic Forgetting problem, which refers to the tendency of agents to ”forget” the previous tasks they were trained on over the course of training.
It’s an active area of research, several heuristics were developed in order to characterise it (Ans & Rousset 1997,
Ans & Rousset 2000, Goodfellow et al. 2014, French 1999, McCloskey & Cohen 1989, Robins 1995, Nguyen et al. 2019).
Approaches to Continual Learning can be categorised into: regularization methods, memory based methods and dynamic architectural methods.
We refer the reader to the survey by Parisi et al.
for an extensive overview on the existing methods.
The idea behind memory-based methods is to store data from previous tasks in a buffer of ﬁxed size, which can then be reused during training on the current task (Chaudhry et al.
2019, Van de Ven & Tolias 2018).
While dynamic architectural methods rely on growing architectures which keep the past knowledge ﬁxed and store new knowledge in new components, such as new nodes or layers.
(Lee et al.
Schwarz et al. 2018)
Finally, regularization methods regularize the objective in order to preserve the knowledge acquired from the previous tasks (Kirkpatrick et al. 2016, Aljundi et al.
2018, Farajtabar et al.
2019, Zenke et al. 2017).
While there is a large literature on the ﬁeld, there is a limited number of theoretical works on
Continual Learning.
Alquier et al.
(2017) deﬁne a compound regret for lifelong learning, as the regret with respect to the oracle who would have known the best common representation for all tasks in advance.
Knoblauch et al.
(2020) show that optimal Continual Learning algorithms generally solve an NP-HARD problem and will require perfect memory not to suffer from catastrophic forgetting.
Benzing (2020) presents mathematical and empirical evidence that the two methods – Synaptic
Intelligence and Memory Aware Synapses – approximate a rescaled version of the Fisher Information.
Continual Learning is not limited to Catastrophic Forgetting, but also closely related to Transfer
Learning.
A desirable property of a Continual Learning algorithm is to enable the agent to carry the acquired knowledge through his lifetime, and transfer it to solve new tasks.
A new theoretical study of the phenomena was presented by Liu et al.
They prove how the task similarity contributes to generalisation, when training with Stochastic Gradient Descent, in a two tasks setting and for over-parameterised two layer RELU neural networks.
The recent ﬁndings on the Neural Tangent Kernel (Jacot et al., 2018) and on the properties of overparameterized neural networks (Du et al. 2018, Arora et al. 2019) provide powerful tools to analyze their training dynamics.
We build up on these advances to construct a theoretical framework for Continual Learning and study the generalisation properties of Orthogonal Gradient Descent.
P RELIMINARIES
Notation We use bold-faced characters for vectors and matrices.
We use (cid:107)·(cid:107) to denote the Euclidean norm of a vector or the spectral norm of a matrix, and (cid:107) · (cid:107) F to denote the Frobenius norm of a matrix.
We use (cid:104)· , ·(cid:105) for the Euclidean dot product, and (cid:104)· , ·(cid:105)H the dot product in the Hilbert space H .
We index the task ID by τ .
The ≤ operator if used with matrices, corresponds to the partial ordering over symmetric matrices.
We denote N the set of natural numbers, R the space of real numbers and N (cid:63) for the set N (cid:114) { 0 } .
We use ⊕ to refer to the direct sum over Euclidean spaces.
3. C ONTINUAL L EARNING
Continual Learning considers a series of tasks {T 1 , T 2 , . . .
} , where each task can be viewed as a separate supervised learning problem.
Similarly to online learning, data from each task is revealed only once.
The goal of Continual Learning is to model each task accurately with a single model.
The challenge is to achieve a good performance on the new tasks, while retaining knowledge from the previous tasks (Nguyen et al., 2018).
We assume the data from each task T τ , τ ∈ N (cid:63) , is drawn from a distribution D τ .
Individual samples are denoted ( x τ,i , y τ,i ) , where i ∈ [ n τ ] .
For a given task T τ , the model is denoted f τ , we use the superscript ( t ) to indicate the training iteration t ∈ N , while we use the superscript (cid:63) to indicate the asymptotic convergence.
For the regression case, given a ridge regularisation coefﬁcient λ ∈ R + , for all t ∈ N , we write the train loss for a task T τ as :
[Equation]
3. OGD FOR C ONTINUAL L EARNING
Let T T the current task, where T ∈ N (cid:63) .
For all i ∈ [ n T ] , let v T,i = ∇ w f (cid:63)T − 1 ( x T − 1 ,i ) , which is the
Jacobian of task T T .
We deﬁne E τ = vec( { v τ,i , i ∈
[ n τ ] } ) , the subspace induced by the Jacobian.
The idea behind OGD (Farajtabar et al., 2019) is to update the weights along the projection of the gradient on the orthogonal space induced by the Jacobians over the previous tasks E 1 ⊕ . . .
⊕ E τ − 1 .
The update rule at an iteration t ∈ N (cid:63) for the task T T is as follows :
[Equation]
The intuition behind OGD is to “preserve the previously acquired knowledge by maintaining a space consisting of the gradient directions of the neural networks predictions on previous tasks” (Farajtabar et al., 2019).
Throughout the paper, we only consider the OGD-GTL variant which stores the gradient with respect to the ground truth logit.
3. N EURAL T ANGENT K ERNEL
In their seminal paper, Jacot et al.
(2018) established the connection between deep networks and kernel methods by introducing the Neural Tangent Kernel (NTK).
They showed that at the inﬁnite width limit, the kernel remains constant throughout training.
Lee et al.
(2019) also showed that a network evolves as a linear model in the inﬁnite width limit when trained on certain losses under gradient descent.
Throughout our analysis, we make the assumption that the neural network is overparameterized, and consider the linear approximation of the neural network around its initialisation:
[Equation]
C ONVERGENCE - C ONTINUAL L EARNING AS A RECURSIVE K ERNEL R EGRESSION
In this section, we derive a closed form expression for the Continual Learning models through tasks.
We ﬁnd that Continual Learning models can be expressed with recursive kernel ridge regression across tasks.
We also ﬁnd a that the NTK of OGD is recursive with respect to the projection of its feature map on the tasks’ spaces.
The result is presented in Theorem 1, a stepping stone towards proving the generalisation bound for OGD in Sec. 5.
4. C ONVERGENCE T HEOREM
Theorem 1 (Continual Learning as a recursive Kernel Regression)
Given T 1 , . . .
, T T a sequence of tasks.
Fix a learning rate sequence ( η τ ) τ ∈
[ T ] and a ridge regularisation coefﬁcient λ ∈ R + .
If, for all τ , the learning rate satisﬁes η τ < 1 (cid:107) κ τ ( X τ , X τ )+ λ I (cid:107) , then for all τ , w τ ( t ) converges linearly to a limit solution w (cid:63)τ such that
[Equation]
where
[Equation]
SGD
[Equation]
for OGD
[Equation]
and { T τ ∈ R d − M τ ,d , τ ∈ [ T ] } are projection matrices from R d to ( τ (cid:76)
[Equation]
M
[Equation]
dim
[Equation]
The theorem describes how the model f
(cid:63)τ evolves across tasks.
It is recursive because the learning is incremental.
For a given task T τ , f (cid:63)τ − 1 ( x ) is the knowledge acquired by the agent up to the task
T τ − 1 .
At this stage, the model only ﬁts the residual ˜y τ
=
y τ − y τ − 1 → τ , which complements the knowledge acquired through previous tasks.
This residual is also a proxy for task similarity.
If the tasks are identical, the residual is equal to zero.
The knowledge increment is captured by the term: κ τ ( x , X τ ) T ( κ τ +1 ( X τ , X τ ) + λ I ) − 1 ˜y τ .
Finally, the task similarity is computed with respect to the most recent feature map (cid:101) φ τ , and κ τ is the NTK with respect to the feature map (cid:101) φ τ .
Remark
The recursive relation from Theorem 1 can also be written as a linear combination of kernel regressors as follows:
[Equation]
where
[Equation]
4. D ISTANCE FROM INITIALISATION THROUGH TASKS
As described in Sec. 4.1, ˜y τ is a residual.
It is equal to zero if the model f
(cid:63)τ − 1 makes perfect predictions on the next task T τ .
The more the next task T τ is different, the further the neural network needs to move from its previous state in order to ﬁt it.
Corollary 1 tracks the distance from initialisation as a function of task similarity.
Corollary 1 For SGD, and for OGD under the additional assumption that { T τ , τ ∈ [ T ] } are orthonormal,
[Equation]
Remark 2 Corollary 1 can be applied to get a similar result to Theorem 3 by Liu et al.
In this remark, we consider mostly their notations.
Their theorem states that under some conditions, for 2-layer neural networks with a RELU activation function, with probability no less than 1 − δ over random initialisation,
[Equation]
where, in their work:
[Equation]
Note that H ∞ P is a Gram matrix, which also corresponds to the NTK of the neural network they consider.
We see an analogy with our result, where we work directly with the NTK, with no assumptions on the neural network.
One important observation is that, to our knowledge, since there are no guarantees for the invertibility of our Gram matrix, we add a ridge regularisation to work with a regularised matrix, which is then invertible.
In our setting, by considering λ → 0 , and with the additional assumption of invertibility of H τ, 0 ,which is valid in the two-layer overparameterized RELU neural network considered in the setting of Liu et al.
(2019), we can recover a similar approximation.
OGD : L EARNING WITHOUT FORGETTING , PROVABLY
In this section, we study the generalisation properties of OGD building-up on Thm. 1.
First, we prove that OGD is robust to catastrophic forgetting with respect to all previous tasks (Theorem 2).
Then, we present the main generalisation theorem for OGD (Thm. 3).
The theorem provides several insights on the relation between task similarity and generalisation.
Finally, we present how the Rademacher complexity relates to task similarity across a large number of tasks (Lemma 1).
The lemma states that the more dissimilar tasks are, the larger the class of functions explored by the neural network, with high probability.
This result highlights the importance of the curriculum for Continual Learning.
5. M EMORISATION PROPERTY OF OGD
The key to obtaining tight generalisation bounds for OGD is Theorem 2.
Theorem 2 (No-forgetting Continual Learning with OGD)
Given a task T τ , for all x k,i ∈ D k , a sample from the training data of a previous task T k , given that the Jacobian of x k,i belongs to
OGD’s memory, it holds that:
[Equation]
As motivated by Farajtabar et al.
(2019), the orthogonality of the gradient updates aims to preserve the acquired knowledge, by not altering the weights along relevant dimensions when learning new tasks.
Theorem 2 implies that, given an inﬁnite memory, the training error on all samples from the previous tasks is unchanged, when training with OGD.
5. G ENERALISATION PROPERTIES OF SGD AND OGD
Now, we state the main generalisation theorem for OGD, which provides generalisation bounds on the data from all the tasks, for SGD and OGD.
Theorem 3 (Generalisation of SGD and OGD for Continual Learning)
Let {T 1 , . . .
T T } be a sequence of tasks.
Let be {D 1 , . . .
, D T } the respective distributions over
R d × {− 1 , 1 } .
Let { ( x τ,i , y τ,i ) , i ∈
[ n t ] , τ ∈ [ T ] } be i.i.d.
samples from D τ , τ ∈ [ T ] .
Denote
X τ = ( x τ, 1 , . .
, x τ,n τ ) , y τ =
( y τ, 1 , . . .
, y τ,n τ ) .
Consider the kernel ridge regression solution f (cid:63)T , then, for any loss function (cid:96) : R × R → [0 , c ] that is c-Lipschitz in the ﬁrst argument, with probability at least 1 − δ ,
[Equation]
for SGD , τ = T ,
[Equation]
for SGD , τ < T .
where
[Equation]
Theorem 3 shows that the generalisation bound for OGD is tighter than SGD.
This results from Thm.
2 , which states that OGD is robust to Catastrophic Forgetting.
The bound is looser for SGD over the past tasks through the forgetting term that appears.
The bound of Theorem 3 comprises the following main terms :
• λ 2 n τ ˜y τ ( κ τ ( X τ , X τ ) + λ I ) − 1 ˜y τ : this term is due to the regularisation and describes that the empirical loss is not equal to zero due to the regularisation term.
This term tends to zero in case there is no regularisation.
For interpretation purposes, in App.
D.2.7, we show that the empirical loss tends to zero in the no-regularisation case.
• R T
impact of task similarity
generalisation
discuss
Sec
• (cid:80) Tk =
τ +1 H k,τ is a residual term that appears for SGD only.
It is due to the catastrophic forgetting that occurs with SGD.
It also depends on the tasks similarity.
These bounds share some similarities with the bounds derived by Arora et al.
(2019), Liu et al.
(2019) and Hu et al.
(2019), where in these works, the bounds were derived for supervised learning settings, and in some cases for two-layer RELU neural networks.
Similarly, the bounds depend on the Gram matrix of the data, with the feature map corresponding to the NTK.
5. T HE IMPACT OF TASK SIMILARITY ON GENERALISATION
Now, we state Lemma 1, which tracks the Rademacher complexity through tasks.
Lemma 1 Keeping the same notations and setting as Theorem 3, the Rademacher Complexity can be bounded as follows:
[Equation]
where F T is the function class covered by the model up to the task T τ
The right hand side term R T in the upper bound of the generalisation theorem follows directly from
Lemma 1, and it draws a complexity measure for Continual Learning .
It states that the upper bound on the Rademacher complexity increases when the tasks are dissimilar.
We deﬁne the NTK task dissimilarity between two subsequent tasks T τ − 1 and T τ as ¯ S τ − 1 →
τ = ˜y Tτ ( κ τ ( X τ , X τ )
+ λ I ) −
˜y τ .
This dissimilarity is a generalisation of the term that appears in the upper bound of Thm. 2 by Liu et al. 2019.
The knowledge from the previous tasks is encoded in the kernel κ τ , through the feature map φ τ .
As an edge case, if two successive tasks are identical, ¯ S τ − 1 → τ = 0 and the upper bound does not increase.
Implications for Curriculum Learning We also observe that the upper bound depends on the task ordering, which may provide a theoretical explanation on the importance of learning with a curriculum (Bengio et al.
In the following, we present an edge case which provided an intuition on how the bound captures the importance of the order.
Consider two dissimilar tasks Tand T 2 .
A sequence of tasks alternating between T 1 and T 2 will lead to a large upper bound, as explained in the ﬁrst paragraph, while a sequence of tasks concatenating two sequences of T 1 then
T 2 will lead to a lower upper bound.
T HE IMPACT OF THE NTK VARIATION ON OGD
In the previous section, we demonstrated that under the NTK regime and an inﬁnite memory, OGD is provably robust to catastrophic forgetting.
In practical settings, these two assumptions do not hold, therefore, in this section, we study the limits of the NTK regime.
We present OGD+, a variant of
OGD we use to study the importance of the NTK variation for Continual Learning in practice.
In order to decouple the NTK variation phenomena in our experiments, we propose then study the
OGD+ algorithm, which is designed to be more robust to the NTK variation, through updating its orthonormal basis with respect to all the tasks at the end of each task , as opposed to OGD.
Algorithm 1 presents the OGD+ algorithm, we highlight the differences with OGD in red.
The main difference is that OGD+ stores the feature maps with respect to the samples from previous tasks, in addition to the feature maps with respect to the samples from the current task, as opposed to OGD.
This small change is motivated by the variation of the NTK in practice.
In order to compute the feature maps with respect to the previous samples, OGD+ saves these samples in a dedicated memory, we call this storage the samples memory .
This memory comes in addition to the orthonormal feature maps memory .
The only role of the samples memory is to compute the updated feature maps at the beginning of each task.
E XPERIMENTS
We study the validity Theorem 2 which states that under the given conditions, OGD is perfectly robust to Catastrophic Forgetting (Sec. 7.1).
We ﬁnd that the Catastrophic Forgetting of OGD decreases with over-parameterization.
Then, we perform an ablation study using OGD+ in order to study the applicability and limits of the constant Jacobian assumption in practice (Sec. 7.2).
We ﬁnd that the assumption holds for some benchmarks, and that updating the Jacobian (OGD+) is critical for the robustness of OGD in the non-overparameterized benchmarks.
Finally, we present a broader picture of the performance of OGD+ against standard Continual Learning baselines (Sec. 7.3)
Benchmarks We use the standard benchmarks similarly to Goodfellow et al.
(2014) and Chaudhry et al.
Permuted MNIST (Goodfellow et al., 2014) consists of a series of MNIST supervised learning tasks, where the pixels of each task are permuted with respect to a ﬁxed permutation.
Rotated
MNIST (Farajtabar et al., 2019) consists of a series of MNIST classiﬁcation tasks, where the images are rotated with respect to a ﬁxed angle, monotonically.
We increment the rotation angle by 5 degrees at each new task.
Split CIFAR-100 (Chaudhry et al., 2019) is constructed by splitting the original
CIFAR-100 dataset (Krizhevsky, 2009) into 20 disjoint subsets, where each subset is formed by sampling without replacement 5 classes out of 100.
In order to assess the robustness to catastrophic forgetting over long tasks sequences, we increase the length of the tasks streams from 5 to 15 for the MNIST benchmarks, and consider all the 20 tasks for the Split CIFAR-100 benchmark.
We also use the CUB200 benchmark (Jung et al., 2020) which contains 200 classes, split into 10 tasks. This benchmark is more overparameterized than the other benchmarks.
Evaluation Denoting a T,τ the accuracy of the model on the task T τ after being trained on the task
T T , we track two metrics introduced by Chaudhry et al.
and Lopez-Paz & Ranzato (2017) :
A VERAGE A CCURACY
( A T ) is the average accuracy after the model has been trained on the task
T T :
[Equation]
A
VERAGE F ORGETTING
( F T ) is the average forgetting after the model has been trained on the task
T T :
[Equation]
O VER - PARAMETERIZATION
We track over-parameterization of a given benchmark as the number of trainable parameters over the average number of samples per task (Arora et al., 2019) :
[Equation]
Architectures For the MNIST benchmarks, we use the same neural network architectures as
Farajtabar et al.
However, since the OGD algorithm doesn’t scale to large neural networks due to memory limits, we considered smaller scale neural networks for the CIFAR100 and CUB benchmarks.
For CIFAR100, we use the LeNet architecture (Lecun et al., 1998).
For CUB200, we keep the pretrained AlexNet similarly to Jung et al.
(2020), then freeze the features layers and replace the default classiﬁer with a smaller neural network (App. F.1.1).
7. A BLATION STUDY : F OR OGD, C ATASTROPHIC F ORGETTING DECREASES WITH O VERPARAMETERIZATION (T HM . 2)
Theorem 2 states that in the NTK regime, given an inﬁnite memory, the train error of OGD is unchanged.
We study the impact of overparameterization on Catastrophic Forgetting of OGD through an ablation study on the number of parameters of the model.
Experiment We train a model on the full task sequence on the MNIST and CIFAR100 benchmarks, then measure the variation of the train accuracy of the memorised samples from the ﬁrst task.
The reason we consider the samples in the memory is the applicability of Thm.
2 to these samples only.
Fixing the datasets sizes, our proxy for overparameterization is the hidden size, which we vary.
Results Figures 1 and 2 show that the train error variation decreases with overparameterization for OGD, on the MNIST and CIFAR100 benchmarks.
This result concurs with Thm. 2.
7. A BLATION STUDY : F OR NON OVER - PARAMETERIZED BENCHMARKS , UPDATING THE J ACOBIAN IS CRITICAL TO OGD’ S ROBUSTNESS (T HM . 2)
Our analysis relies on the over-parameterization assumption, which implies that the Jacobian is constant through tasks.
Theorem 2 follows from this property.
However, in practice, this assumption may not hold and forgetting is observed for OGD.
We study the impact of the variation of the Jacobian in practice on OGD.
In order to measure this impact, we perform an ablation study on OGD, taking into account the
Jacobian’s variation (OGD+) or not (OGD).
OGD+ takes into account the Jacobian’s variation by updating all the stored Jacobians at the end of each task.
Experiment We measure the average forgetting (Chaudhry et al., 2019) without (OGD) or with (OGD+) accounting for the Jacobian’s variation, on the MNIST, CIFAR100 and CUB200 benchmarks.
Results Tables 1 shows that for the Rotated MNIST and Permuted MNIST benchmarks, which are the least overparameterized, OGD+ is more robust to Catastrophic Forgetting than OGD.
This improvement follows from the variation of the Jacobian in these benchmarks and that OGD+ accounts for it.
While on CIFAR100 and CUB200, the most over-parameterized benchmark, the robustness of
OGD and OGD+ is equivalent.
For these benchmarks, since the variation of the Jacobian is smaller, due to overparameterization, OGD+ is equivalent to OGD.
This results conﬁrms our initial hypothesis that the Jacobian’s variation in non-overparameterized settings such as MNIST is an important reason the OGD algorithm is prone to Catastrophic Forgetting.
This result also highlights the importance of developing a theoretical framework for the non-overparameterized setting, in order to capture the properties of OGD outside the NTK regime. [Table]
[Table]
7. B ENCHMARKING OGD+ AGAINST OTHER C ONTINUAL L EARNING BASELINES
In order to provide a broader picture on the robustness of OGD+ to Catastrophic Forgetting, we benchmark it against standard Continual Learning baselines.
Results Table 2 shows that, on the Permuted MNIST and Rotated MNIST benchmarks, which are the least overparameterized, OGD+ draws an improvement over OGD and is competitive with other Continual Learning methods.
On the CIFAR100 and CUB200, the most overparameterized benchmarks, OGD+ is not competitive.
[Table]
C ONCLUSION
We presented a theoretical framework for Continual Learning algorithms in the NTK regime, then leveraged the framework to study the convergence and generalisation properties of the SGD and
OGD algorithms.
We also assessed our theoretical results through experiments and highlighted the applicability of the framework in practice.
Our analysis highlights multiple connections to neighbouring ﬁelds such as Transfer Learning and Curriculum Learning.
Extending the analysis to other Continual Learning algorithms is a promising future direction, which would provide a better understanding of the various properties of these algorithms and how they could be improved.
Additionally, our experiments highlight the limits of the applicability of the framework to nonoverparameterized settings.
In order to gain a better understanding of the properties of the OGD algorithm in this setting, extending the framework beyond the over-parameterization assumption is an important direction.
We hope this work provides new keys to investigate these directions.
A C OMPLEMENTARY DISCUSSION A. NTK VARIATION - T HE IMPORTANCE OF ORTHOGONALITY FOR THE OGD, OGD+ AND A-GEM ALGORITHMS
In the main article, we focused only on the SGD, OGD and OGD+ algorithms.
In this section, we highlight a connection of these algorithms to the A-GEM algorithm.
All these algorithms perform orthogonal projections during the weight update step, however these projections differ in the following ways :
• the span of the space the updates are orthogonal to.
• the rate at which the projection space is updated.
Additionally, the A-GEM algorithm present the additional property of allowing Positive Backward
Transfer by design.
Positive Backward Transfer is a desirable property in the sense that the generalisation on a past task increases while training on the current task.
Similarities among the algorithms Table 3 presents an overview of the connections between the algorithm.
In practice, the models are not trained in the NTK regime.
Since the feature map changes through training, the orthogonality constraint of OGD becomes less relevant and may harm the learning.
On the other hand, while A-GEM performs a projection on a smaller subspace which may not protect all the subspaces, the subspace is updated as it is computed at each training step, the updates are therefore expected to be more relevant.
Finally OGD+ lies at the intersection of both, while it doesn’t update the feature maps at each gradient step, the projection constraint is with respect to a larger space.
[Table]
On the theoretical connection between OGD and A-GEM The A-GEM algorithm (Chaudhry et al., 2019) is a state of the art Continual Learning algorithm on the standard benchmarks.
The idea behind the algorithm is to perform a gradient descent if an estimate of the loss over the previous tasks increases or is unchanged.
Otherwise the gradient is projected orthogonally to the gradient over the loss estimate over the previous tasks.
We ﬁnd that OGD draws a upper bound in comparison to A-GEM-NT in terms of generalisation error, as stated in Proposition 1.
Proposition 1 In the NTK regime, OGD implies A-GEM with no Positive Backward Transfer.
The proof is presented in App.
E.1.
Experiments The extended experiments on the importance of the NTK variation for Continual
Learning (Sec. F.3), show that A-GEM outperforms OGD and OGD+ on most benchmarks.
A probable reason behind this difference is the rate of update of the feature map for the A-GEM algorithm, even though it spans a smaller space than OGD and OGD+.
A. E XPRESSIONS OVERVIEW
For convenience, we present in Table 4 an overview of the notions that were mentioned in the main article, with their respective mathematical expressions.
[Table]
B P ROOFS OVERVIEW
B.
C ONVERGENCE :
Theorem 1 : Convergence of SGD and OGD for Continual Learning
We prove Theorem 1 by induction.
We rewrite the loss function as a regression on the residual ˜y τ instead of y τ .
Then, we rewrite the optimisation objective as an unconstrained strongly convex optimisation problem.
Finally, we compute the unique solution in a closed form.
The full proof is presented in App.
C.1.
Remark 1 : Continual Learning as a recursive Kernel Regression The remark follows directly from the recursive form of Theorem 1.
Corollary 1 : Distance from initialisation through tasks The proof follows immediately from the proof of Theorem 1, in which we compute a closed form of the weights variation.
It is presented in App.
C.2.
B
G ENERALISATION :
Theorem 2 : No-forgetting Continual Learning with OGD The proof relies on the orthogonality property of the OGD update, which implies that the subspaces spanned by the samples in the memory would not have any changes incurred.
The proof is presented in App .
D.
Theorem 3 : Generalisation of SGD and OGD for Continual Learning The proof or Theorem 3 is presented in App. D.2.
The proof follows the following structure :
• Bounding the Rademacher complexity (App. D.2.3) : First, we state the technical Lemmathat upper bounds the Rademacher complexity of the function class that corresponds to the set of linear combinations of kernel regressors.
Then we apply the lemma to the function class obtained through Theorem 1.
Bounding the empirical loss for SGD (App. D.2.5)
: the error is the same as OGD on the last task.
While on the previous tasks, Catastrophic Forgetting can be incurred which implies the appearance of a residual term that corresponds to the forgetting.
• Bounding the empirical loss for OGD (App. D.2.4) :
The proof techniques are very similar to the SGD case, however we leverage Theorem 2 in order to derive tighter bounds due to the absence of Catastrophic Forgetting.
• Wrap-up (App. D.2.6)
: Finally, we wrap-up all the lemmas
to derive the bound of Theorem 3.
Lemma 1 - Implications for Curriculum Learning This results follows from the technical
Lemma 3, which upper bounds the Rademacher complexity of the function class that corresponds to the set of linear combinations of kernel regressors.
B. T HE IMPORTANCE OF THE NTK VARIATION FOR C ONTINUAL L EARNING :
Proposition 1 - OGD implies A-GEM-NT
The proof relies mainly on Theorem 2 stating the robustness of OGD to Catastrophic Forgetting.
We derive the A-GEM update then apply Theorem 2, which leads to the implication.
The full proof is presented in App. D.2.3.
C M ISSING PROOFS OF SECTION 4 - C ONVERGENCE C. P ROOF OF T HEOREM
Stochastic Gradient Descent We start by proving the Stochastic Gradient Descent (SGD) case of
Theorem 1.
Proof
We prove the Theorem 1 by induction.
Our induction hypothesis H τ is the following : H τ : For all k ≤ τ , Theorem 1 holds.
First, we prove that H 1 holds.
The proof is straightforward.
For the ﬁrst task, since there were no previous tasks, OGD on this task is the same as SGD.
Therefore, it is equivalent to minimising the following objective :
[Equation]
where φ ( x ) = ∇ w (cid:63)
f (cid:63)
( x ) .
We replace into the objective with ˜y 1 applying ˜y
1 = y 1 − f 0 ( X 1 ) :
[Equation]
The objective is quadratic and the Hessian is positive deﬁnite, therefore the minimum exists and is unique :
[Equation]
Under the NTK regime assumption :
[Equation]
Then, by replacing into w (cid:63) 1 − w
[Equation] [Equation]
Finally :
[Equation]
Which completes the proof of H 1 .
Let τ ∈ N (cid:63) , we assume that H τ is true, then we show that H τ +1 is true.
On the task T τ +1 , we can write the loss L τ +1 as :
[Equation]
We recall that the optimisation problem on the task T τ +1 is :
[Equation] [Equation]
The optimisation objective is quadratic, unconstrained, with a positive deﬁnite hessian.
Therefore, an optimum exists and is unique :
[Equation]
We deﬁne the kernel κ τ +1 :
R d × R d → R as :
[Equation]
for all x
[Equation]
Finally, we recover a closed form expression for f (cid:63)τ +1 :
First, we use the induction hypothesis H τ :
[Equation]
At this stage, we have proven H t +1 .
[Equation]
Orthogonal Gradient Descent Now, we prove the Orthogonal Gradient Descent (OGD) case of
Theorem 1.
The key difference in the proof is that the OGD optimisation objective is constrained.
The constraints correspond to the orthogonality to the subspace spanned by the memorised feature maps from the previous tasks.
Another key difference occurs during the regularisation, as opposed to the SGD case where the regularisation of the weights spans on the whole space, the regularisation of
OGD only applies to the “learnable” space.
This property follows from the orthogonality constraint, which enforces that the space spanned by the previous tasks is unchanged.
Proof
We prove the Theorem 1 by induction.
Our induction hypothesis H τ is the following : H τ : For all k ≤ τ , Theorem 1 holds.
First, we prove that H 1 holds.
The proof is straightforward.
For the ﬁrst task, since there were no previous tasks, OGD on this task is the same as SGD.
Therefore, it is equivalent to minimising the following objective :
[Equation]
We replace into the objective by the residual term ˜y
= y 1 − f
[Equation]
X
[Equation] [Equation]
where φ ( x ) = ∇ w (cid:63)
f (cid:63)
( x ) .
The objective is quadratic and its Hessian is positive deﬁnite, therefore the minimum exists and is unique :
[Equation]
Under the NTK regime assumption :
[Equation] [Equation]
Then, by replacing into w (cid:63) 1 − w (cid:63) 0 :
[Equation] [Equation]
Finally :
[Equation]
Which completes the proof of H 1 .
Let τ ∈ N (cid:63) , we assume that H τ is true, then we show that H τ +1 is true.
On the task T τ +
[Equation]
we can write the loss L τ +1 as :
[Equation]
We recall that the optimisation problem on the task T τ +1 is :
[Equation]
where V τ +1 is a projection matrix on ( τ (cid:76) k
=
E k ) ⊥ , the Euclidean space induced by the feature maps of the tasks {T k , k ∈ [ τ ] }
[Equation]
Let T τ
+1 ∈ R d × ( d − K τ +1 ) and ˜w ∈ R
d − K τ +1 such as :
[Equation]
We rewrite the objective by plugging in the variables we just deﬁned.
The two objectives are equivalent :
[Equation]
This objective is equivalent to :
[Equation] [Equation]
For clarity
[Equation]
we deﬁne Z τ +1 R n τ +1 × ( d − K τ +1 ) as :
Z τ +1 =
[Equation]
By plugging in Z τ +
[Equation]
we rewrite the objective as :
[Equation] [Equation]
The optimisation objective is quadratic, unconstrained, with a positive deﬁnite hessian.
Therefore, an optimum exists and is unique :
[Equation]
We recover the expression of the optimum in the original space :
[Equation]
We deﬁne the kernel κ τ +1 :
R d × R d → R as :
[Equation]
x
[Equation]
) for all x , x (cid:48)
[Equation]
Now we rewrite w (cid:63)τ +1 − w (cid:63)τ : w
[Equation]
=
[Equation] [Equation]
Finally, we recover a closed form expression for f (cid:63)τ +1 :
First, we use the induction hypothesis H τ :
[Equation]
At this stage, we have proven H t +1 .
We conclude the proof of Thm. 1.
[Equation]
[Equation]
C. P ROOF OF THE C OROLLARY
Stochastic Gradient Descent
The proof follows immediately from Thm. 1.
Proof
In the proof of Theorem 1 (App. C.1), for the SGD case, we showed that :
[Equation]
This result implies that :
[Equation]
Orthogonal Gradient Descent
The proof is exactly the same as the proof above, the difference lies in the kernel deﬁnition
[Equation]
which is implicit
[Equation]
Proof
In the proof of Theorem 1 (App. C.1)
[Equation]
for the OGD case
[Equation]
we showed that :
[Equation] [Equation]
This result implies that :
[Equation]
D M ISSING PROOFS OF SECTION 5 - G ENERALISATION D. P ROOF OF T HEOREM
The intuition behind the proof is : since the gradient updates were performed orthogonally to the feature maps of the training data of the source task, the parameters in this space are unchanged, while the remaining space, which was changed, is orthogonal to these features maps, therefore, the inference is the same and the training error remains the same as at the end of training on the source task.
Proof
In the proof of Theorem 1, App. C.1, we showed that, for T τ a ﬁxed task:
[Equation] [Equation]
We rewrite the recursive relation into a sum:
[Equation]
We observe that, for all k ∈ [ T ] : w
[Equation]
On the other hand, for OGD+, given a sample x from D τ , for all k (cid:48) ∈ [ τ + 1 , T ] :
[Equation]
Therefore, for all k (cid:48) ∈
[ k + 1 , τ ] :
[Equation]
Therefore :
[Equation]
We conclude.
[Equation]
D. P ROOF OF T HEOREMD.2. R EMINDERS Reminder on RKHS norm
Let κ a kernel, and H the reproducing kernel Hilbert space (RKHS) corresponding to the kernel κ .
Recall that the RKHS norm of a function f ( x ) =
α T κ ( x , X ) is :
[Equation] [Equation]
[Equation]
Reminder on Generalization and Rademacher Complexity Consider a loss function l : R × R → R .
The population loss over the distribution D , and the empirical loss over n samples
D = { ( x i , y i ) , i ∈ [ n ] } from the same distribution D are deﬁned as :
[Equation]
[Equation] [Equation]
Theorem 4 Suppose the loss function is bounded in [0 , c ] and is ρ − Lipschitz in the ﬁrst argument.
Then, with probability at least 1 − δ over sample S of size n :
[Equation] [Equation]
D.2.
B
OUNDING
(cid:13)(cid:13)(cid:13)
[Equation]
f (cid:63)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Lemma 2 Let H τ the Hilbert space associated to the kernel κ τ .
We recall that :
[Equation] [Equation]
Then :
[Equation]
Proof
We start from the deﬁnition of the RKHS norm of ˜ f (cid:63) τ :
[Equation] [Equation]
Since ( κ τ ( X τ , X τ ) + λ I ) − 1 ≤ ( κ τ ( X τ , X τ ))
−≤ ˜y Tτ (
[Equation]
τ
[Equation]
[Equation] [Equation]
( X
[Equation]
, X
[Equation] [Equation]
1 ˜y
[Equation] [Equation]
D.2. B OUNDING THE R ADEMACHER C OMPLEXITY
The goal of this section is to upper bound the Rademacher Complexity.
First, we derive a general upper bound for the Rademacher Complexity of a linear combination of kernels in Lemma 3 .
Then, we apply this bound to the linear combination of kernels obtained through Thm. 1, which describes the model through the Continual Learning.
A general bound for the Rademacher Complexity
Lemma 3 (Rademacher Complexity of a linear combination of kernels) Let κ t : X × X
→ R , t ∈ [ T ] kernels such that :
[Equation]
To every kernel κ t , we associate a feature map φ t : X → H t , where H t is a Hilbert space with inner product (cid:104)· , ·(cid:105)H t , and for all x , x (cid:48) ∈ X , κ t ( x , x (cid:48) )
= (cid:104) φ t ( x ) , φ t ( x (cid:48) )
(cid:105)H t
For all T ∈ N (cid:63) , given a sequence { B τ , τ ∈ [ T ] } we deﬁne F T as follows :
[Equation]
Let X 1 , · , X n be random elements of X .
Then for the class F , we have :
[Equation]
Proof
Let f ∈ F , and let x ∈ X :
[Equation]
For all τ ∈ [ T ]
[Equation]
we associate a feature map φ τ : X → H τ ∀ x , x
[Equation]
Therefore :
[Equation]
On the other hand
[Equation]
the following holds ∀ t ∈
[ T ] :
[Equation]
Therefore :
[Equation]
Now
[Equation]
we derive an upper bound of the Rademacher complexity of F T :
[Equation]
Now we apply on each function f
τ the upper bound from Lemma 22 by Bartlett & Mendelson
[Equation]
Bounding the Rademacher Complexity for Continual Learning
Lemma 4 Keeping the same notations and setting as Theorem 3, the Rademacher Complexity can be bounded as follows:
[Equation]
where F T is the function class spanned by the model up to the task T τ .
Proof
For all T ∈ N (cid:63)
[Equation]
given a sequence { B τ , τ ∈ [ T ] } we deﬁne F T as follows :
[Equation]
where we set B τ as :
[Equation]
Following Lemma 2 for all τ ∈ [ T ] :
[Equation]
[Equation]
which means that the function class F t contains the Continual Learning model up to task T τ .
We apply Lemma 3 in order to upper bound the Rademacher complexity :
[Equation]
We made the assumption that for all τ ∈ [ T ] tr( κ τ ( X τ
[Equation]
τ
[Equation]
O
[Equation]
D.2. B OUNDING THE EMPIRICAL LOSS FOR OGD
Lemma
Given a current task T T , the empirical losses on the data from all previous tasks ( T τ , τ ≤
T ) can be bounded as follows :
Let T ∈ N ﬁxed.
Then, for all τ ∈ [ T ]
[Equation]
[Equation] [Equation]
Case 1 -Task T T Proof
We start from the deﬁnition of the empirical loss :
[Equation]
We replace into f (cid:63)T by its expression from Theorem 1 :
[Equation]
Now, we apply Lemma 2 in order to upper bound the right hand side norm :
[Equation]
Case 2 - Tasks {T τ , τ ∈ [1 , T − 1] } :
The proof is very similar to Case 1, we apply Theorem 2, which is the key property of OGD in terms of no forgetting.
Proof
We start from the deﬁnition of the empirical loss :
[Equation]
Now, applying Theorem 2 which implies that f (cid:63)τ ( X τ ) =
f (cid:63)T ( X τ ) :
[Equation]
Then with a similar analysis as Case 1, we get :
[Equation]
D.2. B OUNDING THE EMPIRICAL LOSS FOR SGD
As opposed to the analysis for OGD, forgetting can occur for OGD and Theorem 2 is no longer valid.
Similarly to the OGD empirical loss analysis, we study the case on the data from the last task and the case on the data from all the other tasks.
Lemma 6 The empirical losses on the source and target tasks can be bounded as follows :
Let T ∈ N ﬁxed.
Then, for all τ ∈ [ T ]
[Equation]
For all τ ∈ [ T − 1]
[Equation]
Case 1 -Task T T
For this case, the analysis is the same as for OGD, as no forgetting occurs with respect to the data the model is being trained on.
Case 2 - Tasks {T τ , τ ∈ [1 , T − 1] } : Since SGD is not guaranteed to be robust to Catastrophic
Forgetting, this case comprises additional residual terms that correspond to forgetting.
Proof
Let τ ∈
[ T − 1]
[Equation]
From Corr.
[Equation]
we recall that :
[Equation]
Then :
[Equation]
We can upper bound A similarly to the previous paragraphs :
[Equation]
Now
[Equation]
we upper bound B .
Let k ∈
[ τ +
[Equation]
T ] :
[Equation]
We conclude by plugging back the upper bounds of A and B
[Equation]
Therefore :
[Equation]
D.2. P ROOF OF THE G ENERALISATION T HEOREM
(T HM . 3)
Now, we prove the Theorem 3 by applying the lemmas we developed above.
Proof
With probability 1 − δ we have : sup
[Equation]
Then, by replacing into L S τ ( f (cid:63)T ) with Lemmas 5 and 6, we get the upper bounds of the theorem for the various cases of SGD and OGD.
[Equation]
D.2. A LTERNATIVE PROOF FOR THE EMPIRICAL LOSS BOUNDS - N O REGULARISATION CASE
This section is complementary and aims to provide a better interpretation of the bounds in Theorem 3.
We prove that in the case where there is no regularisation, the training error converges to zero.
This result illustrates better the intuition behind the λ term that appears in the upper bound of the Theorem 3, which is under the regularisation assumption.
First, we derive the differential equation of the model’s output dynamics in Lemma 7.
Then, we apply this Lemma to prove the convergence to zero in Lemma 8.
Finally, in Lemma 9, we show that for the past tasks, the training error equals to zero all the time.
The training dynamics of OGD
Lemma 7 (Differential equation of the model’s output dynamics) Let T ∈ N (cid:63) , the index of the task currently being trained on, with OGD.
For all τ < T , deﬁning (cid:101) u
τ
as : (cid:101)
[Equation]
The model’s output dynamics while training on the task T τ is as follows :
[Equation]
Proof
Let τ ∈ N (cid:63) , we derive the dynamics while training on the task T τ , while training on this task.
First, we deﬁne :
[Equation]
We also deﬁne the projection matrix P τ ∈ R p × p as :
[Equation]
The matrix P τ performs the projection from the original weight space R p to the trainable weight space during training on task T τ , which corresponds to ( τ (cid:76)
[Equation]
Our starting point is the SGD update rule :
[Equation]
where :
[Equation]
Therefore
[Equation]
we get the following expression for the gradient of the loss :
[Equation]
Then, the following holds :
[Equation]
It follows that :
[Equation]
The training error converges to zero
Lemma
For all tasks T τ , τ ∈ [ T ] , the training error
L τ ( w τ ( t )) converges to 0 when t tends to inﬁnity .
Proof
We start with the deﬁnition of the training error on task T :
[Equation]
Under the assumption that κ τ ( X τ , X τ ) is positive deﬁnite, for η ≤
[Equation]
X
[Equation]
the eigenvalues become less than 1, therefore :
[Equation]
The training error on the past tasks is zero
Lemma
For all tasks T τ , τ ∈ [ T ] , the training error
L S τ ( f (cid:63)T ) = 0 .
Proof This lemma follow immediately from Lemma 8 which states that the training error converges to zero for OGD on all tasks and Theorem 2 which states that the training error on the past tasks is unchanged.
E M ISSING PROOFS OF A PP A - T HE IMPORTANCE OF THE NTK VARIATION FOR C ONTINUAL L EARNING E. P ROOF OF P ROPOSITION 1 - OGD IMPLIES A-GEM-NT
We present the proof of Proposition 1, which relies mainly on Theorem 2 stating the robustness of
OGD to Catastrophic Forgetting.
Proof
Let T τ a task ﬁxed, we recall that in the NTK regime, the model can be expressed as :
[Equation]
Given a task T k and its associated memory M k , we recall that the loss can be expressed as :
[Equation]
Similarly to the A-GEM paper, we deﬁne the gradients vectors g and { g k , k ∈ [ τ − 1] } as :
[Equation] [Equation]
Now we extend the expressions :
[Equation]
For OGD, following Thm.
2 variant for ﬁnite memory :
[Equation]
Finally, since the gradient updates are orthogonal to M k :
[Equation]
Therefore, for all k ∈ [ τ − 1] :
[Equation]
We conclude, OGD implies learning with GEM and A-GEM with no Positive Backward Transfer.
[Equation]
F E XPERIMENTS F. R EPRODUCIBILITY F.1. A RCHITECTURES
Due to the memory limitations encountered while running OGD and OGD+ on the CIFAR100 and CUB200 dataset, we use smaller architectures which are different from Jung et al.
For the MNIST benchmark, we keep the same architecture as Farajtabar et al. MNIST : Similarly to Farajtabar et al. (2019), the neural network is a three-layer MLP withhidden units in two layers, each layer uses RELU activation function.
The model has 10 logits, which do not use any activation function.
CIFAR100 : The neural network is a multi-head LeNet Lecun et al.
(1998) network with Batch Normalisation Ioffe & Szegedy (2015) and 200 hidden units for the penultimate layer.
CUB200 : Similarly to Jung et al. (2020), our base architecture is AlexNet (Krizhevsky et al., 2012). In order to scale for the OGD type algorithms, we changed the classiﬁer module to a smaller 3 layer
RELU neural network, with dropout (Table 5).
[Table] [Table]
E XPERIMENT SETUP
We run each experiment 5 times, the seeds set is the same across all experiments sets.
We report the mean and standard deviation of the measurements.
F.1. OGD+ I MPLEMENTATION D ETAILS
Memory :
During the OGD+ memory update step, for each task, the new associated feature maps replace the memory slots of the previous feature maps for the same task.
The goal is to ensure a balance of the feature maps from all tasks in the memory.
Multi-head models : For the dataset streams Split MNIST and Split CIFAR-100, we consider multi-headed neural networks.
We only store the feature maps with respect to the shared weights, the projection step is not performed for the heads’ weights
. F.1. H YPERPARAMETERS
We use the the same hyperparameters as Farajtabar et al.
for the algorithms SGD, OGD and
OGD+, on the MNIST benchmarks. We also keep a small learning rate, in order to preserve the locality assumption of OGD, and in order to verify the conditions of the theorems.
We report the shared hyperparameters across the benchmarks and algorithms in Table 6. For the other benchmarks and algorithms, we report the grid search space in Sec.
F.1.5.
We present our hyperparameter search ranges for each Continual Learning method. We selected the hyperparameter sets that maximise the average accuracy.
The results and scripts of the hyperparameter search and the best hyperparameters are provided on the corresponding github repository.
[Table]
MNIST Benchmarks
• EWC, SI and MAS : we ﬁxed the seed to 0, then performed a grid search over the regularisation parameter in [0.1, 1, 10, 100, 1.000, 10.000] • SGD, OGD and OGD+ : we used the same hyperparameters as Farajtabar et al. • Stable SGD : we ﬁxed the seed to 0 then performed a grid search over all combinations of : – gamma : [0.5, 0.6, 0.7, 0.8, 0.9] – learning rate : [0.01, 0.1, 0.25] – batch size : [10, 32, 64] – dropout [0, 0.1, 0.2, 0.3, 0.4, 0.5]
CIFAR100 and CUB200 Benchmarks
• EWC, SI and MAS
: we ﬁxed the seed to 1, then performed a grid search over : – regularisation coefﬁcient : [0.1, 1, 10, 100, 1.000, 10.000] – learning rate : [0.00001, 0.001, 0.01, 0.1] – batch size : [32, 64, 256] • SGD, OGD and OGD+ :
For the MNIST benchmarks, we used the same hyperparameters as Farajtabar et al.
While for the CIFAR100 and CUB200 benchmarks
[Table]
we run the following grid search : – learning rate : [0.00001, 0.001, 0.01, 0.1] – batch size : [32, 64, 256] – epochs : [1, 20, 50] • Stable SGD : we performed the following grid search : – gamma : [0.5, 0.6, 0.7, 0.8, 0.9] – learning rate : [0.001, 0.01, 0.1] – batch size : [10, 64] – dropout [0, 0.1, 0.2, 0.3, 0.4, 0.5] – epochs : [1, 10, 50]
F. E XPERIMENTS : M EMORISATION PROPERTY OF OGD (T HM . 2) F.2. T HE IMPORTANCE OF THE OVERPARAMETERIZATION ASSUMPTION
Thm.
2 states that in the NTK regime, given an inﬁnite memory, the train error of OGD is unchanged.
Experiments We track the variation of the train accuracy of the samples in the memory through tasks after being trained on all subsequent tasks.
We consider the MLP hidden layer size as a proxy for overparameterization.
We run the experiments on the Permuted MNIST and Rotated MNIST benchmarks, with the OGD algorithm.
We vary the hidden size from 100 to 400.
Results Figure 3 shows that the variation of the train accuracy of OGD+ decreases uniformly with the hidden size.
This result indicates the forgetting of OGD decreases with overparameterization.
F.2. T HE IMPORTANCE OF THE MEMORY SIZE ASSUMPTION
Thm.
2 states that in the NTK regime, given an inﬁnite memory, the train error of OGD is unchanged.
Experiments We track the variation of the train accuracy of the samples in the memory through tasks after being trained on all subsequent tasks, as a function of the memory size per task.
Results Figure 4 shows that the mean train accuracy variation decreases uniformly with the memory size.
F. T HE IMPORTANCE OF THE NTK VARIATION FOR C ONTINUAL L EARNING (S EC . 6)
In this section, we present extended results on the importance of the NTK variation for Continual
Learning.
For completion, we present the variation of the test accuracy of the model at the end of training for all tasks.
Also, we run the benchmarks on the A-GEM algorithm as a supporting materials for the discussion in App. A. [Table] F.3.
[Table]
OTATED MNIST
[Table]
[Figure]
F. C OMPARISON WITH OTHER C ONTINUAL L EARNING BASELINES
In order to get a broader picture on the robustness of OGD+ to Catastrophic Forgetting, we benchmark it against standard Continual Learning baselines.
F.4. E VALUATION
Denoting a T,τ the accuracy of the model on the task T τ after being trained on the task T T , and ¯ b the vector of test accuracies at initialisation, we track the following metrics presented by Chaudhry et al.
[Equation]
and Lopez-Paz & Ranzato (2017) :
[Equation]
F. B ASELINES
In addition to the standard baselines, SGD, EWC, SI and MAS, we compare OGD+ against OGD and
Stable SGD, which was recently introduced by Mirzadeh et al.
They show that Stable SGD outperforms OGD on all benchmarks, in order for the evaluation to be as fair and comprehensive as possible, we also include this method in our benchmarks.
F.5. C OMPLEMENTARY R ESULTS - C ONTINUAL L EARNING M ETRICS
Permuted MNIST Table 9 shows that OGD+ outperforms the baselines on AAC, while it remains competitive on AFM and BWT.
Method
AAC
BWT
FWT
AFM
Naive SGD 76.31 (±1.89) -19.27 (±2.0) 1.12 (±1.
[Table]
Rotated MNIST Table 10 shows that OGD+ is competitive with the baselines on AAC and AFM, while it underperforms on the other metrics.
Also it draws an improvement over OGD on AAC, BWT and AFM. One probable reason is the relatively low overparameterization of the setting as shown in Table 1.
[Table]
Split CIFAR100 Table 11 shows that OGD+ is not competitive on Split CIFAR100 on all metrics.
Also it draws the same performance as OGD on all metrics, one probable reason is the overparameterization of the setting as shown in Table
[Table]
CUB200 Table 12 shows that OGD+ is not competitive on the CUB200 benchmark on all metrics.
Also it draws the same performance as OGD on all metrics, one probable reason is the overparameterization of the setting as shown in Table 1.
[Table]
F.5. C OMPLEMENTARY R ESULTS - V ALIDATION ACCURACY THROUGH TRAINING
[Figure]