R EGULARIZING T RAJECTORIES TO M ITIGATE C ATASTROPHIC F ORGETTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective.
However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters.
In this paper we argue for the importance of regularizing optimization trajectories directly.
We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks.
We show that using the conatural gradient systematically reduces forgetting in continual learning.
Moreover, it helps combat overﬁtting when learning a new task in a low-resource scenario
I NTRODUCTION
It is good to have an end to journey toward; but it is the journey that matters, in the end.
Ursula K. Le Guin
Endowing machine learning models with the capability to learn a variety of tasks in a sequential manner is critical to obtain agents that are both versatile and persistent.
However, continual learning of multiple tasks is hampered by catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990), the tendency of previously acquired knowledge to be overwritten when learning a new task.
Techniques to mitigate catastrophic forgetting can be roughly categorized into 3 lines of work (see
Parisi et al.
for a comprehensive overview
regularization-based approaches, where forgetting is mitigated by the addition of a penalty term in the learning objective (Kirkpatrick et al.
; Chaudhry et al.
(2018a), inter alia ), 2.
dynamic architectures approaches, which incrementally increase the model’s capacity to accomodate the new tasks (Rusu et al., 2016), and 3.
memorybased approaches, which retain data from learned tasks for later reuse (Lopez-Paz & Ranzato, 2017;
Chaudhry et al., 2018b; 2019).
Among these, regularization-based approaches are particularly appealing because they do not increase the model size and do not require access to past data.
This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons.
Moreover, they are of independent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation (Kirkpatrick et al., 2017).
A good regularizer ensures that, when learning a new task, gradient descent will ultimately converge to parameters that yield good results on the new task while preserving performance on previously learned tasks.
Critically, this is predicated upon successful optimization of the regularized objective, a fact that has been largely taken for granted in previous work.
Non-convexity of the loss function, along with noise in the data (due to small or biased datasets) or in the gradients (due to stochastic gradient descent), can yield optimization trajectories — and ultimately convergence points — that are highly non-deterministic, even for the same starting parameters.
As we demonstrate in this paper, this can cause unintended catastrophic forgetting along the optimization path.
This is illustrated in a toy setting in Figure 1: a two parameter model is trained to perform task T 2 (an arbitrary bi-modal loss function) after having learned task T 1 (a logistic regression task).
Standard ﬁnetuning, even in 1 We commit to releasing the code to implement our method and reproduce our experiments upon acceptance.
[Figure]
and the presence of a regularized objective (EWC; Kirkpatrick et al.
(2017)), quickly changes the loss of T 1 and tends converge to a solution with high T 1 loss.
We propose to remedy this issue by regularizing the optimization trajectory itself, speciﬁcally by preconditioning gradient descent with the empirical Fisher information of previously learned tasks ( § 3).
This yields what we refer to as a co-natural gradient, an update rule inspired by the natural gradient (Amari, 1997), but taking the Fisher information of previous tasks as a natural Riemannian metric 2 of the parameter space, instead of the Fisher information of the task being optimized for.
When we introduce our proposed co-natural gradient for the toy example of Figure 1, the learning trajectory follows a path that changes the loss on T 1 much more slowly, and tends to converges to the optimum that incurs the lowest performance degradation on T 1 .
We test the validity of our approach in a continual learning scenario ( § 4).
We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of ≈ 1 . 5 to 9 , and greatly improves performance over simple ﬁnetuning, without modiﬁcation to the training objective.
We further investigate the special case of transfer learning in a two-task, low-resource scenario.
In this speciﬁc case, control over the optimization trajectory is particularly useful because the optimizer has to rely on early stopping to prevent overﬁtting to the meager amount of training data in the target task.
We show that the co-natural gradient yields the best trade-offs between source and target domain performance over a variety of hyper-parameters ( § 5).
B ACKGROUND AND N OTATIONS
We ﬁrst give a brief overview of the continual learning paradigm and existing approaches for overcoming catastrophic forgetting.
2. N OTATION
Let us deﬁne a task as a triplet containing an input space X and an output space Y , both measurable spaces, as well as a distribution D over X × Y .
In general, learning a task will consist of training a model to approximate the conditional distribution p ( y | x ) induced by D .
Consider a probabilistic model p θ parametrized by θ ∈ R d where d is the size of the model, trained to perform a source task S
=
(cid:104)X S , Y S , D S (cid:105) to some level of performance, yielding parameters θ S .
In the most simple instance of continual learning, we are tasked with learning a second target task
T =
(cid:104)X T , Y T , D T (cid:105) .
In general in a multitask setting, it is not the case that the input or output spaces are the same.
The discrepancy between input/output space can be addressed in various ways, e.g. by adding a minimal number of task-speciﬁc parameters (for example, different softmax layers for different label sets).
To simplify exposition, we set these more speciﬁc considerations aside for now, and assume that X S = X T and Y S
= Y T .
At any given point during training for task T , our objective will be to minimize the loss function
L T ( θ ) – generally the expected log-likelihood E x,y ∼D T [ − log p θ ( y | x )] .
Typically, this will be performed by iteratively adding incremental update vectors δ ∈ R d to the parameters θ ←− θ + δ .
2. E XISTING A PPROACHES FOR C ONTINUAL L EARNING
In this paper, we focus on those models that have a ﬁxed architecture over the course of continual learning.
The study of continual learning for models of ﬁxed capacity can be split into two distinct (but often overlapping) streams of work:
Regularization-based approaches introduce a penalty in the loss function L T , typically quadratic, pushing the weights θ back towards θ S :
[Equation]
where Ω S is a matrix, typically diagonal, that encodes the respective importance of each parameter with respect to task S , and λ is a regularization strength hyper-parameter.
Various choices have been proposed for Ω S ; the diagonal empirical Fisher information matrix (Kirkpatrick et al., 2017), or pathintegral based importance measures (Zenke et al., 2017; Chaudhry et al., 2018a).
More elaborate regularizers have been proposed based on e.g. a Bayesian formulation of continual learning (Nguyen et al., 2017; Ahn et al., 2019) or a distillation term (Li & Hoiem, 2016; Dhar et al., 2019).
The main advantage of these approaches is that they do not rely on having access to training data of previous tasks.
Memory-based approaches store data from previously seen tasks for re-use in continued learning, either as a form of constraint, by e.g. ensuring that training on the new task doesn’t increase the loss on previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b), or for replay i.e. by retraining on instances from previous tasks (Rebufﬁ et al., 2017; Chaudhry et al., 2019; Aljundi et al., 2019b;a).
Various techniques have been proposed for the selection of samples to store in the memory (Chaudhry et al., 2019; Aljundi et al., 2019b) or for retrieval of the samples to be used for replay Aljundi et al. (2019a).
All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay, with the notable exception of GEM (Lopez-Paz & Ranzato, 2017;
Chaudhry et al., 2018b), where the gradients are projected onto the orthogonal complement of previous task’s gradients.
However, this method has been shown to perform poorly in comparison with simple replay (Chaudhry et al., 2019), and it still necessitates access to data from previous tasks.
R EGULARIZING THE T RAJECTORY
After brieﬂy recalling how the usual update is obtained in gradient descent, we derive a new, conatural update designed to better preserve the distribution induced by the model over previous tasks
. 3. W ARM UP : THE S TANDARD G RADIENT D ESCENT U PDATE
At point θ in the parameter space, gradient descent ﬁnds the optimal update δ that is (1) small and (2) locally minimizes the decrease in loss L ( θ + δ ) − L ( θ ) ( ≈ δ (cid:124) ∇ θ L at the ﬁrst order).
Traditionally this can be formulated as minimizing the Lagrangian:
[Equation]
[Table]
[Equation]
with Lagrangian multiplier µ > 0 .
Minimizing L for δ yields the well-known optimal update δ ∗ :
[Equation]
where 12 µ corresponds to the learning rate (see Appendix A.1 for the full derivation).
3. KL R EGULARIZATION OF T RAJECTORIES
The (cid:107) δ (cid:107) 2 term in L implicitly expresses the underlying assumption that the best measure of distance between parameters θ and θ + δ is the Euclidean distance.
In a continual learning setting however, the quantity we are most interested in preserving is the probability distribution that θ models on the source task S :
[Equation]
Therefore, a more natural distance between θ and θ + δ is the Kullback-Leibler divergence
KL ( p Sθ (cid:107) p
Sθ + δ ) (Kullback & Leibler, 1951).
For preventing catastrophic forgetting along the optimization path, we incorporate incorporate this KL term into the Lagrangian L itself:
[Equation]
Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least.
Notably, this is not a function of the previous objective L S , so knowledge of the original training objective is not necessary during continual learning (which is typically the case in path-integral based regularization methods (Zenke et al., 2017) or experience replay (Chaudhry et al., 2019)).
3. C O - NATURAL G RADIENT O PTIMIZATION
Presuming that δ is small, we can perform a second order Taylor approximation of the function δ (cid:55)→
KL ( p Sθ (cid:107) p
Sθ + δ ) around 0.
Considering that both the zeroeth and ﬁrst order terms are null because 0 is a global minimizer of δ (cid:55)→ KL ( p Sθ (cid:107) p Sθ + δ ) , this reduces the Lagrangian to a quadratic optimization problem (we refer the reader to Pascanu & Bengio (2013) for a more detailed derivation.):
[Equation]
where F Sθ is the Hessian of the KL divergence around θ .
A crucial, well-known property of this matrix is that it coincides with the Fisher information matrix 3 E x,y ∼ p θ
[( ∇ log p Sθ )( ∇ log p Sθ ) T ] (the expectation being taken over the model’s distribution p θ ; see Appendix A.1 for details).
This is appealing from a computational perspective because the Fisher can be computed by means of ﬁrst order derivatives only.
Minimizing for δ yields the following optimal update:
[Equation]
where coefﬁcients µ and ν are folded into two hyper-parameters: the learning rate λ and a damping coefﬁcient α (the step-by-step derivation can be found in Appendix A.1).
In practice, especially with low damping coefﬁcients, it is common to obtain updates that are too large (typically when some parameters have no effect on the KL divergence).
To address this, we re-normalize δ ∗ to have the same norm as the original gradient, (cid:107)∇L T (cid:107) .
For computational reasons, we will make 3 key practical approximations to the Fisher:
F Sθ ≈ F Sθ S : we maintain the Fisher computed at θ S , instead of recomputing F S at every step of training.
This relieves us of the computational burden of updating the Fisher for every new value of θ .
F S is diagonal: this is a common approximation in practice with two appealing properties.
First, this
makes it possible to store the d diagonal Fisher coefﬁcients in memory.
Second
, this trivializes the inverse operation (simply invert the diagonal elements).
Empirical
Fisher
: this
common approximation replaces
the expectation under the model’s distribution by the expected log-likelihood of the true distribution:
E x,y ∼ p S
[( ∇ log p Sθ )( ∇ log p Sθ ) T ] (mind the subscript).
This is particularly useful in tasks with a large or unbounded number of classes ( e.g. structured prediction), where summing over all possible outputs is intractable.
We can then compute the diagonal of the empirical
Fisher using Monte Carlo sampling: 1 N (cid:80) Ni =1 [ ∇ log p
Sθ ( y i | x i )] 2 with ( x i , y i ) sampled from
D S (we use N = 1000 for all experiments).
This formulation bears many similarities with the natural gradient from Amari (1997), which also uses the KL divergence as a metric for choosing the optimal update δ ∗ .
There is a however a crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature of the KL divergence of D T to speed-up convergence, our proposed method leverages the curvature of the KL divergence on D S to slow-down divergence from p Sθ S .
To highlight the resemblance and complementarity between these two concepts, we refer to the new update as the co-natural gradient.
3. B EYOND T WO T ASKS
In a continual learning scenario, we are confronted with a large number of tasks T 1 . . .
T n presented in sequential order.
When learning T n , we can change the Lagrangian L from 5 to incorporate the constraints for all previous tasks T 1 . . .
T n − 1 :
[Equation]
[Equation]
=
[Equation]
(cid:124)
[Equation]
L
[Equation]
n + µ (cid:107) δ (cid:107) 2 + n − 1 (cid:88) i
=
ν i KL ( p T
i θ (cid:107)
[Equation]
This in turn changes the Fisher in Eq.
8 to ˜ F n − 1 := 12 (cid:80) n − 1 i =
ν i F T i .
The choice of the coefﬁcients ν i is crucial.
Setting all ν i to the same value, i.e. assigning the same importance to all tasks is suboptimal for a few reasons.
First and foremost, it is unreasonable to expect of a model with ﬁnite capacity to remember an unbounded number of tasks (as tasks “ﬁll-up” the model capacity, ˜
F n − 1 is likely to become more “homogeneous”).
Second, as training progresses and θ changes, our approximation that F T i θ ≈ F T i θ Ti is less and less likely to hold.
We address this issue in the same fashion as Schwarz et al.
(2018), by keeping a rolling exponential average of the Fisher matrices:
[Equation]
In this case, previous tasks are gracefully forgotten at an exponential rate controlled by γ .
We account for the damping α term in Eq.
7 by setting ˜ F 0 :
=
αγ I .
In preliminary experiments, we have found γ = 0 .
9 to yield consistently good results, and use this value in all presented experiments.
C ONTINUAL L EARNING E XPERIMENTS 4. E XPERIMENTAL SETTING
To corroborate our hypothesis that controlling the optimization trajectory with the co-natural gradient reduces catastrophic forgetting, we perform experiments on two continual learning testbeds:
• Split CIFAR :
The CIFAR100 dataset, split into 20 independent 5-way classiﬁcation tasks.
Similarly to Chaudhry et al.
(2018b), we use a smaller version of the ResNet architecture (He et al., 2016). [Table]
[Table] • Omniglot : the Omniglot dataset (Lake et al., 2015) consists of 50 independent character recognition datasets on different alphabet.
We adopt the setting of Schwarz et al.
and consider each alphabet as a separate task.
On this dataset we use the same small CNN architecture as Schwarz et al.
• Split MiniImageNet : The MiniImageNet dataset (a subset of the popular ImageNet (Deng et al., 2009) dataset 5 ; Vinyals et al. Split the dataset into 20 independent 5-way classiﬁcation tasks, similarly to Split CIFAR, and use the same smaller ResNet.
We adopt the experimental setup from Chaudhry et al.
(2019): in each dataset we create a “validation set” of 3 tasks, used to select the best hyper-parameters, and keep the remaining tasks for evaluation.
This split is chosen at random and kept the same across all experiments.
In these datasets, the nature and possibly the number of classes for each task changes.
We account for this by training a separate softmax layer for each task, and apply continual learning only to the remaining, “feature-extraction” part of the model.
We report results along two common metrics for continual learning: average accuracy , the accuracy at the end of training averaged over all tasks, and forgetting .
Forgetting is deﬁned in Chaudhry et al.
(2018a)
as the difference in performance on a task between the current model and the best performing model on this task.
Formally if A Tt represents the accuracy on task T at step t of training, the forgetting F Tt at step t is deﬁned as F Tt = max τ<t
A Tτ − A Tt .
‘Low forgetting’ means that the model tend to keep the same level of performance on a task it has learned.
We implement the co-natural update rule on top of 3 baselines:
• Finetuning : Simply train the model on the task at hand, without any form of regularization.
• EWC : Proposed by Kirkpatrick et al.
(2017), it is a simple but effective quadratic regularization approach.
While neither the most recent nor sophisticate regularization technique, it is a natural baseline for us to compare to in that it also consists in a Fisher-based penalty — although in the perform [Figure] loss function instead of the optimization dynamics.
We also use the rolling Fisher described in Section 3.4, making our EWC baseline equivalent to the superior online EWC introduced by Schwarz et al.
• ER : Experience replay with a ﬁxed sized episodic memory proposed by Chaudhry et al. While not
directly comparable to
EWC in that it presupposes access to data from previous tasks, ER is a simple approach that boasts the best performances on a variety of benchmarks
(Chaudhry et al., 2019).
In all experiments , we use memory size 1,000 with reservoir sampling.
Training proceeds as follows: we perform exhaustive search on all the hyper-parameter combinations using the validation tasks.
Every combination is reran 3 times (the order of tasks, model initialization and order of training examples changes with each restart), and rated by accuracy averaged over tasks and restarts.
We then evaluate the best hyper-parameters by continual training on the evaluation tasks.
Results are reported over 5 random restarts (3 for MiniImageNet), and we control for statistical signiﬁcance using a paired t-test (we pair together runs with the same task ordering).
We refer to Appendix A.2 for more details regarding ﬁne-grained design choices.
4. R ESULTS
The upper half of Table 1 reports the average accuracy of all the tasks at the end of training (higher is better).
We observe that the co-natural gradient always improves greatly over simple ﬁnetuning, and occasionally over EWC and ER.
We note that on both datasets, bare-bone co-natural ﬁnetuning matches or exceeds the performance of EWC and ER even though it requires strictly fewer resources (no need to store the previous parameters as in EWC, or data in ER).
Even more appreciable is the effect of the co-natural trajectories on forgetting, as shown in the lower half of Table 1.
As evidenced by the results in the lowest row, using the co-natural gradient systematically results in large drops in forgetting across all approaches and both datasets, even when the average accuracy is not increased.
To get a qualitative assessment of the learning trajectories that yield such results, we visualize the accuracy curves of 10 out of the 47 evaluation tasks of Omniglot in Figure 2.
We observe that previous approaches do poorly at keeping stable levels of performance over a long period of time (especially for tasks learned early in training), a problem that is largely resolved by the co-natural preconditioning.
This seems to come at the cost of more intransigence (Chaudhry et al., 2018a), i.e. some of the later tasks are not being learnt properly.
In models of ﬁxed capacity, there is a natural trade-off between intransigence and forgetting (see also the “stability-plasticity” dilemma in neuroscience Grossberg (1982)).
Our results position the co-natural gradient as a strong lowforgetting/moderate intransigence basis for future work.
L OW -R ESOURCE A DAPTATION E XPERIMENTS
In this section we take a closer look at the speciﬁc case of adapting a model from a single task to another, when we only have access to a minimal amount of data in the target task.
In this case, [Figure] controlling the learning trajectory is particularly important because the model is being trained on an unreliable sample of the true distribution of the target task, and we have to rely on early-stopping to prevent overﬁtting.
We show that using the co-natural gradient during adaptation helps both at preserving source task performance and reach higher overall target task performance.
5. E XPERIMENTAL S ETTING
We perform experiments on two different scenarios:
Image classiﬁcation We take MiniImagenet as a source task and CUB (a 200-way birds species classiﬁcation dataset; Welinder et al.
as a target task.
To guarantee a strong base model despite the small size of MiniImageNet, we start off from a ResNet18 model (He et al., 2016) pretrained on the full ImageNet, which we retroﬁt to MiniImageNet by replacing the last fully connected layer with a separate linear layer regressed over the MiniImageNet training data.
To simulate a low-resource setting, we sub-sample the CUB training set to 200 images ( ≈ 1 per class).
Scores for these tasks are reported in terms of accuracy.
Machine translation We consider adaptation of an English to French model trained on WMT (a dataset of parallel sentences crawled from parliamentary proceedings, news commentary and web page crawls; Bojar et al.
to MTNT (a dataset of Reddit comments; Michel & Neubig (2018)).
Our model is a Transformer (Vaswani et al., 2017) pretrained on WMT15.
Similarly to CUB, we simulate a low-resource setting by taking a sub-sample of 1000 sentence pairs as a training set.
Scores for these two datasets are reported in terms of BLEU score.
6 (Papineni et al., 2002)
Here we do not allow any access to data in the source task when training on the target task.
We compare four methods Finetuning (our baseline), Co-natural ﬁnetuning , EWC (which has been proven effective for domain adaptation, see Thompson et al.
and Co-natural EWC .
Given that different methods might lead to different trade-offs between source and target task performance, with some variation depending on the hyper-parameters ( e.g. learning rate, regularization strength...), we take inspiration from Thompson et al.
and graphically report results for all hyper-parameter conﬁguration of each method on the 2 dimensional space deﬁned by the score on source and target tasks 7 .
Additionally, we highlight the Pareto frontier of each method i.e. the set of conﬁgurations that are not strictly worse than any other conﬁguration for the same model.
5. R ESULTS
The adaptation results for both scenarios are reported in Figure 3.
We ﬁnd that in both cases, the co-natural gradient not only helps preserving the source task performance, but to some extent it also allows the model to reach better performance on the target task as well.
We take this to corroborate our starting hypothesis: while introducing a regularizer does help, controlling the optimization dynamics actively helps counteract overﬁtting to the very small amount of training data, because the co-natural pre-conditioning makes it harder for stochastic gradient descent to push the model towards directions that would also hurt the source task.
C ONCLUSION
We have presented the co-natural gradient, a technique that regularizes the optimization trajectory of models trained in a continual setting.
We have shown that the co-natural gradient stands on its own as an efﬁcient approach for overcoming catastrophic forgetting, and that it effectively complements and stabilizes other existing techniques at a minimal cost.
We believe that the co-natural gradient — and more generally, trajectory regularization — can serve as a solid bedrock for building agents that learn without forgetting.
A A PPENDIX A. D ERIVATIONS A.1. T HE S TANDARD G RADIENT U PDATE (E QUATION 3)
We derive the standard update in Eq. 3 by solving the Lagrangian L in Eq.
2 for δ .
Given that its ﬁrst and second derivatives are:
[Equation]
the problem is trivially strictly convex and its global minimizer δ ∗ satisﬁes:
[Equation]
(cid:3)
A.1. E QUIVALENCE OF THE H ESSIAN OF THE KL D IVERGENCE AND THE F ISHER I NFORMATION M ATRIX
To simplify notation, let us perform the change of variables θ + δ → x .
We show that the Hessian of the KL coincides with the Fisher on θ : in other words, ∇ 2 KL ( p θ (cid:107) p x ) (cid:12)(cid:12) x
= θ =
E p θ [( ∇ log p θ )( ∇ log p θ ) (cid:124) ] .
Under mild regularity assumptions 8 , we can write:
[Equation]
Now note that ∇ 2 log p x can be rewritten via standard derivatives manipulations as ∇ 2 p x p x −
[Equation]
This leads to:
[Equation]
When taken at θ , the ﬁrst term evaluates to 9 :
[Equation]
By using the identity ∇ p x p x = ∇ log p x and evaluating at x = θ , the second term gives us:
[Equation]
A.1. O BTAINING THE C O - NATURAL U PDATE (E QUATION 7)
We solve the Lagrangian from Eq.
6 in a similar fashion as in A.1.1.
First we compute its gradient and Hessian:
[Equation]
While not as straightforwardly as the one in A.1.1, this problem is also strongly convex: indeed
F Sθ is positive semi-deﬁnite (as an expectation of PSD matrices) and the addition of µI ensures that ∇ 2 L is positive deﬁnite.
We ﬁnd the unique solution by solving:
[Equation]
Set λ := 12 µ and α :
=
νµ to get Eq.
7 (cid:3)
A. A DDITIONAL E XPERIMENTAL S ETTINGS FOR C ONTINUAL L EARNING
This section is intended to facilitate the reproduction of our results.
The full details can be found with our code at anonymized url .
A.2. S PLIT CIFAR
We split the dataset into 20 disjoint sub-tasks with each 5 classes, 2500 training examples andtest examples.
This split, performed at random, is kept the same across all experiments, only the order of these tasks is changed.
During continual training, we train the model for one epoch on each task with batch size 10, following the setup in Chaudhry et al.
(2018b).
A.2. O MNIGLOT
We consider each alphabet as a separate task, and split each task such that every character is present 12, 4 and 4 times in the training, validation and test set respectively (out of the 20 images for each character).
During continual training, we train for 2500 steps with batch size 32 (in keeping with
Schwarz et al.
We ignore the validation data and simply evaluate on the test set at the end of training.
A.2. G RID - SEARCH PARAMETERS
For each method, we perform grid-search over the following parameter values:
• Learning rate (all methods): 0.1, 0.03, 0.
• Regularization strength (EWC, Co-natural EWC): 0.5, 1,• Fisher damping coefﬁcient (Co-natural ﬁnetuning, Co-natural EWC): 0,1.0,0.1 for Split CIFAR and 0,0.1,0.01 for Omniglot
For ER, we simply set the batch size to the same value as standard training (10 and 32 for Split
CIFAR and Omniglot respectively).
Note that whenever applicable, we re-normalize the diagonal
Fisher so that the sum of its weights is equal to the number of parameters in the model.
This is so that the hyper-parameter choice is less dependent on the size of the model.
In particular this means that the magnitude of each diagonal element is much bigger, which is why we do grid-search over smaller regularization parameters for EWC.