L INEAR MODE CONNECTIVITY IN MULTITASK AND CONTINUAL LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to ﬁnd a solution that performs well on all considered tasks.
The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting.
That is, the solution found for a subsequent task does not perform well on the previous ones anymore.
However, the relationship between the different minima that the two training regimes arrive at is not well understood.
What sets them apart?
Is there a local structure that could explain the difference in performance achieved by the two different schemes?
Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected.
We empirically ﬁnd that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both.
We thoroughly analyze this observation and discuss its signiﬁcance for the continual learning process.
Furthermore, we exploit this ﬁnding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution.
We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.
I NTRODUCTION
[Figure]
g
[Figure]
f o r t a
[Figure]
k
[Figure]
l o w e
rr
o r f o r b
o t h
[Figure]
considered.
First ˆ w 1 is learned on task 1.
Afterwards we either reach ˆ w 2 by learning second task or w ∗ 2 by training on both tasks simultaneously w ∗
ˆ w 1 and between w ∗
ˆ w 2 .
One major consequence of learning multiple tasks in a continual learning (CL) setting — where tasks are learned sequentially, and the model can only have access to one task at a time — is catastrophic forgetting (McCloskey & Cohen, 1989).
This is in contrast to multitask learning (MTL), where the learner has simultaneous access to all tasks, which generally learns to perform well on all tasks without suffering from catastrophic forgetting.
This limitation hinders the ability of the model to learn continually and efﬁciently.
In recent years, several approaches have been proposed to tackle this problem.
They have mostly tried to mitigate catastrophic forgetting by using different approximations of the multitask loss.
For example, some regularization methods take a quadratic approximation of the loss of previous tasks (e.g. Kirkpatrick et al., 2017;
Yin et al., 2020).
As another example, rehearsal methods attempt to directly use compressed past data either by selecting a representative subset (e.g. Chaudhry et al., 2019; Titsias et al., 2019) or relying on generative models (e.g. Shin et al., 2017; Robins, 1995).
In this work, we depart from the literature and start from the non-conventional question of understanding “What is the relationship, potentially in terms of local geometric properties, between the multitask and the continual learning minima?” .
Our work is inspired by recent work on mode connectivity (Draxler et al., 2018; Garipov et al., 2018;
Frankle et al., 2020) ﬁnding that different optima obtained by gradient-based optimization methods are connected by simple paths of non-increasing loss.
We try to understand whether the multitask and continual solutions are also connected by a manifold of low error, and what is the simplest form this manifold can take.
Surprisingly, we ﬁnd that a linear manifold, as illustrated in Fig.
1 right, reliably connects the multitask solution to the continual ones, granted that the multitask shares same initialization with the continual learning as described below.
This is a signiﬁcant ﬁnding in terms of understanding the phenomenon of catastrophic forgetting through the lens of loss landscapes and optimization trajectory and also for designing better continual learning algorithms.
To reach this conclusion, we consider a particular learning regime described in Fig.
1 left, where after learning the ﬁrst task using the data D 1 , we either sequentially learn a second task obtaining ˆ w 2 or continue by training on both tasks simultaneously (i.e., train on D 1 + D 2 ), obtaining the multitask solution w ∗ 2 .
We investigate the relationship between the two solutions ˆ w 2 and w ∗ 2 .
Note that w ∗ 2 is not the typical multitask solution, which would normally start from w 0 and train on both datasets.
We chose this slightly non-conventional setup to minimize the potential number of confounding factors that lead to discrepancies between the two solutions (Fort et al., 2019).
We also rely on the observation from (Frankle et al., 2020) that initialization can have a big impact on the connectivity between the solutions found on the same task, and sharing the same starting point, as we do between ˆ w 2 and w ∗ 2 , might warrant a linear path of low error between the two solutions.
Moreover, Neyshabur et al.
(2020) noted that in the context of transfer learning, there is no performance barrier between two minima that start from pre-trained weights, which suggests that the pre-trained weights guide the optimization to a ﬂat basin of the loss landscape.
In contrast, barriers clearly exist if these two minima start from randomly initialized weights.
Our contributions can be summarized as follows:
To the best of our knowledge, our work is the ﬁrst to study the connectivity between continual learning and multitask learning solutions
We show that compared to conventional similarity measures such as Euclidean distance or Central Kernel Alignment (Kornblith et al., 2019), which are incapable of meaningfully relating these minima, the connectivity through a manifold of low error can reliably be established.
And this connecting path is linear, even when considering more than 20 tasks in a row.
Motivated by this we propose an effective CL algorithm
(Mode Connectivity SGD or MC-SGD) that is able to outperform several established methods on standard CL benchmarks.
1. R ELATED W ORK
With the trending popularity of deep learning, continual learning has gained a critical importance because the catastrophic forgetting problem imposes key challenges to deploy deep learning models in various applications (e.g Lange et al., 2019; Kemker et al., 2018).
A growing body of research has attempted to tackle this problem in recent years (e.g Parisi et al., 2018;
Toneva et al., 2018;
Nguyen et al., 2019; Farajtabar et al., 2019; Hsu et al., 2018; Rusu et al., 2016; Li et al., 2019; Kirkpatrick et al., 2017; Zenke et al., 2017; Shin et al., 2017; Rolnick et al., 2018; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b; Riemer et al., 2018; Mirzadeh et al., 2020; Wallingford et al., 2020).
Among these works, our proposed MC-SGD bares most similarities to rehearsal based methods such us (e.g. Shin et al., 2017; Chaudhry et al., 2018b) and regularization based methods (e.g. Kirkpatrick et al., 2017; Zenke et al., 2017) similar to (Titsias et al., 2019).
Following (Lange et al., 2019), one can categorize continual learning methods into three general categories, based on how they approach dealing with catastrophic forgetting.
Experience replay : These methods build and store a memory of the knowledge learned so far (Rebufﬁ et al., 2016; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Riemer et al., 2018; Rios & Itti, 2018; Zhang et al., 2019). As examples, Averaged Gradient Episodic Memory (A-GEM) (Chaudhry et al., 2018b) builds an episodic memory of parameter gradients, while ER-Reservoir (Chaudhry et al., 2019) uses a Reservoir sampling method to maintain the episodic memory.
Regularization : These methods explicitly apply regularization techniques to ensure parameters do not change too much (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017; Aljundi et al., 2018; Kolouri et al., 2019). They can either a Bayesian (Nguyen et al., 2017; Titsias et al., 2019; Schwarz et al., 2018; Ebrahimi et al., 2019; Ritter et al., 2018) or frequentist views (Farajtabar et al., 2019; He & Jaeger, 2018; Zeng et al., 2018). For instance, Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2019) projects the prediction gradients from new tasks on the subspace of previous tasks’ gradients to preserve the knowledge.
Parameter isolation : Parameter isolation methods allocate different subsets of the parameters to each task (Rusu et al., 2016; Yoon et al., 2018; Jerfel et al., 2019; Rao et al., 2019; Li et al., 2019).
From the stability-plasticity perspective, these methods implement gating mechanisms that improves the stability and controls the plasticity by activating different gates for each task.
Masse et al.
(2018) proposes a bio-inspired approach for a context-dependent gating that activates non-overlapping subset of parameters for any speciﬁc task.
Supermask in Superposition (Wortsman et al., 2020) is another parameter isolation method that starts with a randomly initialized, ﬁxed base network and for each task ﬁnds a subnetwork (supermask) such that the model achieves good performance.
Recently,
Mirzadeh et al.
(2020) have shown that dropout implicitly creates different pathways or gates for tasks, therefore, it reduces their mutual interference and leads to less forgetting.
Continual learning as a problem expands beyond dealing with catastrophic forgetting, one of the hopes behind sequential learning is that it can potentially enable positive forward transfer, as one can build on the previously acquired knowledge.
In this sense it connects to topics such as MetaLearning (Beaulieu et al., 2020;
Jerfel et al., 2019; He & Jaeger, 2018; Riemer et al., 2018), Few-Shot
Learning (Wen et al., 2018; Gidaris & Komodakis, 2018), Multi-task and Transfer Learning (He &
Jaeger, 2018; Jerfel et al., 2019).
It also aims to work in scenarios where task boundaries are not well deﬁned or provided, or when the data distribution shifts slowly or when a multi-task solution does not exist (Rao et al., 2019;
Aljundi et al., 2019;
He et al., 2019; Kaplanis et al., 2019).
Mode connectivity (Draxler et al., 2018; Garipov et al., 2018), is a novel tool to understand the loss landscape of deep neural networks.
It postulates that different optima obtained by gradient-based optimization methods are connected by simple paths of non-increasing loss (i.e., low-loss valleys).
Recently, various works provided different theoretical explanations for mode connectivity in different scenarios (Venturi et al., 2019; Kuditipudi et al., 2019) either by relying on over-parametrization or concepts such as noise-stability or dropout-stability.
(Neyshabur et al., 2020) investigated the connection between minima obtained by pre-trained models versus freshly initialized ones.
They note that there is no performance barrier between solutions coming from pre-trained models, but there can be a barrier between solutions of different randomly initialized models.
(Frankle et al., 2020) shows that different minima that share the same initialization point are connected by a linear path, even with weight pruning.
We rely on this observation when designing our setting that the multitask and continual learning minima share a common starting point.
T HE RELATION BETWEEN MULTITASK AND CONTINUAL MINIMA
One question driving this work is understanding the relationship between the two different solutions: multitask learning vs. continual learning.
In particular, we are interested in scenarios where a multitask solution for the considered tasks exists (for a discussion when this does not hold see (He et al., 2019)) and when both learning regimes have the same objective, ﬁnding a solution that performs well on all tasks.
We posit that this difference can not be reliably explained by simple and typical distance metrics used to measure similarity between the trained models.
In particular we consider
Euclidean distance and Central Kernel Alignment (CKA)
(Kornblith et al., 2019).
However, these solutions are connected by paths of low error, and, provided that learning starts from the same initial conditions, these paths can have a linear form.
In Fig.
2 left column we can see the performance of Naive SGD for multitask and continual learning.
Details on the experimental setup can be found in the Appendix C.
The dashed line represents the multitask solution at convergence, which achieves strong performance on all tasks.
It further shows the performance of all tasks during the sequential learning experience (each point represents the performance after learning another task), highlighting how performance on past tasks degrades considerably.
Note that as described in Fig.
1 and further detailed in Appendix D.2 Fig.
15, in the multitask learning scenario, tasks are added sequentially to the loss, to construct a parallel with the continual learning setting.
This will be the case throughout this work.
Eucledian distance .
It might be reasonable to expect that the less the parameter changes, the less the forgetting will be.
One can motivate this heuristic on a Taylor expansion of the loss, as done in (Mirzadeh et al., 2020), where, the forgetting is deﬁned as:
[Equation]
Here, L 1 ( w ) is the empirical loss for task 1, and λ maxis the largest eigenvalue of its Hessian at ˆ w 1 .
Note that all terms of the Taylor expansion are multiplied with ˆ w 2 − ˆ w 1 and hence the norm of this [Figure] delta will affect the amount of forgetfulness.
In fact, this is frequently done when pruning neural networks (Zhu & Gupta, 2018; Han et al., 2015), where weights are zeroed out based on magnitude, producing minimal Euclidean distance to unprunned model.
But, as observed in Fig.
2 (middle column), Eucledian distance does not correlate with not suffering from catastrophic forgetting: the CL solution on task 1 ( ˆ w 1 ) is closer to the CL solution of task 5 ( ˆ w 5 ) than the multitask solution of tasks 5 ( w ∗ 5 ).
One explanation could be that the bound deﬁned by Eq.
(1) will not be tight if the vector ˆ w 5 − ˆ w 1 does not lie in the direction of the largest eigenvector.
Appendix D.1 contains further details on this topic.
Centered Kernel Alignment .
Centered Kernel Alignment (CKA) (Kornblith et al., 2019) measures the similarity of two representations on the same set of examples.
Given N examples and two activation outputs on these examples, R 1 ∈ R N × d 1 and R 2 ∈ R N × d 2 , CKA is deﬁned by:
[Equation]
where, (cid:107) .
(cid:107) F is the Frobenius norm.
Recent work by Ramasesh et al.
(2020) studies catastrophic forgetting on the CIFAR dataset by measuring the CKA similarity score of different layers of ˆ w 1 and ˆ w 2 .
They argue that the later layers suffer more from catastrophic forgetting by showing that
the
CKA similarity of initial layers decreases less after training on sequential tasks.
However, the CKA score suffers from a few shortcomings.
If the number of training epochs per task is small (e.g., in streaming case), the CKA does not change much, even though the accuracy for previous tasks drops drastically.
For instance, in Fig.
2 right column, we show that the pairwise CKA between different layers of the ﬁrst task minimum ( ˆ w 1 ) and CL and multitask minima of task 2 and task 5 are roughly the same.
Although, the phenomenon observed in (Ramasesh et al., 2020) is still realizable by a very tiny margin.
Moreover, we can see that the multitask minimum of task 2 ( w ∗ 2 ), and task 5 ( w ∗ 5 ) are more similar to ˆ w 1 compared to CL minima ( ˆ w 2 and ˆ w 5 ).
2. M ODE C ONNECTIVITY
Mode connectivity has been studied empirically (Draxler et al., 2018; Garipov et al., 2018) and theoretically with some assumptions (e.g., over-parameterization or noise stability) (Kuditipudi et al., 2019; Venturi et al., 2019).
We note that these previous works were focused on “single-task” setup, where the data stays i.i.d. , and more importantly, the networks start from different initialization.
Because of this assumption, it is shown that the minima are not linearly connected ( i.e. , the loss is not necessarily decreasing in the interpolation line between the two minima).
Instead, the path consists of several line-segments or curves (Kuditipudi et al., 2019).
However, between our continual and multitask settings, a more favorable property can exist, namely that learning starts from the same initialization (i.e., ˆ w 1 = w ∗ 1 ).
For a “single task”, common initialization leads to linear path between minima (Frankle et al., 2020).
Therefore, our ﬁrst step is to investigate whether the same linear connectivity holds between multitask and CL solutions.
As we show in Fig.
3, this is true for both MNIST and CIFAR benchmarks.
In this ﬁgure, we can see that the multitask minima for task 2 ( w ∗ 2 ) is connected to both task 1 CL minima ( ˆ w 1 ) and task 2 CL minima ( ˆ w 2 ).
We note that the surface plots in Fig.
3 are not of low-dimensional projection types and are rather “interpolation plots“ in parameter space computed on the hyper-plane connecting these three minima.
More details are provided in Appendix C.5.
[Table]
[Figure]
[Figure]
line between the ﬁrst task solution( ˆ w 1 ) and subsequent continual solutions, while the loss remains low on the interpolation line between ˆ w 1 and subsequent multitask minima (a and c). The same observation also holds for the second task solution ( ˆ w 2 ) (b and d)
To demonstrate linear connectivity holds for subsequent MTL and CL minima, we plot the validation loss on the line-segment interpolating between all subsequent minima for both task 1 and 2 on Rotated
MNIST and CIFAR in Fig.
The low loss interpolation to w ∗ ’s indicates that indeed along this linear path, the loss stays low.
W HEN AND WHERE DOES LINEAR MODE CONNECTIVITY HOLD ?
The fact that linear regions of low error connect MTL and CL minima does not necessarily imply it is trivial to ﬁnd one such path.
In this section, we try to analyze this ﬁnding through the geometry of loss functions and, more speciﬁcally, its curvature, which identiﬁes the amount of change when we move from solution of task 1 ( ˆ w 1 ) to either MTL ( w ∗ 2 ) or CL ( ˆ w 2 ) solution.
This way, we can explain why existing methods such as EWC that are also trying to penalize the change in directions of high curvature fail to ﬁnd a plausible direction.
The short answer is that they only rely on the second-order
Taylor approximation of the loss function, which may be easily violated.
We elaborate as follows. Let us start by trying to understand what is implied by the fact that these linear paths exist.
Focusing on the linear path between ˆ w 1 and w ∗ 2 , one potential justiﬁcation is that while minimizing the multitask loss, the updates move us in the direction of low curvature for the Hessian of the ﬁrst loss around ˆ w 1 .
To see why, let’s make the mild assumption that the function is smooth, which typically tends to hold in the over-parameterized regime (e.g., similar assumptions are needed for the NTK regime to hold (Bietti & Mairal, 2019)).
Then we can not have a rapid change in the curvature.
Hence, if moving along the line connecting ˆ w 1 and w ∗ 2 , the loss of task 1 stays very small, it means the curvature also has to be small as it can not show rapid ﬂuctuations that will not be reﬂected in the loss.
Implicitly the curvature is low at ˆ w 1 as well; hence this direction necessarily has to be within the subspace spanned by the eigenvectors with low curvature.
This seems not to hold for ˆ w 1 and ˆ w 2 , so we might expect that the line connecting them to move in directions of higher curvature.
A symmetric argument can be made for w ∗ 2 and ˆ w 2 , looking now at the Hessian and loss of task 2.
The smoothness argument makes it unlikely that curvature shows rapid ﬂuctuations; hence loss staying low implies the alignment with eigenvectors of low curvature.
Before moving forward, let us empirically validate the argument in the previous paragraph. To do so, we estimate the top 50 eigenvalues/eigenvectors of the Hessian matrix around task 1 minima of rotation MNIST benchmark, similar to the setting we discussed in Section 2.
We use the deﬂated power iteration method to obtain the eigenspectrum of the Hessian.
Fig.
5a shows the largest eigenvalues.
We note that similar to the observation by Fort & Ganguli (2019) and Mirzadeh et al.
(2020), the eigenspectrum of the Hessian includes a bulk plus C larger outlier eigenvalues where C is the number of classes.
To quantify the conﬁnement between the subspace spanned by the Hessian eigenvectors and the movement directions, we measure the cosine angle of these directions and the Hessian eigenvectors.
Fig.
5b shows that the direction from ˆ w 1 to ˆ w 2 lies within the subspace spanned by the top eigenvectors.
Note that this agrees with our hypothesis.
However, the direction between ˆ w 1 to w ∗ 2 does not lie within this subspace as the cosine angle between this direction and the eigenvectors are all near zero, suggesting that this direction is orthogonal to all these eigenvalues and hence is spanned by eigenvectors with low eigenvalues.
The orthogonality has also been tried as a hard constraint in a number of continual learning papers explicitly (Farajtabar et al., 2019; He & Jaeger, 2017).
This explains the less suffering from catastrophic forgetting as we move in directions of low curvature. [Figure] i [Figure]
n
[Figure]
a l
[Figure]
− ˆ w 1 ,~v ) | | cos( ˆ w 2 − ˆ w 1 ,~v ) | (b) 1 5 10 15 20 25 30 35 40 45 50 Eigenvectors (largest → smallest ) 0 .0 .
C o s i n e S i m i
l a r i t y | cos( w ∗ 2 − ˆ w 1 ,~v ) | | cos( ˆ w 2 − ˆ w 1 ,~v )
|
[Figure]
Now, let’s examine whether moving in the direction of low curvature is sufﬁcient in addition to being a necessary condition or not?
Fig.
5c shows that this is not sufﬁcient.
By increasing the number of epochs from 5 to 20 we show that both directions to ˆ w 2 and w ∗ 2 do not lie in this subspace of the top eigenvectors.
That is, for both, if we move on the segment away from ˆ w 1 , we are aligned with none of the directions associated with high curvature.
But ˆ w 2 suffers from catastrophic forgetting and hence a considerable increase in the loss of task 1, while w ∗ 2 does not.
This suggests that while initially moving towards ˆ w 2 the curvature stays low, but eventually higherorder terms increase the curvature of task 1 loss before reaching ˆ w 2 .
Overall this implies the multiple w ∗ that we observe, which gradually solve more and more tasks, lie within a region where a secondorder Taylor approximation of the ﬁrst task still holds, and higher-order derivatives do not play an important role.
And within this region, learning is restricted to the subspace spanned only by eigenvectors corresponding to small eigenvalues for all previous tasks.
We empirically show that we can learn up to 20 complex tasks and up to 50 permuted MNIST tasks in this regime (see Fig.
7 here and 20 in Appendix D.5), hence it seems this region, where the second-order Taylor approximation holds, is quite expressive and sufﬁcient to learn many tasks without interference between them.
Additionally, similar linearizations seem to hold reliably in over parametrized models throughout learning as shown by the NTK-based literature (e.g Jacot et al., 2018; Bietti & Mairal, 2019) and these assumptions are implicitly made by several successful continual learning algorithms (e.g. Kirkpatrick et al., 2017; Farajtabar et al., 2019; Mirzadeh et al., 2020).
In particular, this observation is interesting when viewed from the perspective of regularization-based CL methods.
Many of these methods rely on a second-order Taylor expansion (if not of previous losses then of the KL term with respect to how much the model changed) and forbid change in weights with high curvature.
The fact that multitask learning adheres to the same principles, moves in the direction of low curvature and restricts itself to the region where this approximation holds, suggests that this remains a promising direction.
It also highlights the importance of better approximations of curvature and the importance of remaining within the area where this approximation holds.
In Appendix B, we further investigate different aspects of our setting that might make it possible to ﬁnd reliable solutions for all subsequent tasks in an area around the solution of the ﬁrst task where the second-order Taylor approximation holds.
We show, that as assumed earlier, sharing the same initial condition is vital for linear connectivity to occur.
Furthermore, the shared structure between tasks like low-level features in images or semantically consistent labeling plays an important role as well.
If we destroy any structure between tasks, we can not ﬁnd a multitask solution for task 1 andthat is linearly connected to the ﬁrst task.
C ONTINUAL L EARNING WITH M ODE C ONNECTIVITY
Based on the observation in Section 2.1 that there exist minima with linear mode connectivity to previous CL minima, in this section, we propose a continual learning algorithm that exploits this property.
The proposed algorithm can be viewed as a combination of both replay and regularization methods in continual learning.
4. MC-SGD
Mode Connectivity SGD (MC-SGD) is a light-weight method that constrains the continual minima to lie in a low-loss valley to all previous minima.
For simplicity, we start by providing the loss function for two tasks, then we extend it for an arbitrary number of tasks.
We deﬁne the loss objective that enforces linear connectivity to both ˆ w 1 and ˆ w 2 as below:
[Equation]
[Equation]
where L 1 and L 1 are the loss function for task 1 and task 2 respectively and α parameterizes the line connecting w to ˆ w 1 and ˆ w 2 respectively in the ﬁrst and second term.
Essentially, Eq.
(3) constrains the MC-SGD minima to have a low-loss path to both ˆ w 1 and ˆ w 2 .
The integral in this equation can be approximated by averaging the loss over a few random α in [0 , 1] .
However, our experiments showed that picking as few as n = 5 equally spaced points between 0 and 1 is more than sufﬁcient to get good results.
We can further decompose (3) into:
[Equation]
It’s evident from Eq.
(4) that ¯ w should minimize both task 1 and task 2 loss, in addition to having low-loss paths to both of the CL minima.
In a continual learning setting, however, we would not have access to L 1 once we start learning on
L 2 .
Therefore we rely on experience replay like ER (Chaudhry et al., 2019) and AGEM (Chaudhry et al., 2018b).
We use a small replay buffer with randomly sampled examples to approximate L 1 , with the hope that given the restricted form of Eq.
(4) through the additional regularizer, we might need lesser data then if we would have used it to approximate directly the multitask loss.
We can further extend the mode connectivity loss to an online version that is cheap to compute:
[Equation]
Eq.
(5) can be viewed as follows: when learning task t , assuming that we have a high-performance solution for the previous t − 1 tasks (i.e., ¯ w t − 1 ), we try to minimize Eq.
(3) as if in this equation, ˆ wis ¯ w t − 1 and ˆ w 2 is ˆ w t .
The replay memory still consists of samples from all the previous t − 1 tasks.
E XPERIMENTS AND RESULTS
The experimental setup, such as benchmarks, network architectures, continual learning setting (e.g., number of tasks, episodic memory size, and training epochs per task), hyper-parameters, and evaluation metrics are chosen to be similar to several other studies (Chaudhry et al., 2018b; Mirzadeh et al., 2020; Chaudhry et al., 2019; Farajtabar et al., 2019; Chaudhry et al., 2019).
For all experiments, we report the average and standard deviation over ﬁve runs with different random seeds.
Appendix C provides a detailed discussion on our experimental setup.
Benchmarks.
We report on three standard continual learning benchmarks: Permuted MNIST (Goodfellow et al., 2013), Rotated MNIST, and Split CIFAR-100.
Although we are aware of the shortcomings of Permuted MNIST (Farquhar & Gal, 2018), for the sake of consistency with literature, we report the result on this benchmark.
However, we also report our results on Rotated MNIST and
CIFAR-100, that are more challenging and realistic datasets for continual learning, especially when the number of tasks is large.
Each task of permuted MNIST is generated by random shufﬂing of the pixels of images such that the permutation remains the same for the images of within the same task, but changes across different tasks.
Rotated MNIST is generated by the continual rotation of the
MNIST images where each task applies a ﬁxed random image rotation (between 0 and 180 ◦ ).
Split
CIFAR is a variant of CIFAR-100 where each task contains the data from 5 random classes.
Architectures.
For the rotation and permutation MNIST benchmarks, we use a two-layer MLP with 256 ReLU units in each layer.
For the split CIFAR-100 benchmark, we use a ResNet18, with three times fewer feature maps across all layers.
Evaluation.
Following several studies in the literature (Chaudhry et al., 2018b; Mirzadeh et al., 2020), we report two metrics to compare CL algorithms when the number of tasks is large.
(1) Average Accuracy: The average validation accuracy after the model has been continually learned task t , is deﬁned by A t = 1 t[-(cid:80) t i =
-]
[Equation]
t i =1 a t,i , where, a t,i is the validation accuracy on dataset i after the model ﬁnished learning task t .
(2) Average Forgetting: The average forgetting measures the backward transfer in continual learning.
This metric is calculated by the difference of the peak accuracy and ending accuracy of each task, after the continual learning experience has ﬁnished.
For a continual learning benchmark with T tasks, it is deﬁned by F =
[Equation]
{ 1 ,...,T − 1 } ( a t,i − a T,i ) .
Further details are provided in the
Appendix C.
5. C OMPARISON WITH OTHER METHODS
In this experiment, we compare MC-SGD with various established continual learning algorithms.
Our setup consists of 20 tasks for three benchmarks: permuted MNIST, Rotated MNIST, and split
CIFAR-100.
The episodic memory size for A-GEM, ER-Reservoir, and MC-SGD is limited to be one example per class per task 1 .
Table 1 compares the average accuracy and average forgetting (i.e., A t and F deﬁned above) for each method once the continual learning experience is ﬁnished (i.e., after learning task 20).
Moreover,
Fig.
6 shows the evolution of average accuracy for each method on each benchmark.
We can see in the ﬁgure that MC-SGD outperforms other methods with its performance gap increasing as the number of tasks increases.
In Appendix D.5, we show this trend also holds when the number of tasks increases from 20 to 50 for Permuted MNIST benchmark.
Overall, the results validate a few aspects of our method.
MC-SGD seems to make more efﬁcient use of the episodic memory compared to replay based techniques A-GEM and ER-Reservoir, perhaps due to incorporating the additional knowledge of linear connectivity as a regularizer.
Moreover, if we see our approach as restricting learning in the subspace of low curvature, this subspace is better approximated compared to EWC, which takes an explicit Taylor approximation of the KL term and adds a similar constraint (that learning can not change weights with high curvature).
[Table]
5. M ODE C ONNECTIVITY OF THE MC-SGD MINIMA
Next
[Table]
we show that by minimizing Eq.
(5), the minima found by MC-SGD are almost linearly connected to both the starting point of learning a new task (which represents the multitask solution of
[Figure]
all previous task) as well as the solution of the task being learned.
Fig.
7 shows the interpolation plots for rotated MNIST and split CIFAR-100.
In Fig.
7a, we show the validation loss for task 1 on the interpolation line between ˆ w 1 and four subsequent MC-SGD, and continual minima obtained during the learning of all subsequent tasks.
We choose ˆ w 5 and ¯ w 5 as minima in the early stages of learning, ˆ w 10 and ¯ w 10 for the middle stages of learning, and ˆ w 15 , ¯ w 15 , ˆ w 20 , and ¯ w 20 for the late stages of learning in our illustrations.
We can see the losses on the interpolation lines between CL and MC minima are nearly ﬂat compared to the losses on the lines among CL minima.
Moreover, Fig.
7b shows the interpolation plot for task 5 to make sure the conclusion holds for later minima as well.
Similarly, Figs.
7c and 7d show the interpolation plots of split CIFAR-100 for task 1 and 5.
[Figure]
C ONCLUSION
While both continual and multitask learning aim to ﬁnd solutions that perform well across all tasks
[Figure]
the difference in their training regimes leads to two very different outcomes: multitask learning typically ends up achieving this goal, while continual learning suffers from catastrophic forgetting, performing well only on the recently seen tasks.
In this work we investigated the relationship between these two solutions when all other potential confounding factors are minimized.
We considered a simple training regime in which multitask learning and continual learning start from similar conditions when incorporating a new task.
We showed that in this condition, multitask minima are connected to continual learning ones by a linear path of low error, while continual learning solutions are not similarly connected.
This can be understood through the glance of Taylor expansions as multitask objective restricts learning in the directions of low curvature within an area where a second order approximation holds.
Such solutions seem to exists even when the process is applied repeatedly, solving more than 20 tasks in a row.
Finally, we explored this observation and proposed a new algorithm for continual learning called
Mode Connectivity SGD (MC-SGD).
It relies on the assumption that there always exist a solution able to solve all seen tasks so far that is connected by a linear path of low loss.
MC-SGD utilizes a replay buffer to approximate the loss of tasks that are not observed.
However, compared to two other popular rehearsal-based methods, it performs better with less data as it exploits the linear path to further constrain learning.
The proposed algorithm performs surprisingly well on a few classical benchmarks for continual learning.
We believe that the efﬁciency of this algorithm, as well as the analysis of the local structure of the loss surface around the continual learning and multitask learning shed more light on how multitask learning manages to achieve better performance than continual learning.
A O UTLINE
The appendix is organized as follows:
Appendix B provides detailed discussion on the required assumptions for mode connectivity.
Appendix C gives a comprehensive discussion on the experimental setup for each experiment, hyperparameters, visualization, and implementation.
In addition, we provide anonymous links to the code and details to reproduce the results.
Appendix D contains additional experiments and results, including detailed plots for mode connectivity of continual and multitask minima in Sec. 2, and an additional experiment on permuted MNIST dataset with 50 tasks.
B B REAKING THE LINEAR CONNECTIVITY BETWEEN THE CONTINUAL LEARNING AND MULTITASK SOLUTION B. D IFFERENT INITIALIZATION
Furthermore, it is useful to understand if we can break the linear path between the multitask and continual learning minima.
An initial assumption was that this is caused by the sharing of the same initialization, where both ˆ w 2 and w ∗ 2 are learned starting from ˆ w 1 .
Fig.
8 shows how violating this assumption breaks the linear mode connectivity for the split CIFAR-100 benchmark.
Fig.
8a shows the validation loss interpolation between ˆ w 1 and the the next four multitask and CL minima.
As illustrated, the multitask minima is no longer linearly connected to ˆ w 1 and is separated by a wall.
The same phenomenon holds for the next minima, and in Fig.
8c, we show this for ˆ w 2 .
We note that for MNIST datasets, as reported by Frankle et al.
(2020), the linear connectivity still holds even if the MTL minimum starts from other initialization points.
Related to this, Neyshabur et al.
(2020) investigated the connection between minima obtained by pre-trained models versus freshly initialized ones.
They pointed out that there is no performance barrier between solutions coming from pre-trained models, but there can be a barrier between solutions of different randomly initialized models.
Here, ˆ w 1 acts like a pre-trained model for multitask objective, placing the optimizer in a good basin for task 1.
[Figure]
B. S HARED STRUCTURE OR SEMANTICS OF DATA
Another potentially important role might be played the shared structure between tasks.
If the different tasks do not share a common structure it might be unlikely that there exists a solution for one task in the neighborhood of the other.
To manipulate the shared structure of tasks, we can either corrupt input images or corrupt the labels.
To increase the distribution shift across tasks, in this experiment, we use a continual learning setup with two tasks:
The ﬁrst task is the MNIST dataset
[Figure]
and the second task is the Fashion-MNIST dataset.
To corrupt the input data, we add different amounts of independent Gaussian noise to the input images of the second task (i.e, Fashion MNIST) and report the results in Fig.
Fig.
9b to 9d show the gradual separation of w ∗ 2 and ˆ w 2 as the amount of noise increases.
The validation loss values on the linear interpolation between these two minima are also shown in Fig. 9a.
Fig. 10b to 10d show the effect of randomly changing the labels for different portions of the images of task 2.
As illustrated, the more labels we corrupt, the more disconnected ˆ w 2 and w ∗ 2 become.
The loss values on the linear interpolation path between these minima are shown in Fig.
10a for different percentage of label corruption.
[Figure] [Figure]
0 .2 .5 .ˆ
ww ∗ˆ w00.50.
>
V a li d a t i o n L o ss (b) No label corruption     ˆ ww ∗ˆ w00.50.
>
V a li d a t i o n L o ss (c) 5% label corruption     ˆ ww ∗ˆ w00.50.
> V a li d a t i o n L o ss ( [Figure]
[Figure]
corruption Yet a very trivial case of breaking mode connectivity is to cut a few classes from the second task. To this end, we perform the following experiment: After training on the ﬁrst task and obtaining ˆ w 1 , for the multitask training on the second task, we randomly choose k classes out of the 10 classes and remove the ﬁrst task’s examples that belong to any of these k classes from the training dataset of multitask model.
In Fig.
11a, we show that by removing different number of classes from the training set of multitask model, the linear mode connectivity does not hold anymore as the validation loss on task 1 increases on the line between ˆ w 1 and w ∗ 2 .
Moreover, while without removing any classes we have linear mode connectivity as illustrated in Fig. 11b, by removing 2 classes, ˆ w 2 is no longer connected to ˆ w(Fig. 11c).
The trend continues if we remove more classes (i.e, 4 classes in Fig. 11d).
[Figure]
(a)      ˆ ww ∗ˆ w0 0.2 0. 1 1.53> V a li d a t i o n L o ss ( T a s k 1 ) (b) Keep all classes      ˆ ww ∗ˆ w0 0.2 0.
1 1.53>
V a li d a t i o n
L o ss ( T a s k 1 ) (c) Remove 2 classes      ˆ ww ∗ˆ w0 0.2 0.
1 1.53>
V a li d a t i o n
L o ss ( T a s k 1 ) (d) Remove 4 classes we remove more and more classes
We emphasize that for common continual learning benchmarks, the assumption of linear mode connectivity holds, and this section aims to push the linear mode connectivity to its boundaries in order to appropriately scope this interesting phenomenon that has signiﬁcant consequences to understanding and development of continual and multitask learning.
C D ETAILS OF EXPERIMENTAL SETUP
In this section, we answer two important questions regarding our experiments and result:
What decisions did we make for our experiments (e.g., continual learning setup, model architectures, benchmarks, etc.)?
Why did we make those decisions and how close our setup is to other established continual learning works?
To this end, we start with the ﬁrst question by reviewing our experimental setup, followed by hyperparameters we used in the main experiments.
To answer the second question, in each sub-section, we show that our setup is close to several established works in the continual learning literature.
C. B ASELINES AND A RCHITECTURES
We chose the baselines and architectures based on three criteria.
The ﬁrst criterion is the diversity of methods.
In addition to the naive SGD, we have selected
A-GEM (Chaudhry et al., 2018b) and ER-Reservoir (Chaudhry et al., 2019) as representative of episodic memory methods, EWC (Kirkpatrick et al., 2017) for regularization methods.
The second criterion is reproducibility.
While there are several other continual learning methods, we only reported the methods that we could replicate their results either by their ofﬁcial implementations or self-implementations.
The ﬁnal selection criterion is performance.
A-GEM and ER-Reservoir are high-performance episodic memory methods.
In addition, we have added the stable-SGD (Mirzadeh et al., 2020) as the current state of the art continual learning method, in addition to the previous state of the art methods such as A-GEM and ER-Reservoir.
At the end of this section, we show that by carefully choosing a similar continual learning setup to common setup in the literature, we have made it easier to compare our results with other works that we could not replicate ourselves.
C. C ONTINUAL L EARNING S ETUP
In the MNIST and CIFAR experiment in Section 2, we have chosen 5 tasks so that we can show microscopic metrics (e.g., the evolution of accuracy for all tasks) thorough the continual learning experience.
In this experiment, our goal was not to reduce catastrophic forgetting.
In contrast, in that experiment, we have aggravated the catastrophic forgetting to show that CKA and Euclidean distance are not helpful, even if the amount of forgetting is high.
More speciﬁcally, we have not tuned the hyper-parameters, and we increased the catastrophic forgetting by using a per task rotation of 22 . 5 ◦ in the rotation MNIST benchmark.
Moreover, for MNSIT experiment, we have used a two-layer
MLP with 100 ReLU units.
The ResNet18 in that experiment is similar to the ResNet18 in Section 5.
Finally, in this setup, we have used the learning rate of 0 .
1 for MNIST dataset and 0 .
for the CIFAR experiment with batch size of 64.
Finally, in this experiment, the number of of epochs per task is 5.
We discuss the rationale behind this setup in Appendix C.
In the comparison experiment in section 5.1, we have increased the number of tasks to 20 tasks, and the model is trained on each task for one epoch.
This benchmark is more challenging since we used one example per class per task (i.e., 200 for rotated and permuted MNIST, and 100 for split
CIFAR-100).
Thus, the model is allowed to see the data only once, and hence, the setting becomes closer to the real-world online setting.
For the CIFAR experiment, to make sure the ﬁrst task solution is within a good basin, we train the only ﬁrst task for ﬁve epochs in this experiment and continue the training for one epoch for the next tasks.
While our results might seem initially lower than the results reported in other continual learning works, we emphasize that this is due to the limited episodic memory we have used in our setup.
There are two reasons behind this choice of memory size and number of epochs.
First, to make it closer to common setup in other related works such as A-GEM and ER-Reservoir that are state of the art in episodic memory methods.
Second, the limited episodic memory and number of epochs allow measuring “how effective each continual learning method uses the previously seen data”.
In this experiment, we use a two-layer MLP with 256 ReLU units for
MNIST datasets and ResNet18 with three times less feature for CIFAR dataset.
We note that to have a similar setup with our baselines, the task identiﬁers are used to select the correct output head only in CIFAR-100 experiments.
C. H YPER -P ARAMETERS
For the experiment in Section 5.1, we have used the following grid for each model.
We note that for other algorithms (e.g., A-GEM, and EWC), we ensured that our grid contains the optimal values that the original papers reported.
[Table]
• learning rate: [0.25, 0.1, 0.01 (MNIST, CIFAR-100), 0.001] • batch size:EWC • learning rate: [0.25, 0.1 (MNIST, CIFAR-100), 0.01, 0.001] • batch size: [ 10 , 64] • λ (regularization): [1, 10 (MNIST, CIFAR-100), 100]
[Table]
GEM • learning rate: [0.1, 0.1 (MNIST), 0.01 (CIFAR-100), 0.001] • batch size: [ 10 , 64]
[Table]
• learning rate: [0.25, 0.1 (MNIST), 0.01 (CIFAR-100), 0.001] • batch size: [ 10 , 64]
• initial learning rate: [0.25, 0.1 (MNIST, CIFAR-100), 0.01, 0.001] • learning rate decay: [0.9, 0.85, 0.8 (CIFAR-100), 0.6 (MNIST)] • batch size: [ 10 , 64] • dropout: [ 0.25 (MNIST), 0.1 (CIFAR-100)]
M ODE C ONNECTIVITY SGD
To obtain continual minima (i.e., ˆ w 1 to ˆ w 20 ), we use the same hyper-parameters as the stable SGD.
To minimize the objective function in Eq.
(3) and obtain ¯ w 1 to ¯ w 20 , we use the following grid:
• number of samples: [3, 5 , 10 , 20 ].
We found that empirically more than ﬁve samples on each line would give sufﬁcient information to minimize Eq.
(3) for both MNIST and CIFAR experiments.
• learning rate: [0.001, 0.01 (CIFAR-100), 0.05 (MNIST), 0.1, 0.2 (MNIST, CIFAR-100)].
C. H OW CLOSE IS OUR EXPERIMENTAL SETUP TO THE OTHER WORKS IN THE LITERATURE ?
We end this section by showing that our setup is close to the common setup in the continual learning literature.
More speciﬁcally, we discuss the following aspects of our setup:
Benchmarks : While we believe continual learning literature can beneﬁt from a more diverse set of tasks such as reinforcement learning tasks, the majority of the focus in the ﬁeld is devoted to computer vision tasks.
One possible explanation is that despite having more control over the continual learning experience, and despite the fact that it is possible to train high-performance models in single task settings, still, models perform relatively poorly in continual learning setting.
Hence, we followed the common benchmarks in the literature such as rotated and permuted MNIST, and split CIFAR-100.
While we believe the community needs to agree on a more diverse set of benchmarks, the performance gap between continual learning minima and multitask minima is yet to be ﬁlled even on these simpler benchmarks.
Architectures : The choice of two-layer MLP with 256 hidden neurons for MNIST benchmarks and
ResNet18 for CIFAR-100 benchmark is common in the literature (Chaudhry et al., 2018a; Mirzadeh et al., 2020; Chaudhry et al., 2019).
We have used same architectures to make it easier to compare our results with other works as well.
C. I MPLEMENTATION D ETAILS
In this section we discuss the visualization and implementation details.
C.5. H OW ARE SURFACE PLANES GENERATED ?
To plot a surface plane (e.g., Fig. 3), we need three points to obtain two basis vectors.
Suppose we have a set of three parameters w 1 , w 2 , and w 3 where each parameter is in vector in high-dimensional space.
These vectors can be obtained by computing the ﬂattened version of each layer in the neural network and then concatenating them.
For the three mentioned points, we perform the following procedure:
Calculate two basis vectors: (cid:126)u = w 2 − w 1 , and (cid:126)v = w 3 − w 1 . 2.
Orthogonalize the basis vectors by calculating v = v − cos( u, v ) u 3.
Deﬁne a Cartesian coordinate system in the (x,y) plane that maps each coordinate to a parameter space by calculating p ( x, y ) =
w 1 + u · x + v · y 4.
Finally, for a deﬁned grid on this coordinate system, we calculate the empirical loss of each ( x, y ) coordinate using function p ( x, y ) .
Finally, we note that to generate plots that show loss along the interpolation between minima (e.g.,
Fig. 4), we simply calculate the interpolation solution between two minima and calculate the loss for that solution.
For example, the minima on the interpolation line that connects ˆ w 1 to ˆ w 2 can be represented with w α = ˆ w 1 + α
( ˆ w 2 − ˆ w 1 ) where 0 ≤ α ≤ 1 .
We then calculate the empirical loss for w α .
C. R EPRODUCING MC-SGD
We provide an anonymous executable version of MC-SGD on the CodaLab platform.
CodaLab keeps track of the full provenance generating a particular result, and thus, executable papers on this platform allow others to verify the results independently.
To this end, we provide ﬁve anonymous runs of MC-SGD on rotation MNIST with 20 tasks to verify the results in Table 1.
Each run uses a different random seed and can be fully replicated as it has all the required dependencies and the source code.
Here are links to run 1, run 2, run 3, run 4, and run 5.
Each of the above links has a “stdout” section that logs the evolution of the average accuracy of
MC-SGD.
In addition, each run generates a “checkpoint” folder that includes weight ﬁles to MC-SGD weight ﬁles at the end of each task. C.6. MC-SGD I MPLEMENTATION
To implement MC-SGD, we use PyTorch (Paszke et al., 2019) because of its dynamic graph capability.
The challenging part of the implementation is calculating Eq.
(5), where we need to compute a summation of gradients on two lines. To this end, we ﬁrst calculate the accumulated gradients over the mentioned line: def get_grads_on_interpolation_line(w_hat, w_bar, alphas, memory): """ Get accumulated gradients on interpolation line between w_hat and w_bar """ accumulated_gradients_on_line = 0. # for each point on the interpolation line for alpha in alphas: interpolation_grads = [] # (1) calculate the parameters on the interpolation line interpolation_w =
w_hat +
(w_bar - w_hat)
* alpha model = make_model_from_param_vector(interpolation_w) loss = calculate_loss(model, memory)
# (2) calculate the loss to obtain gradients loss.backward() # Now the gradients are stored in the graph # (3) collect the gradients for param in model.parameters(): # get gradients for each module/block/layer interpolation_grads.append(param.grad.view(-1))
interpolation_grads = torch.cat(interpolation_grads) # (4) accumulate gradients accumulated_gradients_on_line +=
interpolation_grads return accumulated_gradients_on_line
D A DDITIONAL R ESULTS D. C ONVENTIONAL DISTANCE MEASURES
In this section, we provide detailed (cid:96) 2 distance and pair-wise CKA scores of the models introduced in Sec. 2.
As explained in Sec. 2, (cid:96) 2 distance can be a misleading metric when it comes to comparing the multitask solution to the continual learning one.
I.e., the continual solution on task 5 seems to be closer to the solution of task 1 than the multitask solution, even though the continual solution suffers from catastrophic forgetting.
This could be indicative to the fact that compared to the pruning scenario, curvature might play a bigger role in continual learning and can not be ignored (e.g., the delta between parameters might align differently to the eigenvectors of the Hessian and of higher order derivatives).
However, the bound deﬁned by Eq.
(1) is not accurate enough if the vector ˆ w 2 − ˆ w 1 does not lie in the direction of the largest eigenvector.
For instance, it might be the case (as we show later) that the optimization trajectory moves in a direction with low curvature (e.g., a low-loss valley), and thus the loss does not increase.
The ineffectiveness of Euclidean distance is also veriﬁed empirically by middle column of Fig.
Even though the MTL minima for tasks are further from CL minima they have much higher accuracy.
For the reference and further evidence, the full table of pairwise distances between all minima is shown in Fig.
D. A LL MINIMA ARE LINEARLY CONNECTED
[Table]
[Figure]
[Figure] [Figure]
[Figure]
In this section, we ﬁrst show the extended learning setup for 3 tasks in Fig. 15.
Eseentially, this is the extended version Fig.1 where an additional task is added.
[-CKA( ˆ w 1 , w ∗ 2 ) block1block2block3block
-]
[-CKA( ˆ w 1 , ˆ w 5 ) block1block2block3block
-]
Moreover, we provide a more detailed result for the interpolation plots in Fig.
4 where we show a zoomed version of those plots.
For easier comparison, we repeat the plots from Fig.
4 here in Fig.
16b, 16d, 17b, and 17d with the same order.
Fig .16a shows two paths (i.e., ˆ w 1 to ˆ w 2 and ˆ w 1 to w ∗ 2 ) from Fig. 16b.
Although the path from ˆ w 1 to ˆ w 2 (in yellow) seems nearly non-increasing Fig.
16b, in Fig .16a
we show that this path is in fact an increasing path while the loss on the interpolation path between ˆ w 1 and ˆ w 2 is non increasing.
Similarly, this phenomenon is shown in Fig.
16c for ˆ w 2 to ˆ w 3 and w ∗ 3 on rotated MNIST dataset and in Fig.
17 for split CIFAR-100 dataset.
[Figure] [Figure]
S TABLE SGD M ETHOD WITH M EMORY
One interesting question to pursue would be how would the stable SGD method perform if given the same size of episodic memory.
To this end, we add an episodic memory to the stable-SGD method with similar size of MC-SGD (i.e., one example per class per task).
We show the performance of the stable SGD method in Table 2 on Rotated MNIST and Permuted
MNIST with 20 tasks, with a similar setup in Sec. 5.1.
We can see in Table 2 the additional episodic memory improves the average accuracy by 2.1% in Permuted MNIST and 2.5% in Rotated MNIST.
[Table]
[Figure]
Moreover, in Fig.
18, we compare the loss on the interpolation paths between SGD, Stable-SGD with memory
[Figure]
and MC-SGD methods.
To show Stable SGD minima, we use (cid:101) w notation.
We can see that the additional episodic memory helped the stable-SGD method to gain a signiﬁcant boost.
Still, the regularization term in the MC-SGD method helps it ﬁnd minima with better mode connectivity.
[Figure]
D. M ODE C ONNECTIVITY OF EWC M INIMA
One interesting question is that if regularization methods such as EWC can implicitly help obtain minima that are linearly connected.
Similar to the Fig.
7 and Fig.
[Figure]
we show the interpolation paths for EWC minima In Fig.
We can see that although the EWC minima perform better than SGD, they are not linearly connected to the ﬁrst task’s minimizer.
[Figure]
D. P ERMUTED MNIST WITH 50 TASKS
In this experiment, the MLP hidden layers increased from 256 to 512 for this experiment since the number of tasks increased.
Moreover, the size of episodic memory is increased from 200 to 500 (one example per class per task).
[Figure]