E FFICIENT C ONTINUAL L EARNING WITH M ODULAR N ETWORKS AND T ASK -D RIVEN P RIORS
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing literature in Continual Learning (CL) has focused on overcoming catastrophic forgetting, the inability of the learner to recall how to perform tasks observed in the past.
There are however other desirable properties of a CL system, such as the ability to transfer knowledge from previous tasks and to scale memory and compute sub-linearly with the number of tasks.
Since most current benchmarks focus only on forgetting using short streams of tasks, we ﬁrst propose a new suite of benchmarks to probe CL algorithms across these new axes.
Finally, we introduce a new modular architecture, whose modules represent atomic skills that can be composed to perform a certain task.
Learning a task reduces to ﬁguring out which past modules to re-use, and which new modules to instantiate to solve the current task.
Our learning algorithm leverages a task-driven prior over the exponential search space of all possible ways to combine modules, enabling efﬁcient learning on long streams of tasks.
Our experiments show that this modular architecture and learning algorithm perform competitively on widely used CL benchmarks while yielding superior performance on the more challenging benchmarks we introduce in this work.
I NTRODUCTION
Continual Learning (CL) is a learning framework whereby an agent learns through a sequence of tasks (Ring, 1994; Thrun, 1994; 1998), observing each task once and only once.
Much of the focus of the CL literature has been on avoiding catastrophic forgetting (McClelland et al., 1995; McCloskey & Cohen, 1989; Goodfellow et al., 2013), the inability of the learner to recall how to perform a task learned in the past.
In our view, remembering how to perform a previous task is particularly important because it promotes knowledge accrual and transfer.
CL has then the potential to address one of the major limitations of modern machine learning: its reliance on large amounts of labeled data.
An agent may learn well a new task even when provided with little labeled data if it can leverage the knowledge accrued while learning previous tasks.
Our ﬁrst contribution is then to pinpoint general properties that a good CL learner should have, besides avoiding forgetting.
In § 3 we explain how the learner should be able to transfer knowledge from related tasks seen in the past.
At the same time, the learner should be able to scale sub-linearly with the number of tasks, both in terms of memory and compute, when these are related.
Our second contribution is to introduce a new benchmark suite, dubbed CTrL, to test the above properties, since current benchmarks only focus on forgetting.
For the sake of simplicity and as a ﬁrst step towards a more holistic evaluation of CL models, in this work we restrict our attention to supervised learning tasks and basic transfer learning properties.
Our experiments show that while commonly used benchmarks do not discriminate well between different approaches, our newly introduced benchmark let us dissect performance across several new dimensions of transfer and scalability (see Fig.
1 for instance), helping machine learning developers better understand the strengths and weaknesses of various approaches.
Our last contribution is a new model that is designed according to the above mentioned properties of CL methods.
It is based on a modular neural network architecture (Eigen et al., 2014; Denoyer & Gallinari, 2015; Fernando et al., 2017; Li et al., 2019) with a novel task-driven prior ( § 4).
Every task is solved by the composition of a handful of neural modules which can be either borrowed from past tasks or freshly trained on the new task.
In principle, modularization takes care of all the
[Figure]
fundamental properties we care about, as i) by design there is no forgetting as modules from past tasks are not updated when new tasks arrive, ii) transfer is enabled via sharing modules across tasks, and iii) the overall model scales sublinearly with the number of tasks as long as similar tasks share modules.
The key issue is how to efﬁciently select modules, as the search space grows exponentially in their number.
In this work, we overcome this problem by leveraging a data driven prior over the space of possible architectures, which allows only local perturbations around the architecture of the previous task whose features best solve the current task ( § 4.2).
Our experiments in § 5, which employ a stricter and more realistic evaluation protocol whereby streams are observed only once but data from each task can be played multiple times, show that this model performs at least as well as state-of-the-art methods on standard benchmarks, and much better on our new and more challenging benchmark, exhibiting better transfer and ability to scale to streams with a hundred tasks.
R ELATED W ORK
CL methods can be categorized into three main families of approaches.
Regularization based methods use a single shared predictor across all tasks with the only exception that there can be a taskspeciﬁc classiﬁcation head depending on the setting.
They rely on various regularization methods to prevent forgetting.
Kirkpatrick et al.
; Schwarz et al.
(2018) use an approximation of the
Fisher Information matrix while Zenke et al.
(2017) using the distance of each weight to its initialization as a measure of importance.
These approaches work well in streams containing a limited number of tasks but will inevitably either forget or stop learning as streams grow in size and diversity (van de Ven & Tolias, 2019), due to their structural rigidity and ﬁxed capacity.
Similarly, rehearsal based methods also share a single predictor across all tasks but attack forgetting by using rehearsal on samples from past tasks.
For instance, Lopez-Paz & Ranzato (2017); Chaudhry et al. (2019b); Rolnick et al.
(2019) store past samples in a replay buffer, while Shin et al.
(2017) learn to generate new samples from the data distribution of previous tasks and Zhang et al.
(2019) computes per-class prototypes.
These methods share the same drawback of regularization methods:
Their capacity is ﬁxed and pre-determined which makes them ineffective at handling long streams.
Finally, approaches based on evolving architectures directly tackle the issue of the limited capacity by enabling the architecture to grow over time.
Rusu et al.
(2016) introduce a new network on each task, with connection to all previous layers, resulting in a network that grows super-linearly with the number of tasks.
This issue was later addressed by Schwarz et al.
(2018) who propose to distill the new network back to the original one after each task, henceforth yielding a ﬁxed capacity predictor which is going to have severe limitations on long streams.
Yoon et al.
Hung et al.
(2019) propose a heuristic algorithm to automatically add and prune weights.
Li et al.
(2019) propose to softly select between reusing, adapting, and introducing a new module at every layer.
Similarly, Xu & Zhu (2018) propose to add ﬁlters once a new task arrives using REINFORCE (Williams, 1992), leading to larger and larger networks even at inference time as time goes by.
These two works are the most similar to ours, with the major difference that we restrict the search space over architectures, enabling much better scaling to longer streams.
While their search space (and RAM consumption) grows over time, ours is constant.
Our approach is modular , and only a small (and constant) number of modules is employed for any given task both at training and test time.
Non-modular approaches, like those relying on individual neuron gating (Adel et al., 2020; Serr`a et al., 2018; Kessler et al., 2019), lack such runtime efﬁciency which limits their applicability to long streams.
Rajasegaran et al.
; Fernando et al.
(2017) propose to learn a modular architecture, where each task is identiﬁed by a path in a graph of modules like we do.
However, they lack the prior over the search space.
They both use random search which is rather inefﬁcient as the number of modules grows.
There are of course other works that introduce new benchmarks for CL.
Most recently, Wortsman et al.
(2020) have proposed a stream with 2500 tasks all derived from MNIST permutations.
Unfortunately, this may provide little insight in terms of how well models transfer knowledge across tasks.
Other benchmarks like CORe50 (Lomonaco & Maltoni, 2017) and CUB-200 (Wah et al., 2011) are more realistic but do not enable precise assessment of how well models transfer and scale.
CL is also related to other learning paradigms, such as meta-learning (Finn et al., 2017;
Nichol et al., 2018; Duan et al., 2016), but these only consider the problem of quickly adapting to a new task while in CL we are also interested in preventing forgetting and learning better over time.
For instance, Alet et al.
(2018) proposed a modular approach for robotic applications.
However, only the performance on the last task was measured.
There is also a body of literature on modular networks for multi-task and multi-domain learning (Ruder et al., 2019; Rebufﬁ et al., 2017; Zhao et al., 2020).
The major differences are the static nature of the learning problem they consider and the lack of emphasis on scaling to a large number of tasks.
E VALUATING C ONTINUAL L EARNING M ODELS
Let us start with a general formalization of the CL framework.
We assume that tasks arrive in sequence, and that each task is associated with an integer task descriptor t = 1 , 2 , ... which corresponds to the order of the tasks.
Task descriptors are provided to the learner both during training and test time.
Each task is deﬁned by a labeled dataset D t .
We denote with S a sequence of such tasks.
A predictor for a given task t is denoted by f : X t × Z
→ Y t .
The predictor has internally some trainable parameters whose values depend on the stream of tasks S seen in the past, therefore the prediction is: f ( x, t |S ) .
Notice that in general, different streams lead to different predictors for the same task: f ( x, t |S ) (cid:54) =
f ( x, t |S (cid:48) ) .
Desirable Properties of CL models And Metrics:
Since we are focusing on supervised learning tasks, it is natural to evaluate models in terms of accuracy.
We denote the prediction accuracy of the predictor f as ∆( f ( x, t |S ) , y ) , where x is the input, t is the task descriptor of x , S is the stream of tasks seen by the learner and y is the ground truth label.
In this work, we consider four major properties of a CL algorithm.
First, the algorithm has to yield predictors that are accurate by the end of the learning experience.
This is measured by their average accuracy at the end of the learning experience:
[Equation]
Second, the CL algorithm should yield predictors that do not forget, i.e. that are able to perform a task seen in the past without signiﬁcant loss of accuracy.
Forgetting is deﬁned as:
[Equation]
This measure of forgetting has been called backward transfer (Lopez-Paz & Ranzato, 2017), and it measures the average loss of accuracy on a task at the end of training compared to when the task was just learned.
Negative values indicate the model has been forgetting.
Positive values indicate that the model has been improving on past tasks by learning subsequent tasks.
Third, the CL algorithm should yield predictors that are capable of transferring knowledge from past tasks when solving a new task.
Transfer can be measured by:
[Equation]
which measures the difference of performance between a model that has learned through a whole sequence of tasks and a model that has learned the last task in isolation.
We would expect this quantity to be positive if there exist previous tasks that are related to the last task.
Negative values imply the model has suffered some form of interference or even lack of plasticity when the predictor has too little capacity left to learn the new task.
Finally, the CL algorithm has to yield predictors that scale sublinearly with the number of tasks both in terms of memory and compute.
In order to quantify this, we simply report the total memory usage and compute by the end of the learning experience during training.
We therefore include in the memory consumption everything a learner has to keep around to be able to continually learn (e.g., regularization parameters of EWC or the replay buffer for experience replay).
Streams The metrics introduced above can be applied to any stream of tasks.
While current benchmarks are constructed to assess forgetting, they fall short at enabling a comprehensive evaluation of transfer and scalability because they do not control for task relatedness and they are composed of too few tasks.
Therefore, we propose a new suite of streams.
If t is a task in the stream, we denote with t − and t + a task whose data is sampled from the same distribution as t , but with a much smaller or larger labeled dataset, respectively.
Finally, t (cid:48) and t (cid:48)(cid:48) are tasks that are similar to task t , while we assume no relation between t i and t j , for all i (cid:54)
=
j .
We consider ﬁve axes of transfer and deﬁne a stream for each of them.
While other dimensions certainly exist, here we are focusing on basic properties that any desirable model should possess.
Direct Transfer : we deﬁne the stream S - =
( t +1 , t 2 , t 3 , t 4 , t 5 , t − 1 ) where the last task is the same as the ﬁrst one
but with much less data to learn from.
This is useful to assess whether the learner can directly transfer knowledge from the relevant task.
Knowledge Update : we deﬁne the stream S + =
( t − 1 , t 2 , t 3 , t 4 , t 5 , t +1 ) where the last task has much more data than the ﬁrst task with intermediate tasks that are unrelated.
In this case, there should not be much need to transfer anything from previous tasks, and the system can just use the last task to update its knowledge of the ﬁrst task.
Transfer to similar Input/Output Distributions : we deﬁne two streams where the last task is similar to the ﬁrst task but the input distribution changes S in =
( t 1 , t 2 , t 3 , t 4 , t 5 , t (cid:48) 1 ) , or the output distribution changes S out
=
( t 1 , t 2 , t 3 , t 4 , t 5 , t (cid:48)(cid:48) 1 ) .
Plasticity: this is a stream where all tasks are unrelated, S pl =
( t 1 , t 2 , t 3 , t 4 , t 5 ) , which is useful to measure the ”ability to still learn” and potential interference (erroneous transfer from unrelated tasks) when learning the last task.
All these tasks are evaluated using T in eq.
Other dimensions of transfer (e.g., transfer with compositional task descriptors or under noisy conditions) are avenues of future work.
Finally, we evaluate scalability using S long , a stream with 100 tasks of various degrees of relatedness and with varying amounts of training data.
See § 5.1 for more details.
M ODULAR N ETWORKS WITH T ASK D RIVEN P RIOR (MNTDP)
In this section we describe an approach, dubbed MNTDP, designed according to the properties discussed in § 3.
The class of predictor functions f ( x, t |S ) we consider in this work is modular , in the sense that predictors are composed of modules that are potentially shared across different (but related) tasks.
A module can be any parametric function, for instance, a ResNet block (He et al., 2015).
The only restriction we use in this work is that all modules in a layer should differ only in terms of their actual parameter values, while modules across layers can implement different classes of functions.
For instance, in the toy illustration of Fig.
2-A (see caption for details), there are two predictors, each composed of three modules (all ResNet blocks), the ﬁrst one being shared.
4. T RAINING
Once the new task t arrives, the algorithm follows three steps.
First, it temporarily adds new randomly initialized modules at every layer (these are denoted by dashed boxes in Fig.
2-B)
and it then deﬁnes a search space over all possible ways to combine modules.
Second, it minimizes a loss function, which in our case is the cross-entropy loss, over both ways to combine modules and module parameters, see Fig.
2-C. Note that only the parameters of the newly added modules are subject to training, de facto preventing forgetting of previous tasks by construction but also preventing positive backward transfer.
Finally, it takes the resulting predictor for the current task and it adds the
A
B
C
D
MNTDP-S
MNTDP-
[Figure]
parameters of the newly added modules (if any) back to the existing library of module parameters, see Fig.
2-D.
Since predictors are uniquely identiﬁed by which modules compose them, they can also be described by the path in the grid of module parameters.
We denote the j -th path in the graph by π j .
The parameters of the modules in path π j are denoted by θ ( π j ) .
Note that in general θ ( π j ) ∩ θ ( π i ) (cid:54) = ∅ , for i (cid:54) =
j since some modules may be shared across two different paths.
Let Π be the set of all possible paths in the graph.
This has a size equal to the product of the number of modules at every layer, after adding the new randomly initialized modules.
If Γ is a distribution over Π which is subject to learning (and initialized uniformly), then the loss function is:
[Equation]
where f ( x, t |S , θ ( π j ) is the predictor using parameters θ ( π j ) , L is the loss, and the minimization over the parameters is limited to only the newly introduced modules.
The resulting distribution Γ ∗ is a delta distribution, assuming no ties between paths.
Once the best path has been found and its parameters have been learned, the corresponding parameters of the new modules in the optimal path are added to the existing set of modules while the other ones are disregarded (Fig.
2-D).
In this work, we consider two instances of the learning problem in eq. 4, which differ in a) how they optimize over paths and b) how they share parameters across modules.
Stochastic version: This algorithm alternates between one step of gradient descent over the paths via REINFORCE (Williams, 1992) as in (Veniat & Denoyer, 2018), and one step of gradient descent over the parameters for a given path.
The distribution Γ is modeled by a product of multinomial distributions, one for each layer of the model.
These select one module at each layer, ultimately determining a particular path.
Newly added modules may be shared across several paths which yields a model that can support several predictors while retaining a very parsimonious memory footprint thanks to parameter sharing.
This version of the model, dubbed MNTDP-S, is outlined in the left part of Fig.
2-C and in Alg.
1 in the Appendix.
In order to encourage the model to explore multiple paths, we use an entropy regularization on Γ during training.
Deterministic version: This algorithm minimizes the objective over paths in eq.
4 via exhaustive search, see Alg. 2 in Appendix and the right part of Fig.
2-C. Here, paths do not share any newly added module and we train one independent network per path, and then select the path yielding the lowest loss on the validation set.
While this requires much more memory, it may also lead to better overall performance because each new module is cloned and trained just for a single path.
Moreover, training predictors on each path can be trivially and cheaply parallelized on modern GPU devices.
4. D ATA -D RIVEN P RIOR
Unfortunately, the algorithms as described above do not scale to a large number of tasks (and henceforth modules) because the search space grows exponentially.
This is also the case for other evolving architecture approaches proposed in the literature (Li et al., 2019; Rajasegaran et al., 2019; Yoon et al., 2018; Xu & Zhu, 2018) as discussed in § 2.
If there were N modules per layer and L layers, the search space would have size N L .
In order to restrict the search space, we only allow paths that branch to the right: A newly added module at layer l can only connect to another newly added module at layer l + 1 , but it cannot connect to an already trained module at layer l +1 .
The underlying assumption is that for most tasks we expect changes in the output distribution as opposed to the input distribution, and therefore if tasks are related, the base trunk is a good candidate for being shared.
We will see in § 5.3 what happens when this assumption is not satisﬁed, e.g., when applying this to S in .
To further restrict the search space we employ a data-driven prior .
The intuition is to limit the search space to perturbations of the path corresponding to the past task (or to the top- k paths) that is the most similar to the current task.
There are several methods to assess which task is the closest to the current task without accessing data from past tasks and also different ways to perturb a path.
We propose a very simple approach, but others could have been used.
We take the predictors from all the past tasks and select the path that yields the best nearest neighbor classiﬁcation accuracy when feeding data from the current task using the features just before the classiﬁcation head.
This process is shown in Fig.
2-B. The search space is reduced from T L to only L , and Γ of eq.
4 is allowed to have non-zero support only in this restricted search space, yielding a much lower computational and memory footprint which is constant with respect to the number of tasks.
The designer of the model has now direct control (by varying k , for instance) over the trade-off between accuracy and computational
/memory budget.
By construction, the model does not forget, because we do not update modules of previous tasks.
The model can transfer well because it can re-use modules from related tasks encountered in the past while not being constrained in terms of its capacity.
And ﬁnally, the model scales sub-linearly in the number of tasks because modules can be shared across similar tasks.
We will validate empirically in § 5.3 whether the choice of the restricted search space works well in practice.
E XPERIMENTS
In this section we ﬁrst introduce our benchmark in § 5.1 and the modeling details § 5.2, and then report results both on standard benchmarks as well as ours in § 5.3.
5. T HE CT R L B ENCHMARK
The CTrL (Continual Transfer Learning) benchmark is a collection of streams of tasks built over seven popular computer vision datasets, namely: CIFAR10 and CIFAR100 (Krizhevsky, 2009),
DTD (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), MNIST (LeCun et al., 1998), RainbowMNIST (Finn et al., 2019) and Fashion MNIST (Xiao et al., 2017); see Table 3 in Appendix for basic statistics.
These datasets are desirable because they are diverse (hence tasks derived from some of these datasets can be considered unrelated), they have a fairly large number of training examples to simulate tasks that do not need to transfer, and they have low spatial resolution enabling fast evaluation.
CTrL is designed according to the methodology described in § 3, to enable evaluation of various transfer learning properties and the ability of models to scale to a large number of tasks.
Each task consists of a training, validation, and test datasets corresponding to a 5-way and 10-way classiﬁcation problem for the transfer streams and the long stream, respectively.
The last task of S out consists of a shufﬂing of the output labels of the ﬁrst task.
The last task of S in is the same as its ﬁrst task except that MNIST images have a different background color.
S long has 100 tasks, and it is constructed by ﬁrst sampling a dataset, then 5 classes at random, and ﬁnally the amount of training data from a distribution that favors small tasks by the end of the learning experience.
See Tab. 5 and 6 in Appendix for details.
Therefore, S long tests not only the ability of a model to scale to a relatively large number of tasks but also to transfer more efﬁciently with age.
All images have been rescaled to 32x32 pixels in RGB color format, and per-channel normalized using statistics computed on the training set of each task.
During training, we perform data augmentation by using random crops (4 pixels padding and 32x32 crops) and random horizontal reﬂection.
Please, refer to Appendix A for further details.
5. M ETHODOLOGY AND M ODELING D ETAILS
Models learn over each task in sequence; data from each task can be replayed several times but each stream is observed only once .
Since each task has a validation dataset, hyper-parameters (e.g., learning rate and number of weight updates) are task-speciﬁc and they are cross-validated on the validation set of each task.
Once the learning experience ends, we test the resulting predictor on the test sets of all the tasks.
Notice that this is a stricter paradigm than what is usually employed in the literature (Kirkpatrick et al., 2016), where hyper-parameters are set at the stream level (by replaying the stream several times).
Our model selection criterion is more realistic because it does not assume that the learner has access to future tasks when cross-validating on the current task, and this is more consistent with the CL’s assumptions of operating on a stream of data.
All models use the same backbone.
Unless otherwise speciﬁed, this is a small variant of the ResNet18 architecture which is divided into 7 modules; please, refer to Table 8 for details of how many layers and parameters each block contains.
Each predictor is trained by minimizing the cross-entropy loss with a small L2 weight decay on the parameters.
In our experiments, MNTDP adds 7 new randomly initialized modules, one for every block.
The search space does not allow connecting old blocks from new blocks, and it considers two scenarios: using old blocks from the past task that is deemed most similar ( k = 1 , the default setting) or considering the whole set of old blocks ( k = all) resulting in a much larger search space.
We compare to several baselines: Independent Models which instantiates a randomly initialized predictor for every task (as many paths as tasks without any module overlap), 2)
Finetuning which trains a single model to solve all the tasks without any regularization and initializes from the model of the previous task (a single path shared across all tasks), 3) New-Head which also shares the trunk of the network across all tasks but not the classiﬁcation head which is task-speciﬁc, 4) NewLeg which shares all layers across tasks except for the very ﬁrst input layer which is task-speciﬁc, 5) EWC (Kirkpatrick et al., 2016) which is like “ﬁnetuning” but with a regularizer to alleviate forgetting, 6) Experience Replay (Chaudhry et al., 2019b) which is like ﬁnetuning except that the model has access to some samples from the past tasks to rehearse and alleviate forgetting (we use 15 samples per class to obtain a memory consumption similar to other baselines), 7) Progressive
Neural Networks (PNNs) (Rusu et al., 2016) which adds both a new module at every layer as well as lateral connections once a new task arrives.
8) HAT (Serr ` a et al., 2018): which learns an attention mask over the parameters of the backbone network for each task.
Since HAT’s opensource implementation uses AlexNet (Krizhevsky et al., 2012) as a backbone, we also implemented a version of MNTDP using AlexNet for a fair comparison.
Moreover, we considered two versions of HAT, the default as provided by the authors and a version, dubbed HAT-wide, that is as wide as our ﬁnal MNTDP model (or as wide as we can ﬁt into GPU memory).
9) DEN (Yoon et al., 2018) and 10) RCL (Xu & Zhu, 2018) which both evolve architectures.
For these two models since there is no publicly available implementation with CNNs, we only report their performance on PermutedMNIIST using fully connected nets (see Fig.
3a and Tab. 11 in Appendix E.2).
We report performance across different axes as discussed in § 3: Average accuracy as in eq. 1, forgetting as in eq.
2, transfer as in eq.
3 and applicable only to the transfer streams, “Mem.”
[MB] which refers to the average memory consumed by the end of training, and “Flops” [T] which corresponds to the average amount of computation used by the end of training.
5. R ESULTS
Existing benchmarks:
In Fig.
3 we compare MNTDP against several baselines on two standard streams with 10 tasks, Permuted MNIST, and Split CIFAR100.
We observe that all models do fairly well, with EWC falling a bit behind the others in terms of average accuracy.
PNNs has good average accuracy but requires more memory and compute.
Compared to MNTDP, both RCL and HAT have lower average accuracy and require more compute.
MNTDP-D yields the best average accuracy, but requires more computation than “independent models”; notice however that its wall-clock training
[Figure] [Figure]
Finetune 0.
-0.1 New-head 0.
-0.4 New-leg 0.
-0.4 Online EWC † 0.
-0.4 ER 0.
-0.2 PNN 0.
-0.1 MNTDP
[Table]
time is actually the same as “independent models” since all candidate paths (seven in our case) can be trained in parallel on modern GPU devices.
In fact, it turns out that on these standard streams
MNTDP trivially reduces to “independent models” without any module sharing, since each task is fairly distinct and has a relatively large amount of data.
It is therefore not possible to assess how well models can transfer knowledge across tasks, nor it is possible to assess how well models scale.
Fortunately, we can leverage the CTrL benchmark to better study these properties, as described next.
CTrL:
We ﬁrst evaluate models in terms of their ability to transfer by evaluating them on the streams S - , S + , S in , S out and S pl introduced in § 3.
Tab. 1 shows that “independent models” is again a strong baseline, because even on the ﬁrst four streams, all tasks except the last one are unrelated and therefore instantiating an independent model is optimal.
However, MNTDP yields the best average accuracy overall.
MNTDP-D achieves the best transfer on streams S - , S + , S out and S pl , and indeed it discovers the correct path in each of these cases (e.g., it discovers to reuse the path of the ﬁrst task when learning on S - and to just swap the top modules when learning the last task on S + ).
Examples of discovered path are presented in appendix E.4.
MNTDP underperforms on S in because its prior does not match the data distribution, since in this case, it is the input distribution that has changed but swapping the ﬁrst module is out of MNTDP search space.
This highlights the importance of the choice of prior for this algorithm.
In general, MNTDP offers a clear trade-off between accuracy, i.e. how broad the prior is which determines how many paths can be evaluated, and memory/compute budget.
Computationally MNTDP-D is the most demanding, but in practice its wall clock time is comparable to “independent” because GPU devices can store in memory all the paths (in our case, seven) and efﬁciently train them all in parallel.
We observe also that MNTDP-S has a clear advantage in terms of compute at the expense of a lower overall average accuracy, as sharing modules across paths during training can lead to sub-optimal convergence.
Overall, MNTDP has a much higher average accuracy than methods with a ﬁxed capacity.
It also beats PNNs, which seems to struggle with interference when learning on S pl , as all new modules connect to all old modules which are irrelevant for the last task.
Moreover, PNNs uses much more memory.
“New-leg” and “new-head” models perform very well only when the tasks in the stream match their assumptions, showing the advantage of the adaptive prior of MNTDP.
Finally, EWC shows great transfer on S - , S in , S out , which probe the ability of the model to retain information.
<
A > < F > Mem.
PFLOPs
Independent 0.
Finetune 0.
New-head 0.
On.
EWC † 0.
MNTDP-S 0.
MNTDP-D 0.
[Table]
HAT
[Table] [Table]
≈
[Table]
[Figure]
However, it fails at S + , S pl that require additional capacity allocation to learn a new task.
Fig.gives a holistic view by reporting the normalized performance across all these dimensions.
We conclude by reporting results on S long composed of 100 tasks.
Tab.
2 reports the results of all the approaches we could train without running into out-of-memory.
MNTDP-D yields an absolute 18% improvement over the baseline independent model while using less than half of its memory, thanks to the discovery of many paths with shared modules.
Its actual runtime is close to independent model because of GPU parallel computing.
To match the capacity of MNTDP, we scale HAT’s backbone to the maximal size that can ﬁt in a Titan X GPU Memory (6.5x, wide version).
The wider architecture greatly increases inference time in later tasks (see also discussion on memory complexity at test time in Appendix D), while our modular approach uses the same backbone for every task and yet it achieves better performance.
Fig.
4 shows the average accuracy up to the current task over time.
MNTDP-D attains the best performance while growing sublinearly in terms of memory usage.
Methods that do not evolve their architecture, like EWC, greatly suffer in terms of average accuracy.
Ablation:
We ﬁrst study the importance of the prior.
Instead of selecting the nearest neighbor path, we pick one path corresponding to one of the previous tasks at random.
In this case, T ( S - ) decreases from 0 to − 0 .
2 and T ( S out ) goes from 0 to − 0 .
With a random path, MNTDP learns not to share any module, demonstrating that it is indeed important to form a good prior over the search space.
Appendix E reports additional results demonstrating how on small streams MNTDP is robust to the choice of k in the prior since we attain similar performance using k = 1 and k
= all , although only k = 1 let us scale to S long .
Finally, we explore the robustness to the number of modules by splitting each module in two, yielding a total of 10 modules per path, and by merging adjacent modules yielding a total of 3 modules for the same overall number of parameters.
We ﬁnd that
T ( S out ) decreases from 0 to -0.1, with a 9% decrease on t − 1 , when the number of modules decreases but stays the same when the number of modules increases, suggesting that the algorithm has to have a sufﬁcient number of modules to ﬂexibly grow.
C ONCLUSIONS
We introduced a new benchmark to enable a more comprehensive evaluation of CL algorithms, not only in terms of average accuracy and forgetting but also knowledge transfer and scaling.
We have also proposed a modular network that can gracefully scale thanks to an adaptive prior over the search space of possible ways to connect modules.
Our experiments show that our approach yields a very desirable trade-off between accuracy and compute/memory usage, suggesting that modularization in a restricted search space is a promising avenue of investigation for continual learning and knowledge transfer.
A D ATASETS AND S TREAMS
The CTrL benchmark is built using the standard datasets listed in Tab. 3.
Some examples from these datasets are shown in Fig.
The composition of the different tasks is given in Tab. 4 and an instance of the long stream is presented in Tab. 5 and 6.
The tasks in S − , S + , S in , S out and S pl are all 10-way classiﬁcation tasks.
In S − , the ﬁrst task has 4000 training examples while the last one which is the same as the ﬁrst task, has only 400.
The vice versa is true for S + instead.
The last task of S in is the same as the ﬁrst task, except that the background color of the MNIST digit is different.
The last task of S out is the same as the ﬁrst task, except that label ids have been shufﬂed, therefore, if “horse” was associated to label id 3 in the ﬁrst task, it is now associated to label id 5 in the last task.
The S long stream is composed of both large and small tasks that have 5000 (or whatever is the maximum available) and 25 training examples, respectively.
Each task is built by choosing one of the datasets at random, and 5 categories at random in this dataset.
During task 1-33, the fraction of small tasks is 50%, this increases to 75% for tasks 34-66, and to 100% for tasks 67-100.
This is a challenging setting allowing to assess not only scalability, but also transfer ability and sample efﬁciency of a
Scripts to rebuild the given streams and evaluate models will be released upon publication.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
cifar [sea, pear, house, spider, aquarium \ ﬁsh]
cifar [kangaroo, ray, tank, crocodile, table]
cifar [trout, rose, pear, lizard, baby]
svhn [3, 2, 8, 1, 5]
cifar [skyscraper, bear, rocket, tank, spider]
cifar [telephone, porcupine, ﬂatﬁsh, plate, shrew]
cifar [lawn \ mower, crocodile, tiger, bed, bear]
svhn [3, 7, 1, 5, 6]
fashion-mnist [Ankle boot, Sneaker, T-shirt/top, Coat, Bag]
mnist [6, 9, 0, 3, 7]
cifar [automobile, truck, deer, bird, dog]  cifar [dog, airplane, frog, deer, automobile]
svhn [1, 9, 5, 3, 6]
cifar [whale, orange, chimpanzee, poppy, sweet \ pepper]
cifar [worm, camel, bus, keyboard, spider]
fashion-mnist [T-shirt/top, Coat, Ankle boot, Shirt, Dress]
cifar [dog, deer, ship, truck, cat]
cifar [cat, dog, airplane, ship, deer]
svhn [7, 6, 4, 2, 9]
mnist [9, 7, 1, 3, 2]
cifar [mushroom, butterﬂy, bed, boy
, motorcycle]  500fashion-mnist
[Shirt, Pullover, Bag, Sandal, T-shirt/top]  5000cifar [rabbit, bear, aquarium \ ﬁsh, bee, bowl]  500fashion-mnist
[Coat, T-shirt/top, Pullover, Shirt, Sandal]  5000fashion-mnist
[Pullover, Dress, Coat, Shirt, Sandal]  5000mnist [3, 9, 7, 6, 4]  4940cifar [deer, bird, dog, automobile, frog]  5000svhn [
[Table]
4]  5000cifar [
[Table]
, skunk, poppy, bridge, sweet \ pepper]  500cifar [caterpillar, can, motorcycle, rabbit, wardrobe]
[Table]
[9, 6, 2, 4, 8]  5000mnist [7, 1, 2, 0, 6]  4938cifar [dog, automobile, ship, airplane, cat]  5000mnist [0, 7, 6, 2, 4]  4920cifar [bird, deer, airplane, dog, ship]  5000cifar [mountain, bicycle, caterpillar, spider, possum]  500svhn [8, 3, 4, 0, 6]  5000svhn [1, 5, 9, 0, 8]  5000cifar [can, dolphin, house, pickup \ truck, crab]  500cifar [squirrel, possum, crocodile, mountain, hamster]  500mnist [7, 0, 1, 6, 2]  4938fashion-mnist
[T-shirt/top, Dress, Trouser, Shirt, Sneaker]
5000cifar [cat, frog, automobile, dog, airplane]  5000cifar [automobile, cat, dog, ship, horse]  5000cifar [cup, otter, orchid, kangaroo, rose]  500mnist [1, 5, 7, 2, 9]  4892svhn [6, 5, 3, 2, 7]  5000cifar [dog, deer, cat, frog, bird]  5000mnist [6, 2, 5, 9, 4]  4832cifar [pear, rocket, sea, road, orange]  500svhn [0, 8, 4, 6, 1]  5000cifar [truck, horse, ship, deer, dog]
[Table]
B L EARNING A LGORITHM
We provide a more complete description of the two learning algorithms for MNTDP-S and MNTDPD.
In the deterministic case, all the architectures (given how we restrict the search space, we have 7 of them) are trained over the training set, and the best path is retained based on its score on the validation set.
To avoid overﬁtting when using MNDTDP-S, we split the training set into two halves
D t 1 and D t 2 .
The ﬁrst part is used to update the module parameters θ , while the second is used to update the parameters in the distribution over paths, Γ .
If both sets of parameters were trained on the same dataset, Γ would favor paths prone to overﬁtting since they will results in a large decrease of its training loss and therefore a larger reward.
When they are trained on different sets, Γ has to select paths with a reasonable amount of free parameters, allowing θ to learn and generalize enough to decrease the loss on both training sets.
Then, the most promising architecture is selected based on arg max Γ , and ﬁne-tuned over the whole D t .
In this case, the validation set is used only for hyper-parameters selection.
Source code to reproduce the experiments will be provided upon publication.
1 Data: Dataset of task t : D t .; 2 Past predictors: f ( x,j |S ) for j = 1 ,...,t − 1 ; 3 Find closest task: j ∗ = argmax j E ( x,y )
∼D t [∆( NN ( f ( x,j |S )) ,y )] , where NN is the 5-nearest neighbor classiﬁer in feature space; 4 Deﬁne search space: Take path corresponding to predictor of task j ∗ and add a new randomly initialized module at every layer.
Γ : distribution over paths; π i : i -th path; 5 Split train set in two halves: D t 1 and D t 2 ; 6 while loss in eq.
4 has not converged doget sample
[Table]
[Table] [Table]
iteration
[Table]
2] ;sample
[Table]
;if odd iteration or max
[Table]
>
[Table]
99 then
forward/backward and update θ ( π k ) (only newly added modules) elseforward/backward and update Γ ; end end 12 Let i
[Table]
be the path with largest values in Γ , then
[Table]
f ( x,t |S ∪ t ) to π i ∗ .
[Table]
C A RCHITECTURE OF THE M ODELS AND H YPER -P ARAMETERS
The model used on Permuted-MNIST is presented in table 7.
On all other tasks, the base architecture for all baselines and MNTDP is the same CNN, a ResNet convolutional neural network as reported in Table 8 or AlexNet as reported in Table 9.
The table also shows how layers are grouped into modules for MNTDP and PNN.
[Table]
As presented in section 5.2, the stream can be visited only once, preventing stream-level hyperparameters tuning.
Exceptions are made for HAT (Serr`a et al., 2018) because we have been using authors’ implementation, and for EWC and Online-EWC since these approaches fail in the proposed setting.
The constraint strength hyper-parameter λ must be tuned at the stream level since a tasklevel tuning of λ results in little or no constraint at all, leading to severe catastrophic forgetting.
The stream-level hyper-parameters optimization considers 9 values for λ { 1, 5, 10, 50, 100, 500, 10 3 , 5 × 10 3 , 10 4 } .
Note that this gives an unfair advantage to EWC, Online-EWC and HAT, as all other methods including MNTDP use task-level cross-validation as described in § 5.2.
[Table] [Table]
For all methods and experiments, we use the Adam optimizer (Kingma & Ba, 2015) with β 1 = 0 .
9 , β 2 = 0 .
999 and (cid:15) =
10 − 8 .
For each task and each baseline, two learning rates { 10 − 2 , 10 − 3 } and 3 weight decay strengths { 0 , 10 − 5 , 10 − 4 } are considered.
Early stopping is performed on each task to identify the best step to stop training.
When the current task validation accuracy stops increasing for 300 iterations, we restore the learner to its state after the best iteration and stop training on the current task.
For MNTDP-S, we consider two additional learning rates for the Γ optimization { 10 − 2 , 10 − 3 } .
An entropy regularization term on Γ is added to the loss to encourage exploration, preventing an early convergence towards a sub-optimal path.
The weight for this regularization term is set tothroughout our experiments
Finally, since small tasks in S long have very few examples in the validation sets, we use test-time augmentation to prevent overﬁtting during the grid search.
For each validation sample, we add four augmented copies following the same data augmentation procedure used during training.
D M EMORY COMPLEXITY OF THE M ODELS
In the main paper we report the overall memory consumption by the end of training.
However, modular architectures like MNTDP use only a sparse subset of modules for a given task.
In this section, we report the memory complexity both at training and test time.
Table 10 shows that all methods but DEN and RCL are as fast to evaluate at test time as an independent network because they all use a single path.
E A DDITIONAL R ESULTS E. METRIC
LCA
In addition to the metrics used in the main paper, we report the area under the learning curve after β optimization steps (Chaudhry et al., 2019a) as a metric to assess learning speed:
[Equation]
Where A b,t ( f ) corresponds to the test set accuracy on task t of learner f after having seen b batches from t
[Table]
E. C LASSICAL S TREAMS
Table 11 and 12 report performance across all axes of evaluation on the standard Permuted MNIST and Split CIFAR 100 streams.
[Table] [Table]
E. T HE CT R L BENCHMARK
Figure 1 and 6 provide radar plots of the models evaluated on the CTrL benchmark.
The companion tables of these plots are in Tab. 13, 14, 15, 16 and
[Figure]
E. D ISCOVERED PATHS
(0, 'INs') (0, 0) (0, 1) (0, 2) (1, 2) (3, 2) (4, 2) (0, 3) (2, 3) (0, 4) (0, 5) (0, 6) (0, 7) (5, 7) (0, 'OUT') (1, 'INs') (1, 3) (1, 4) (1, 5) (1, 6) (1, 7) (1, 'OUT') (2, 'INs') (2, 4) (2, 5) (2, 6) (2, 7) (2, 'OUT') (3, 'INs') (3, 3) (3, 4) (3, 5) (3, 6) (3, 7) (3, 'OUT') (4, 'INs') (4, 3) (4, 4) (4, 5) (4, 6) (4, 7) (4, 'OUT') (5, 'INs') (5, 'OUT')
Firefox ﬁle:///home/tom/lileb-res3/results/deterministic.html
We observe that a signiﬁcant amount of new parameters are introduced for tasks 2, 3, 4 and 5, which are very different from the ﬁrst task.
The model is however able to correctly identify that the last task is very similar to the ﬁrst one, resulting in very large reuse of past modules and only introducing a new classiﬁcation layer
(0, 'INs') (0, 0) (0, 1) (0, 2) (0, 3) (2, 3) (0, 4) (4, 4) (0, 5) (0, 6) (5, 6) (0, 7) (0, 'OUT') (1, 'INs') (1, 0) (1, 1) (1, 2) (1, 3) (3, 3) (1, 4) (1, 5) (1, 6) (1, 7) (1, 'OUT') (2, 'INs') (2, 4) (2, 5) (2, 6) (2, 7) (2, 'OUT') (3, 'INs') (3, 4) (3, 5) (3, 6) (3, 7) (3, 'OUT') (4, 'INs') (4, 5) (4, 6) (4, 7) (4, 'OUT') (5, 'INs') (5, 7) (5, 'OUT')
Firefox ﬁle:///home/tom/lileb-res3/results/stochastic.html
[Table]
MNTDP-S 0.
MNTDP-S (k=all) 0.
MNTDP-D 0.
MNTDP-D* 0.
HAT* † 0.
HAT* † (Wide) 0.
[Figure]
[Table]
MNTDP-S 0.
[Table]
MNTDP
[Table]
S
[Table]
=all) 0.
[Table] [Table]
MNTDP
[Table]
D 0.
[Table] [Table]
[Table] [Table]
MNTDP
[Table] [Table]
[Table] [Table]
[Table] [Table]
HAT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Wide
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Figure]
[Table] [Figure] [Table] [Table]
[Table] [Table] [Table] [Table]
MNTDP
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
MNTDP
[Table]
all)
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Figure]
[Table] [Table]
HAT
[Table]
[Figure]
[Table]
using stream-level cross-validation (see Section 5.2)
[Figure]
FLOPs
[Figure]