L EARN THE T IME TO L EARN : R EPLAY S CHEDULING FOR C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Replay-based continual learning has shown to be successful in mitigating catastrophic forgetting.
Most previous works focus on increasing the sample quality in the commonly small replay memory.
However, in many real-world applications, replay memories would be limited by constraints on processing time rather than storage capacity as most organizations do store all historical data in the cloud.
Inspired by human learning, we demonstrate that scheduling over which tasks to revisit is critical to the ﬁnal performance with ﬁnite memory resources.
To this end, we propose to learn the time to learn for a continual learning system, in which we learn schedules over which tasks to replay at different times using Monte Carlo tree search to illustrate this idea.
We perform extensive evaluation and show that our method can learn replay schedules that signiﬁcantly improve ﬁnal performance across all tasks than baselines without considering the scheduling.
Furthermore, our method can be combined with any other memory selection methods leading to consistently improved performance.
Our results indicate that the learned schedules are also consistent with human learning insights.
I NTRODUCTION
Many organizations deploying machine learning systems receive large volumes of data daily where these new data are often associated with new tasks.
Although all historical data are stored in the cloud in practice, retraining machine learning systems on a daily basis is prohibitive both in time and cost.
In this setting, the systems must continuously adapt to new tasks without forgetting the previously learned abilities.
Continual learning methods (De Lange et al., 2019; McCloskey &
Cohen, 1989; Parisi et al., 2019) address this challenge where, in particular, replay-based methods (Chaudhry et al., 2019; Hayes et al., 2020) have shown to be very effective in achieving great prediction performance and retaining knowledge of old tasks.
Replay-based methods mitigate catastrophic forgetting by revisiting a small set of samples, which is feasible to process compared to the size of the historical data.
In traditional continual learning literature, the replay memory is limited due to the assumption that historical data are not available.
In the real-world setting where historical data are in fact always available, the requirement of small memory remains due to processing time and cost issues.
Most research on replay-based continual learning has been focused on the sample quality in the memory (Aljundi et al., 2019; Borsos et al., 2020; Chaudhry et al., 2019; Chrysakis & Moens, 2020;
Nguyen et al., 2017; Rebufﬁ et al., 2017; Yoon et al., 2021) or data compression to increase the memory capacity (Hayes et al., 2020; Iscen et al., 2020; Pellegrini et al., 2019).
Common for these methods is that the memory allocates an equal amount of space for storing samples from old tasks.
When learning new tasks, the whole memory is replayed to mitigate catastrophic forgetting.
However, in life-long learning settings, this simple strategy would be inefﬁcient as the memory must store a large number of tasks.
Furthermore, these methods ignore the time to learn old tasks again which is important in human learning.
Humans are continual learning systems, and different methods have been developed to enhance memory retention, such as spaced repetition (Dempster, 1989;
Ebbinghaus, 2013; Landauer & Bjork, 1977) which is used often in education.
These education methods focus on the scheduling of learning and rehearsal of previous learned knowledge.
In this work, we argue that ﬁnding the proper schedule of what tasks to replay in the ﬁxed memory setting is critical for continual learning.
To demonstrate our claim, we perform a simple experiment
[Figure]
on the Split MNIST (Zenke et al., 2017) dataset where each task consists of learning the digits 0/1, 2/3, etc. arriving in sequence.
The replay memory contains data from task 1 and can only be replayed at one point in time.
Figure 1 shows how the task performances progress over time when the memory is replayed at different time steps.
In this example, the best ﬁnal performance is achieved when the memory is used when learning task 5. Note that choosing different time points to replay the same memory leads to noticeably different results in the ﬁnal performance.
These results indicate that scheduling the time when to apply replay can inﬂuence the ﬁnal performance signiﬁcantly of a continual learning system.
To this end, we propose learning the time to learn, in which we learn replay schedules of which tasks to replay at different times inspired from human learning (Dempster, 1989).
We illustrate the advantages with replay scheduling by using Monte Carlo tree search (MCTS)
(Coulom, 2006) to learn policies for replay.
More speciﬁcally, we train a neural network on the current task dataset mixed with the scheduled replay samples and measure the ﬁnal performance of the network to evaluate the replay schedules selected by MCTS.
In summary, our contributions are:
• We demonstrate the importance of replay scheduling in continual learning and propose to learn the time to learn which tasks to replay (Section 3.1 and 3.2).
• We use MCTS as an example method to illustrate how replay schedules for continual learning can be learned by establishing a ﬁnite set of memory compositions that can be replayed at every task (Section 3.3).
• We demonstrate with several benchmark datasets that learned scheduling can improve the continual learning performance signiﬁcantly in the ﬁxed size memory setting (Section 4. and 4.4).
Furthermore, we show that our method can be combined with any other memory selection methods (Section 4.3), as well as being efﬁcient in situations where the memory size is even smaller than the number of classes (Section 4.5).
R ELATED W ORK
In this section, we give a brief overview of continual learning methods, essentially replay-based methods, as well as spaced repetition techniques for human continual learning.
Continual Learning.
Traditional continual learning can be divided into three main areas, namely regularization-based, architecture-based, and replay-based approaches.
Regularization-based methods aim to mitigate catastrophic forgetting by protecting parameters inﬂuencing the predictive performance from wide changes and use the rest of the parameters for learning the new tasks (Adel et al., 2019; Chaudhry et al., 2018a;
Kirkpatrick et al., 2017; Li & Hoiem, 2017; Nguyen et al., 2017; Rannen et al., 2017; Schwarz et al., 2018; Zenke et al., 2017).
Architecture-based methods isolate task-speciﬁc parameters by either increasing network capacity (Rusu et al., 2016; Yoon et al., 2019; 2017) or freezing parts of the network (Mallya & Lazebnik, 2018; Serra et al., 2018) to maintain good performance on previous tasks.
Replay-based methods mix samples from old tasks with the current dataset to mitigate catastrophic forgetting, where the replay samples are either stored in an external memory (Chaudhry et al., 2019; Hayes et al., 2020; Isele & Cosgun, 2018; Lopez-Paz &
Ranzato, 2017) or generated using a generative model (Shin et al., 2017; van de Ven & Tolias, 2018).
Regularization-based approaches and dynamic architectures have been combined with replay-based approaches to methods to overcome their limitations (Chaudhry et al., 2018a;b; Douillard et al.,
2020; Ebrahimi et al., 2020; Joseph & Balasubramanian, 2020; Mirzadeh et al., 2020; Nguyen et al., 2017; Pan et al., 2020; Pellegrini et al., 2019; Rolnick et al., 2018; von Oswald et al., 2019).
Our work relates most to replay-based methods with external memory which we spend more time on describing in the next paragraph.
Replay-based Continual Learning.
A commonly used memory selection strategy of replay samples is random selection.
Much research effort has focused on selecting higher quality samples to store in memory (Aljundi et al., 2019; Borsos et al., 2020; Chaudhry et al., 2019; Chrysakis &
Moens, 2020; Hayes et al., 2019; Isele & Cosgun, 2018; Lopez-Paz & Ranzato, 2017; Nguyen et al., 2017; Rebufﬁ et al., 2017; Yoon et al., 2021).
Chaudhry et al.
(2019) reviews several selection strategies in scenarios with tiny memory capacity, e.g., reservoir sampling (Vitter, 1985), ﬁrst-in ﬁrst-out buffer (Lopez-Paz & Ranzato, 2017), k-Means, and Mean-of-Features (Rebufﬁ et al., 2017).
However, more elaborate selection strategies have been shown to give little beneﬁt over random selection for image classiﬁcation problems (Chaudhry et al., 2018a;
Hayes et al., 2020).
More recently, there has been work on compressing raw images to feature representations to increase the number of memory examples for replay (Hayes et al., 2020; Iscen et al., 2020; Pellegrini et al., 2019).
Our approach differs from the above mentioned works since we focus on learning to select which tasks to replay at the current task rather than improving memory selection or compression quality of the samples in the memory.
Replay scheduling can however be combined with any selection strategy as well as storing feature representations.
Human Continual Learning.
Humans are continual learning systems in the sense of learning tasks and concepts sequentially.
Furthermore, humans have an impressive ability to memorize experiences but can forget learned knowledge gradually rather than catastrophically (French, 1999).
Different learning techniques have been suggested for humans to memorize better (Dunlosky et al., 2013; Willis, 2007).
An example is spaced repetition which gradually increases time-intervals between rehearsals for retaining long-term memory (Dempster, 1989).
This technique has been studied frequently and was inspired from the works of Ebbinghaus (2013) on memory retention.
For example, Landauer & Bjork (1977) demonstrated that memory training schedules using adjusted spaced repetition were better at preserving memory than uniformly spaced training.
Hawley et al.
(2008) studies the efﬁcacy of spaced repetition on adults with probable Alzheimer’s disease for learning face-name association.
Several works in continual learning with neural networks are inspired by or have a connection to human learning techniques, including spaced repetition (Amiri, 2019; Amiri et al., 2017;
Feng et al., 2019; Smolen et al., 2016), mechanisms of sleep (Ball et al., 2020; Mallya & Lazebnik, 2018; Schwarz et al., 2018), and reactivation of memories (Hayes et al., 2020; van de
Ven et al., 2020).
Our replay scheduling method is inspired by spaced repetition; we learn schedules of which memory samples to use for replay at different time steps.
M ETHOD
In this section, we describe our method for learning replay schedules for continual learning.
The idea is to learn schedules of which memory examples the network should rehearse at different times.
We use Monte Carlo tree search (MCTS)
(Browne et al., 2012; Coulom, 2006) to learn a scheduling policy by encouraging searches for promising replay schedules based on the classiﬁcation accuracy.
3. P ROBLEM S ETTING
We focus on the setting considering the real-world continual learning needs where all historical data are available but are prohibitively large.
Therefore, only a small amount of historical data can be used when adapting the model to new data due to processing capability consideration.
Thus, the goal is to learn how to select subsets of historical data to efﬁciently mitigate catastrophic forgetting when learning new tasks.
We refer to these subsets of historical data as the replay memory throughout the paper, where the size of the replay memory affects the processing time when learning a new task t .
Moreover, we focus on composing the replay memory based on the seen tasks in the historical data rather than single stored instances.
Next, we introduce the notation of our problem setting which resembles the traditional continual learning setting for image classiﬁcation.
We let a neural network f θ , parameterized by θ , learn T tasks sequentially given their corresponding task datasets D 1 , . . .
, D T arriving in order.
The t -th dataset D t = { ( x ( i ) t , y ( i ) t ) } N t i =1 consists of N t samples where x ( i ) t and y ( i ) t are the i -th data point and class label respectively.
The training objective at task t is given by
[Equation]
where (cid:96) ( · ) is the loss function, e.g., cross-entropy loss in our case.
Since D t is only accessible at time step t , the network f θ is at risk of catastrophically forgetting the previous t − 1 tasks when learning the current task.
Replay-based continual learning methods mitigate the forgetting of old tasks by storing old examples in an external replay memory, that is mixed with the current task dataset during training.
Next, we describe our method for constructing this replay memory.
We assume that historical data from old tasks are accessible at any time step t .
However, since the historical data is prohibitively large, we can only ﬁll a small replay memory M with M historical samples for replay due to processing time constraints.
The challenge is how to ﬁll the replay memory with M samples that efﬁciently retain the knowledge of old tasks when learning new tasks.
We focus on selecting the samples on task-level by deciding on the task proportion ( a 1 , . . .
, a t − 1 ) of samples to fetch from each task, where a i ≥ 0 is the proportion of M examples from task i and (cid:80) t − 1 i =1 a i = 1 .
Consequently, we need a method for choosing these task proportions of which old tasks to replay.
To simplify this selection, we construct a discrete set of choices for possible task proportions telling how many samples from each task to use when constructing the replay memory M .
3. R EPLAY S CHEDULING FOR C ONTINUAL L EARNING
In this section, we describe our replay scheduling method for selecting the replay memory at different time steps.
We deﬁne a replay schedule as a sequence S =
( a 1 , . .
. , a T − 1 ) , where a i = ( a 1 , . .
. , a T − 1 ) for 1 ≤ i ≤ T − 1 , is the sequence of task proportions used for determining how many samples per task to ﬁll the replay memory with at task i .
To make the selection of task proportions tractable, we construct an action space with a discrete number of choices for the task proportions from old tasks.
We use the following ways to construct this action space:
At the time to learn task t , we have t − 1 historical tasks that we can choose from.
We create t − 1 bins b t =
[ b 1 , b 2 , ....
b t − 1 ] and choose a task index to sample for each bin b i ∈ 1 , .., t − 1 .
We treat the bins as interchangeable and only keep the unique choices.
For example, at task 3, we have seen task 1 and 2; so the unique choices of vectors are [1 , 1] , [1 , 2] , [2 , 2] , where [1 , 1] indicates that all memory samples are from task 1, [1 , 2] indicates that half memory is from task 1 and the other half are from task etc.
The task proportion are then computed by counting the number of occurrences of each task index in b t and dividing by the t − 1 , such that a t = bincount ( b t ) /
( t − 1) .
If the memory size
M is not divisible by the task proportion value for task i , we round the number of replay samples from task i , i.e., a i · M , up or down accordingly while keeping the memory size ﬁxed.
From this speciﬁcation, we can build a tree of different replay schedules to evaluate with the network.
Figure 2 shows an example of such a replay schedule tree with Split MNIST (Zenke et al., 2017) where the memory size has been set to M = 8 .
The ﬁgure shows the current tasks with an example image of the task classes on the left, and the right side shows examples of possible replay memories that can be evaluated.
The memory starts as the empty set, i.e. M 1 = ∅ , at task 1.
Before learning task 2, M 2 is ﬁlled with M task 1 examples since this is the only task seen so far.
At task 3, the memory compositions we can choose from are M examples from either task 1 or 2, as well as equally ﬁlling M 3 with four examples each from both tasks. A replay schedule is represented as a path traversal of different replay memory compositions from task 1 to task 5. We have color-coded three examples of possible schedules in Figure 2 to use for illustration: the blue path represents a replay schedule where only task 1 examples are replayed at all future tasks.
The red path represents using an equally distributed amount of memory samples per task in the memory, and the purple path represents a schedule where the memory is ﬁlled M examples from the previously visited task. Note that all other possible paths in the tree are also valid replay schedules.
3. M ONTE C ARLO T REE S EARCH FOR R EPLAY S CHEDULES
Our proposed setup to ﬁnd replay schedules is to discretize the number of possible task proportions per task.
The action space of task proportions then turn into a tree shape where each node represents [Figure] Task· · · · · · · · · · · · · · · · · · · · · · · · a task proportion that corresponds to a speciﬁc composition of the replay memory M , as can be seen in Figure 2.
To ﬁnd the efﬁcient replay schedule S in mitigating catastrophic forgetting, we can perform an exhaustive search (e.g., breadth-ﬁrst search) where all possible replay schedules are evaluated.
However, since the tree grows fast with the number of tasks, we need a scalable method that enables tree searches in large search spaces.
To this end, we propose to use Monte
Carlo tree search (MCTS) (Browne et al., 2012; Coulom, 2006) since MCTS has been successful in applications with similar conditions (Chaudhry & Lee, 2018; Gelly et al., 2006; Silver et al., 2016).
In our setting, MCTS can help us to concentrate the search in directions towards promising replay schedules in terms of classiﬁcation performance and thus learn the time to learn.
Before outlining the steps for performing MCTS, we describe the notation for the tree search.
The tree has T levels where level t corresponds to task t .
Each tree level t contains a set of nodes
V t =
{ v it } K t i =1 where K t is the number of nodes at level t .
Every node v it has a corresponding sequence of task proportions a it , which can be used to retrieve the replay memory from the historical data.
Referring back to Figure 2, the replay memory composition on the root node v 11 is the empty set M = ∅ , since the historical data is empty.
At task 2, the only task proportion is a 12 = ( a 1 , a 2 , ..., a T − 1 ) =
(1 . 0 , 0 . 0 , ..., 0 . 0) , so the replay memory at node v 12 is only ﬁlled with samples from task 1.
In the rest of the paper, we denote a node at task t as v t by ignoring the superscript i to avoid clutter in the notation.
At every visited node v t , we store the corresponding task proportions a t in the replay schedule S .
The ﬁnal replay schedule is then used for constructing the replay memory at each time step t during the continual learning training.
Next, we brieﬂy outline the steps for performing MCTS in the search for replay schedules.
Selection.
An MCTS iteration begins by selecting the next node to visit from the root node.
If the current node v t has visited all its children during the previous iterations, the next node is selected by evaluating the Upper Conﬁdence Tree (UCT) (Kocsis & Szepesv ´ ari, 2006) function for all children.
We use the following UCT function proposed by Chaudhry & Lee (2018) to evaluate the score for moving from node v t to its child v t +1 :
[Equation]
The reward function Q ( · ) contains the rewards for previous searches that has passed through the child v t +1 .
The exploration constant C ≥ 0 determines the degree of exploration of less visited replay schedules based on the number of visits N ( v t ) and N ( v t +1 ) to the corresponding nodes in the tree.
The child node to visit next is the one with the highest UCT score.
Expansion.
Whenever the current node v t has unvisited child nodes, the search tree is expanded with one of the unvisited child nodes v t +1 selected with uniform sampling.
Simulation and Reward.
After the expansion step, the search proceeds by selecting the succeeding nodes uniformly at random until a terminal node v T is reached.
The replay schedule S is then collected by appending the task proportions from the visited nodes during the iteration.
We train the network with the replay memories retrieved from S and evaluate the average accuracy over all tasks after learning the ﬁnal task T as the reward r , i.e., r = 1 T (cid:80)
Ti =1 A ( val ) T,i , where A ( val ) T,i is the validation accuracy of task i after learning task T .
Backpropagation.
The reward r is backpropagated from the expanded node v t up to the root node v 1 , where the number of visits N ( · ) and reward function Q ( · ) for each node is updated.
We provide pseudocode in Algorithm 1 in Appendix B. Finally, we emphasize that we use MCTS to illustrate in a simple setting the importance of learning the time to learn old tasks again.
E XPERIMENTS
We evaluated our replay scheduling method empirically on six common benchmark datasets for continual learning.
We show that scheduling of the replay memory improves signiﬁcantly over replaying equal proportions across the tasks.
Furthermore, we demonstrate that replay scheduling can be as efﬁcient as replaying all available memory samples in settings where only 1 example per class can be stored for replay.
We denote our method as Replay Scheduling MCTS (RS-MCTS) and select the result on the held-out test sets from the replay schedule that yielded the best reward on the validation set during the search.
We perform experiments using 5 different seeds on all datasets.
Our code is available as part of the supplementary material.
The full details on experimental settings are in Appendix A and additional results are in Appendix D.
NEW (dhXe) Datasets.
We conduct experiments on six datasets commonly used as benchmarks in the continual learning literature: Split MNIST (LeCun et al., 1998; Zenke et al., 2017), Fashion-MNIST (Xiao et al., 2017), Split notMNIST (Bulatov, 2011), Permuted MNIST (Goodfellow et al., 2013), Split
CIFAR-100 (Krizhevsky & Hinton, 2009), and Split miniImagenet (Vinyals et al., 2016).
We randomly sample 15% of the training data from each task to use for validation when computing the reward for the MCTS simulations. NEW (8z4B) Baselines.
We compare RS-MCTS to using 1) random replay schedules (Random), 2) equal task schedules (ETS), and 3) a heuristic scheduling method (Heuristic).
The ETS baseline uses equal task proportions, such that M/ ( t − 1) samples per task are replayed during learning of task t , and use both training and validations sets for training such that they use the same amount of data as RSMCTS.
The Heuristic baseline replays the old tasks which accuracy on the validation set is below a certain threshold proportional to the best achieved validation accuracy on the task. Here, the replay memory is ﬁlled with M/k where k is the number of tasks that need to be replayed according to their decrease in validation accuracy. See Appendix C for more details on the Heuristic baseline. Network Architectures.
We use a multi-head output layer for all datasets except for Permuted
MNIST where the network uses single-head output layer.
We use a 2-layer MLP with 256 hidden units for Split MNIST, Split FashionMNIST, Split notMNIST, and Permuted MNIST.
For Split
CIFAR-100, we use the CNN architecture used in Vinyals et al.
and Schwarz et al.
For Split mniImagenet, we apply the reduced ResNet-18 from Lopez-Paz & Ranzato (2017).
Evaluation Metric.
We use the average test accuracy over all tasks after learning the ﬁnal task, i.e., ACC = 1 T (cid:80) Ti =1 A T,i where A T,i is the test accuracy of task i after learning task T .
4. R ESULTS ON R EWARD FOR RS-MCTS
In the ﬁrst experiments, we show that the replay schedules from RS-MCTS yield better performance than replaying an equal amount of samples per task. The replay memory size is ﬁxed to M = 10 for
Split MNIST, FashionMNIST, and notMNIST, and M = 100 for Permuted MNIST, Split CIFAR100, and Split miniImagenet.
Uniform sampling is used as the memory selection method for all
[Figure]
methods in this experiment.
For the 5-task datasets, we provide the optimal replay schedule found from a breadth-ﬁrst search (BFS) over all possible replay schedules in our action space as an upper bound for RS-MCTS.
As the search space grows fast with the number of tasks, it will even with only 5 continual learning tasks (which corresponds to a tree with depth of 4) yield a tree with 1050 leaf nodes.
Thus, BFS becomes computationally infeasible when we have 10 or more tasks. NEW
[Equation] Figure 3 shows the progress of ACC over the MCTS iterations by RS-MCTS for all datasets.
We also show the best ACC metrics for ETS, Heuristic, and BFS as straight lines.
We observe that RS-MCTS outperforms Random and ETS successively with more iterations.
Furthermore, RSMCTS approaches the upper limit from BFS on the 5-task datasets.
For Permuted MNIST and Split CIFAR-100, the Heuristic baseline and RS-MCTS perform on par after 50 iterations.
This shows that Heuristic with careful tuning of the validation accuracy threshold can be a strong baseline when comparing against replay scheduling.
The top row of Table 1 shows the ACC for each method for this experiment.
We note that RS-MCTS outperforms ETS signiﬁcantly for most datasets and performs on par with Heuristic.
4. R EPLAY S CHEDULE V ISUALIZATIONS
[Figure]
We visualize a learned replay schedule from Split
CIFAR-100 with memory size M = 100 to gain insights into the behavior of the scheduling policy from RS-MCTS.
Figure 4 shows a bubble plot of the task proportions that are used for ﬁlling the replay memory at every task.
Each color of the circles corresponds to a historical task and its size represents the proportion of examples that are replayed at the current task.
The sum of all points from all circles at each column is ﬁxed at the different time steps since the memory size M is ﬁxed.
The task proportions vary dynamically over time in a sophisticated nonlinear way which would be hard to replace by a heuristic method.
Moreover, we can observe space repetition style scheduling in many tasks, e.g., task 1-3 are replayed with similar proportion at the initial tasks but eventually starts varying the time interval between replay.
Also, taskand 6 need less replay in their early stages, which could potentially be that they are simpler or correlated with other tasks.
We provide a similar visualization for Split MNIST in Figure 7 in Appendix
NEW (kfup) D.1 to bring more insights to the beneﬁts of replay scheduling.
4. A LTERNATIVE M EMORY S ELECTION M ETHODS
In this section, we show that our method can be combined with any memory selection method for storing replay samples.
In addition to uniform sampling, we apply various memory selection methods commonly used in the continual learning literature, namely k -means clustering, k -center clustering (Nguyen et al., 2017), and Mean-of-Features (MoF) (Rebufﬁ et al., 2017). We compare [Figure] [Figure]
[Table] our method and ETS combined with these different selection methods.
As in Section 4.1, we set the replay memory size M = 10 for the 5-task datasets and M = 100 for the 10- and 20-task datasets.
Table 1 shows the results across all datasets.
We note that using the replay schedule from RS-MCTS
NEW (8z4B) outperforms the baselines when using the alternative selection methods, where MoF performs the best on most datasets.
We show that our method can improve the performance across different choices of memory size. In this experiment, we set the replay memory size to ensure the ETS baseline replays an equal number of samples per class at the ﬁnal task. The memory size is set to M = n cpt · n spc · ( T − 1) , where n cpt is the number of classes per task in the dataset and n spc are the number of samples per class we wish to replay at task T for the ETS baseline.
In Figure 5, we observe that RS-MCTS yield better
FIX (8z4B) task accuracies than ETS, especially for small memory sizes. Both RS-MCTS and ETS perform better than Heuristic as M increases showing that Heuristic requires careful tuning of the validation accuracy threshold.
[Table]
4. E FFICIENCY OF R EPLAY S CHEDULING
[Table] [Figure] [Table]
[Figure]
[Table]
Here, we illustrate the efﬁciency of replay scheduling with comparisons to several common replay-based continual learning baselines in an even more extreme memory setting.
Our goal is to investigate if scheduling over which tasks to replay can be more efﬁcient in situations where the memory size is even smaller than the number of classes.
To this end, we set the replay memory size for our method to M = 2 for the 5-task datasets, such that only 2 samples can be selected for replay at all times.
For the 10and 20-task datasets which have 100 classes, we set M = 50 .
We then compare against the most memory efﬁcient continual learning baselines, namely A-GEM (Chaudhry et al., 2018b), ERRing (Chaudhry et al., 2019) which show promising results with 1 sample per class for replay after learning each task, and with uniform memory selection as reference. Additionally, we compare to using random replay schedNEW (8z4B) ules (Random) with the same memory setting as for RS-MCTS.
We visualize the memory usage when training on one of the 5-task datasets for our method and the baselines in Figure 6.
We also visualize the memory usages for the other benchmarks in the Appendix E.
Table 2 shows the ACC for each method across all datasets.
Despite using signiﬁcantly fewer samples for replay, RS-MCTS performs better or on par with the best baselines on all datasets except
Split CIFAR-100.
These results show that replay scheduling can be even more efﬁcient than the state-of-the-art memory efﬁcient replay-based methods, which indicate that learning the time to learn is an important research direction in continual learning.
C ONCLUSIONS
We propose learning the time to learn, i.e., in a real-world continual learning context, learning schedules of what previous tasks to replay at different times.
To the best of our knowledge, we are the ﬁrst to consider the time to learn in machine learning inspired by human learning techniques.
We demonstrate with an example method that learned replay schedules produce signiﬁcantly improved results under the same memory budget when comparing with the method without scheduling.
Furthermore, the dynamic behavior of the learned schedules showed similarities to human learning techniques, such as spaced repetition, by rehearsing previous tasks with varying time intervals.
Finally, we showed that replay scheduling allows for utilizing the memory more efﬁciently compared to standard benchmarks replaying all memory samples in the tiny memory setting.
In future work, we would like to explore choosing memory samples on an instance level as the current work selects samples on task level.
This would need a policy learning method, that scales to large action spaces which is a research challenge by itself.
Also, our demonstrated method with
MCTS can be inefﬁcient, especially in continual learning settings, in terms of training time and since it needs to learn for each dataset and application separately.
In future work, we will incorporate reinforcement learning methods that generalize.
We will therefore extend replay scheduling to learn general policies that can be directly applied to any new application and domain.
A PPENDIX
This supplementary material is structured as follows:
• Appendix A: Full details of the experimental settings.
• Appendix B: Pseudocode of RS-MCTS in Algorithm 1 to provide more details about our method.
• Appendix C describes the Heuristic scheduling baseline.
• Appendix D shows additional experimental results.
• Appendix E includes additional ﬁgures.
A E XPERIMENTAL S ETTINGS
In this section, we describe the full details of the experimental settings used in this paper.
Datasets: We conduct experiments on six datasets commonly used in the continual learning literature.
Split MNIST (Zenke et al., 2017) is a variant of the MNIST (LeCun et al., 1998) dataset where the classes have been divided into 5 tasks incoming in the order 0/1, 2/3, 4/5, 6/7, and 8/9.
Split Fashion-MNIST (Xiao et al., 2017) is of similar size to MNIST and consists of grayscale images of different clothes, where the classes have been divided into the 5 tasks T-shirt/Trouser,
Pullover/Dress, Coat/Sandals, Shirt/Sneaker, and Bag/Ankle boots.
Similar to MNIST, Split notMNIST (Bulatov, 2011) consists of 10 classes of the letters A-J with various fonts, where the classes are divided into the 5 tasks A/B, C/D, E/F, G/H, and I/J. We use training/test split provided by
Ebrahimi et al.
(2020) for Split notMNIST.
Permuted MNIST Goodfellow et al.
(2013) dataset consists of applying a unique random permutation of the pixels of the images in original MNIST to create each task, except for the ﬁrst task that is to learn the original MNIST dataset.
We reduce the original MNIST dataset to 10k samples and create 9 unique random permutations to get a 10-task version of Permuted MNIST.
In Split CIFAR-100 (Krizhevsky & Hinton, 2009), the 100 classes are divided into 20 tasks with 5 classes for each task (Lopez-Paz & Ranzato, 2017; Rebufﬁ et al., 2017).
Similarly, Split miniImagenet Vinyals et al.
(2016) consists of 100 classes randomly chosen from the original Imagenet dataset where the 100 classes are divided into 20 tasks with 5 classes per task.
Network Architectures: We use a 2-layer MLP with 256 hidden units and ReLU activation for
Split MNIST, Split FashionMNIST, Split notMNIST, and Permuted MNIST.
We use a multi-head output layer for each dataset except Permuted MNIST where the network uses single-head output layer.
For Split CIFAR-100, we use a multi-head CNN architecture built according to the CNN in (Adel et al., 2019; Schwarz et al., 2018; Vinyals et al., 2016), which consists of four 3x3 convolutional blocks, i.e. convolutional layer followed by batch normalization (Ioffe & Szegedy, 2015), with 64 ﬁlters, ReLU activations, and 2x2 Max-pooling.
For Split mniImagenet, we use the reduced
ResNet-18 from Lopez-Paz & Ranzato (2017) with multi-head output layer.
Hyperparameters: We train all networks with the Adam optimizer (Kingma & Ba, 2014) with starting learning rate η = 0 .
001 and hyperparameters
β 1 = 0 .
9 and β 2 = 0 .
Note that the learning rate for Adam is not reset before training on a new task.
Next, we give details on number of training epochs and batch sizes speciﬁc for each dataset:
• Split MNIST: 10 epochs/task, batch size
128. • Split FashionMNIST: 30 epochs/task, batch size
128. • Split notMNIST: 50 epochs/task, batch size 128.
• Permuted MNIST: 20 epochs/task, batch size 128. • Split CIFAR-100: 25 epochs/task, batch size 256.
• Split miniImagenet: 1 epoch/task (task 1 trained for 5 epochs as warm up), batch size 32.
Monte Carlo Tree Search: We run RS-MCTS for 100 iterations in all experiments.
The replay schedules used in the reported results on the held-out test sets are from the replay schedule that gave the highest reward on the validation sets.
The exploration constant for UCT in Equation 2 is set to
C = 0 .
1 in all experiments (Chaudhry & Lee, 2018).
Computational Cost: All experiments were performed on one NVIDIA GeForce RTW 2080Ti.
The wall clock time for ETS on Split MNIST was around 1.5 minutes, and RS-MCTS and BFS takes 40 seconds on average to run one iteration, where BFS runs 1050 iterations in total for Split MNIST.
Implementations: We adapted the implementation released by Borsos et al.
(2020) for the memory selection strategies Uniform sampling, k -means clustering, k -center clustering (Nguyen et al., 2017), and Mean-of-Features (Rebufﬁ et al., 2017).
Furthermore, we follow the implementations released by Chaudhry et al.
and Mirzadeh & Ghasemzadeh (2021) for A-GEM (Chaudhry et al., 2018b) and ER-Ring (Chaudhry et al., 2019).
Experimental Settings for with Single Task Replay Memory Experiment
We motivated the need for replay scheduling in continual learning with Figure 1 in Section 1.
This simple experiment was performed on Split MNIST where the replay memory only contains samples from the ﬁrst task, i.e., learning the classes 0/1.
Furthermore, the memory can only be replayed at one point in time and we show the performance on each task when the memory is replayed at different time steps.
We set the memory size to M = 10 samples such that the memory holds 5 samples from both classes.
We use the same network architecture and hyperparameters as described above for Split MNIST.
The
ACC metric above each subﬁgure corresponds to the ACC for training a network with the single task memory replay at different tasks.
We observe that choosing different time points to replay the same memory leads to noticeably different results in the ﬁnal performance, and in this example, the best ﬁnal performance is achieved when the memory is used when learning task 5.
Therefore, we argue that ﬁnding the proper schedule of what tasks to replay at what time in the ﬁxed memory situation can be critical for continual learning.
B R EPLAY S CHEDULING M ONTE C ARLO T REE S EARCH A LGORITHM
We provide pseudo-code in Algorithm 1 outlining the steps for our method Replay Scheduling
Monte Carlo tree search (RS-MCTS) described in the main paper (Section 3.3).
The MCTS procedure selects actions over which task proportions to ﬁll the replay memory with at every task, where the selected task proportions are stored in the replay schedule S .
The schedule is then passed to E VALUATE R EPLAY S CHEDULE ( · ) where the continual learning part executes the training with replay memories ﬁlled according to the schedule.
The reward for the schedule S is the average validation accuracy over all tasks after learning task T , i.e., ACC, which is backpropagated through the tree to update the statistics of the selected nodes.
The schedule S best yielding the best ACC score is returned to be used for evaluation on the held-out test sets.
The function G ET R EPLAY M EMORY ( · ) is the policy for retrieving the replay memory M from the historical data given the task proportion a .
The number of samples per task determined by the task proportions are rounded up or down accordingly to ﬁll M with M replay samples in total.
The function G ET T ASK P ROPORTION ( · ) simply returns the task proportion that is related to given node.
Require:
Tree nodes v 1: T , Datasets D 1: T , Learning rate η Require: Replay memory size M 1: ACC best ← 0 , S best ← () 2: while within computational budget do 3:
S ← () 4: v t , S ← T REE P OLICY ( v 1 , S ) 5: v T , S ← D EFAULT P OLICY ( v t , S ) 6:
ACC ← E VALUATE R EPLAY S CHEDULE ( D 1: T , S, M ) 7:
B ACKPROPAGATE ( v t , ACC ) 8: if ACC > ACC best then 9:
ACC best ← ACC 10:
S best ← S 11: return ACC best , S best 12: function T REE P OLICY ( v t , S ) 13: while v t is non-terminal do 14: if v t not fully expanded then 15: return E XPANSION ( v t , S ) 16: else 17: v t ← B EST C HILD ( v t ) 18:
S .append ( a t ) , where a t ← G ET T ASK P ROPORTION ( v t ) 19: return v t , S 20: function E XPANSION ( v t , S ) 21:
Sample v t +1 uniformly children v t 22:
S .append
( a
1 ) , where a t +1 ← G ET T ASK P ROPORTION ( v t +1 ) 23:
Add new v to node v t 24: return v t +1 , S 25: function B EST C HILD ( v t ) 26: v t +1 = arg max v t +1 ∈ childrenof v max ( Q ( v t +1 ))
+ C (cid:113) 2log( N ( v t ))
N ( v t +1 ) 27: return v t + 28: function D EFAULT P OLICY ( v t , S ) 29: while v t is non-terminal do 30:
Sample
S .append ( a t +1 ) , where a t +1 ← G ET T ASK P ROPORTION ( v t +1 ) 32:
Update v t ← v t + 33: return v t , S 34: function E VALUATE R EPLAY S CHEDULE ( D 1: T , S, M ) 35:
Initialize neural network f θ 36: for t = 1 , . .
, T do 37: a ← S [ t − 1] 38:
M ← G ET R EPLAY M EMORY ( M, a ) 39: for B ∼ D ( train ) t do 40: θ ← SGD ( B
M , θ , η ) 41:
A ( val
1: T ← E VALUATE A CCURACY ( f
, D ( val ) 1: T ) 42:
ACC ← 1 T (cid:80)
Ti =1 A ( val
T,i 43: return ACC 44: function B ACKPROPAGATE ( v
) 45: while is do 46:
N
Q
48: v
C H EURISTIC S CHEDULING B ASELINE
NEW (8z4B)
We implemented a heuristic scheduling baseline to compare against RS-MCTS.
The baseline keeps a validation set for the old tasks and replays the tasks which validation accuracy is below a certain threshold.
We set the threshold in the following way: Let A t,i be the validation accuracy for task t evaluated at time step i .
The best evaluated validation accuracy for task t at time i is given by
A ( best ) t,i = max( { A t, 1 , ..., A t,i } ) .
The condition for replaying task t on the next time step is then
A t,i ≥ τA ( best ) t,i , where τ ∈ [0 , 1] is a ratio controlling how much the current accuracy on task t is allowed to decrease w.r.t.
the best accuracy.
The replay memory is ﬁlled with M/k , where k is the number of tasks that need to be replayed according to their decrease in validation accuracy.
This heuristic scheduling corresponds to the intuition of re-learning when a task has been forgotten.
Training on the current task is performed without replay if the accuracy on all old tasks is above their corresponding threshold.
Grid search for τ .
We performed a coarse-to-ﬁne grid search for the ratio τ on each dataset.
The best value for τ is selected according to the highest mean accuracy on the validation set averaged over 5 seeds.
The validation set consists of 15% of the training data and is the same for RS-MCTS.
We use the same experimental settings as described in Appendix A.
The memory sizes are set to
M = 10 and M = 100 for the 5-task datasets and the 10/20-task datasets respectively, and we apply uniform sampling as the memory selection method.
We provide the ranges for τ that was used on each dataset and put the best value in bold :
• Split MNIST: τ = [0.9, 0.93, 0.95, 0.96 , 0.97, 0.98, 0.99] • Split FashionMNIST: τ
=
[0.9, 0.93, 0.95, 0.96, 0.97 , 0.98, 0.99] • Split notMNIST: τ = [0.9, 0.93, 0.95, 0.96, 0.97, 0.98 , 0.99] • Permuted MNIST: τ = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75 , 0.8, 0.9, 0.95, 0.97, 0.99] • Split CIFAR-100: τ = [0.3, 0.4, 0.45, 0.5 , 0.55, 0.6, 0.65, 0.7, 0.8, 0.9, 0.95, 0.97, 0.99] • Split miniImagenet: τ = [0.5, 0.6, 0.65, 0.7, 0.75 , 0.8, 0.85, 0.9, 0.95, 0.97, 0.99]
Note that we use these values for τ on all experiments with Heuristic for the corresponding datasets.
The performance for this heuristic highly depends on careful tuning for the ratio τ when the memory size or memory selection method changes, as can be seen in in Figure 5 and Table 1.
[Figure]
D A DDITIONAL E XPERIMENTAL R ESULTS
In this section, we bring more insights to the beneﬁts of replay scheduling in Section D.1 as well as provide metrics for catastrophic forgetting in Section D.2.
D. R EPLAY S CHEDULE V ISUALIZATION FOR S PLIT MNIST
NEW (kfup, dhXe)
In Figure 7, we show the progress in test classiﬁcation performance for each task when using ETS and RS-MCTS with memory size M = 10 on Split MNIST.
For comparison, we also show the performance from a network that is ﬁne-tuning on the current task without using replay.
Both
ETS and RS-MCTS overcome catastrophic forgetting to a large degree compared to the ﬁne-tuning network.
Our method RS-MCTS further improves the performance compared to ETS with the same memory, which indicates that learning the time to learn can be more efﬁcient against catastrophic forgetting.
Especially, Task 1 and 2 seems to be the most difﬁcult task to remember since it has the lowest ﬁnal performance using the ﬁne-tuning network.
Both ETS and RS-MCTS manage to retain their performance on Task 1 using replay, however, RS-MCTS remembers Task 2 better than ETS by around 5%.
To bring more insights to this behavior, we have visualized the task proportions of the replay examples using a bubble plot showing the corresponding replay schedule from RS-MCTS in Figure 7(right).
At Task 3 and 4, we see that the schedule ﬁlls the memory with data from Task 2 and discards replaying Task 1.
This helps the network to retain knowledge about Task 2 better than ETS at the cost of forgetting Task 3 slightly when learning Task 4.
This shows that the learned policy has considered the difﬁculty level of different tasks.
At the next task, the RS-MCTS schedule has decided to rehearse Task 3 and reduces replaying Task 2 when learning Task 5.
This behavior is similar to spaced repetition, where increasing the time interval between rehearsals helps memory retention.
We emphasize that even on datasets with few tasks, using learned replay schedules can overcome catastrophic forgetting better than standard ETS approaches.
D. A NALYSIS OF C ATASTROPHIC F ORGETTING
NEW (dhXe)
In this section, we compare the degree of catastrophic forgetting for our method and the baselines.
We use backward transfer (BWT) from Lopez-Paz & Ranzato (2017) for measuring catastrophic forgetting, which is given by
[Equation]
where A t,i is the test accuracy for task t after learning task i .
Table 3 shows the ACC and BWT metrics for the experiments in Section 4.3.
In general, the BWT metric is consistently better when the corresponding ACC is better.
We ﬁnd an exception in Table 3 on Split CIFAR-100 and Split miniImagenet between Ours and Heuristic with uniform selection method, where Heuristic has better BWT while its mean of ACC is slightly lower than ACC for Ours.
Table 4 shows the ACC and BWT metrics for the experiments in Section 4.5, where we see a similar pattern that better ACC yields better BWT.
The BWT of RS-MCTS is on par with the other baselines except on Split CIFARwhere the ACC on our method was a bit lower than the best baselines.
[Table]
report the mean and standard deviation of ACC and BWT, where all results have been averaged over 5 seeds.
RS-MCTS performs on par with the best baselines for both metrics on all datasets except S-CIFAR-
[Table]
E A DDITIONAL F IGURES
Memory usage.
We visualize the memory usage for Permuted MNIST and the 20-task datasets
Split CIFAR-100 and Split miniImagenet in Figure 8 for our method and the baselines used in the experiment in Section 4.5.
Our method uses a ﬁxed memory size of M = 50 samples for replay on all three datasets.
The memory size capacity for our method is reached after learning task 6 and task 11 on the Permuted MNIST and the 20-task datasets respectively, while the baselines continue incrementing their replay memory size.
[Figure]