L EARNING F AST , L EARNING S LOW : A G ENERAL C ONTINUAL L EARNING M ETHOD BASED ON C OMPLEMENTARY L EARNING S YSTEM
A BSTRACT
Humans excel at continually learning from an ever
-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forget
- ting.
The complementary learning system (CLS) theory suggests that the interplay between rapid instance-based learning and slow structured learning in the brain is crucial for accumulating and retaining knowledge.
Here, we propose CLS-
ER
, a novel dual memory experience replay (ER) method which maintains short-term and long-term semantic memories that interact with the episodic memory.
Our method employs an effective replay mechanism whereby new knowledge is acquired while aligning the decision boundaries with the semantic memories.
CLSER does not utilize the task boundaries or make any assumption about the distribution of the data which makes it versatile and suited for “general continual learning”.
Our approach achieves state-of-the-art performance on standard benchmarks as well as more realistic general continual learning settings
I NTRODUCTION
Continual learning (CL) refers to the ability of a learning agent to continuously interact with a dynamic environment and process a stream of information to acquire new knowledge while consolidating and retaining previously obtained knowledge (Parisi et al., 2019).
This ability to continuously learn from a changing environment is a hallmark of intelligence and a critical missing component in our quest towards making our models truly intelligent.
The major challenge towards enabling
CL in deep neural networks (DNNs) is that the continual acquisition of incrementally available information from non-stationary data distributions generally leads to catastrophic forgetting whereby the performance of the model on previously learned tasks drops drastically as it learns new tasks (McCloskey & Cohen, 1989).
Several approaches have been proposed to address the issue of catastrophic forgetting in CL.
These can be broadly categorized into regularization-based methods (Farajtabar et al., 2020; Kirkpatrick et al., 2017; Ritter et al., 2018; Zenke et al., 2017) which penalizes changes in the network weights, network expansion-based methods (Rusu et al., 2016; Yoon et al., 2017) which dedicate a distinct set of network parameters to distinct tasks, and rehearsal-based methods (Chaudhry et al., 2018;
Lopez-Paz & Ranzato, 2017) which maintains a memory buffer and replays samples from previous tasks.
Amongst these, rehearsal-based methods have proven to be more effective in challenging CL tasks (Farquhar & Gal, 2018).
However, an optimal approach for replaying memory samples and constraining the model update to efﬁciently consolidate knowledge remains an open question.
In the brain, the ability to continually acquire, consolidate, and transfer knowledge over time is mediated by a rich set of neurophysiological processing principles (Parisi et al., 2019; Zenke et al., 2017) and multiple memory systems (Hassabis et al., 2017).
In particular, the CLS theory (Kumaran et al., 2016) posits that efﬁcient learning requires two complementary learning systems: the hippocampus exhibits short-term adaptation and rapid learning of episodic information which is then gradually consolidated to the neocortex for slow learning of structured information.
Furthermore, a recent study by Hayes et al.
identiﬁed the missing elements of biological reply in the replay
WorkingModel
Update
Consistency Loss
S e m a n t
i c
M e m o r y PlasticModel
StableModel
E p i s od i c
M e m o r y Reservoir and Update with rates
Data Stream ...
Plastic Model - Fast learning of recentexperiences - Short-termadaptation - Efficient representationof recent tasks
WorkingModel - Memorizationof episodic-like events - Learnsstatisticalstructure of the perceived event
Storage,retrieval, replay
Stable Model - Slow learning of structuralknowledge - Long-term retention - Efficient representationacross the tasks
Storage,retrieval, replay mechanisms employed in DNNs for CL.
They highlight that many existing approaches only focus on modeling the prefrontal cortex directly and do not have a fast learning network which plays a critical role in enabling efﬁcient CL in the brain.
Inspired by these studies, we hypothesize that mimicking the slow and rapid adaptation of information and having an efﬁcient mechanism for incorporating them into the working memory can enable better CL in DNNs.
To this end, we propose a novel dual memory experience replay method based on the complementary learning systems theory in the brain, dubbed as CLS-ER.
In addition to a small episodic memory, our method builds long-term and short-term semantic memories which mimic the rapid and slow adaptation of information (Figure 1).
As the network weights encode the learned representations of the tasks (Krishnan et al., 2019), the semantic memories are maintained by taking the exponential moving average of the working model’s weights to consolidate information across the tasks with varying time windows and frequencies.
The semantic memories interact with the episodic memory to extract consolidated replay activation patterns and enforce a consistency loss on the update of the working model so that new knowledge is acquired while aligning the decision boundary of the working model with the decision boundaries of semantic memories.
This maintains a balance between the plasticity and stability of the model for effective knowledge consolidation.
CLS-ER provides a general CL method that does not utilize the task boundaries or make any strong assumption regarding the distribution of the data and tasks.
We demonstrate the versatility and effectiveness of our method on a wide range of CL benchmark tasks as well as more challenging scenarios which simulate the complexities of CL in the real world.
R ELATED W ORK
The base method for the rehearsal-based approach, Experience Replay (ER) (Riemer et al., 2018) combines the memory samples with the task samples into the training batch.
Several techniques have since been employed on top of ER.
Meta Experience Replay (MER)
(Riemer et al., 2018) considers replay as a meta-learning problem for maximizing the transfer from previous tasks and minimizing the interference.
iCARL (Rebufﬁ et al., 2017) uses the nearest average representation of past exemplars to classify in an incrementally learned representation space.
Gradient Episodic
Memory (GEM) (Lopez-Paz & Ranzato, 2017) formulates optimization constraints on the exemplars in memory.
Gradient-based Sample Selection (GSS) (Aljundi et al., 2019) aims for memory sample diversity in the gradient space and provides a greedy selection approach.
Function Distance
Regularization (FDR) (Benjamin et al., 2018) saves the network response at the task boundaries and adds a consistency loss on top of ER.
Dark Experience Replay (DER++) relaxes this restriction and samples logits during the entire optimization trajectory.
CLS has been used as a source of inspiration for dual memory learning systems in earlier works (French, 1999; Robins, 1993) but they have not been shown to scale to current computer vision tasks (Parisi et al., 2019).
Recently, Rostami et al.
(2019) utilizes a generative model to couple sequential tasks in a latent embedding space.
Kamra et al.
(2017) utilizes two generative models
T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k
After Task
After Task After Task
After Task
After Task98.7 0.
Stable Model
T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98.6 0.
Working Model
T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98.85 0.
Plastic Model in a dual memory architecture.
However, they utilize the task boundaries and generative replay has its own set of challenges as it is difﬁcult to learn a faithful distribution and performs sub-par in comparison to instance-based replay methods on challenging CL settings.
Generally, the inspiration from CLS theory in DNNs has been mostly limited to episodic memory and mimicking the rapid and slow learning mechanism is majorly ignored (Hayes et al., 2021) which we aim to address.
M ETHOD
We ﬁrst provide an overview of the CLS theory for the brain and how we aim to mimic it for DNNs before introducing the main components of our method and the overall formulation.
3. C OMPLEMENTARY L EARNING S YSTEM T HEORY
The CLS theory posits that effective lifelong learning in the brain requires two complementary learning systems.
The hippocampus rapidly encodes novel information as a short-term memory which is subsequently used to transfer and consolidate knowledge in the neocortex which gradually acquires structured knowledge representation as long-term memory through experience replay.
The interplay between the functionality of the hippocampus and neocortex is crucial for concurrently learning efﬁcient representations (for better generalization) and the speciﬁcs of instance-based episodic memory.
3. C OMPLEMENTARY L EARNING S YSTEM BASED E XPERIENCED R EPLAY
Inspired by the CLS theory, we propose a dual memory experience replay method, CLS-ER, which aims to mimic the interplay between fast learning and slow learning mechanisms for enabling effective CL in DNNs.
Our method maintains short-term and long-term semantic memories of the encountered tasks which interact with the episodic memory for replaying the associated neural activities.
The working model is updated so that it acquires new knowledge while aligning its decision boundary with the semantic memories to enable the consolidation of structured knowledge across the tasks.
Figure 1 highlights the parallels between CLS theory and our method.
Semantic Memories: Central to our method is the maintenance of two semantic memories which accumulate and consolidate information over long-term and short-term periods.
As the acquired knowledge of the learned tasks is encoded in the weights of DNNs (Krishnan et al., 2019), we aim to form our semantic memories by accumulating the knowledge encoded in the corresponding weights of the model as it sequentially learns different tasks.
An efﬁcient method for aggregating the weights of a model is provided by Mean Teacher (Tarvainen & Valpola, 2017) which is a knowledge distillation approach that uses an exponential moving average (EMA) of the student’s weights during training as a teacher for semi-supervised learning.
It can also be considered as forming a self-ensemble of the intermediate model states that leads to better internal representations.
We adapt the Mean Teacher approach to build our semantic memories as it provides a computational and memory-efﬁcient method for accumulating knowledge over the tasks.
As CL involves learning tasks sequentially, the model weights at each training step can be considered as a student model specialized for a particular task.
Therefore, averaging the weights during training can be considered as forming an ensemble of task-speciﬁc student models which effectively aggregates information across the tasks and leads to smoother decision boundaries.
CLS-ER builds long-term (stable model) and short-term (plastic model) semantic memories by maintaining two EMA-weighted models over the working model’s weights.
The stable model is updated less frequently with a larger window size so that it retains more information from the earlier tasks while the plastic model is updated more frequently with a smaller window size so that it adapts faster to information from new tasks (Figure 2).
Section D further demonstrates the beneﬁts of employing two semantic memories instead of a single semantic memory.
Episodic Memory: Replay of samples from the previous tasks stored in a small episodic memory is a common approach in CL that has proven to be effective in mitigating catastrophic forgetting.
As we aim to position CLS-ER as a versatile general incremental learning method, we do not utilize the task boundaries or make any strong assumptions about the distribution of the tasks or samples.
Therefore, to maintain a ﬁxed episodic memory buffer, we employ reservoir sampling (Vitter, 1985) which assigns equal probability to each sample in the stream for being represented in the buffer and randomly replaces the existing memory samples (Algorithm 2).
It is a global distribution matching strategy that ensures that at any given time the distribution of samples in the buffer will approximately match the distribution of all the samples seen so far (Isele & Cosgun, 2018).
Consolidation of Information: The key challenge in CL is the consolidation of new information with the previously acquired information.
This requires an effective balance between the stability and plasticity of the model.
Furthermore, the sharp change in decision boundary as a new task is learned makes the consolidation of information over tasks more challenging.
CLS-ER tackles these challenges through a novel dual memory experience replay mechanism.
The long-term and shortterm semantic memories interact with the episodic memory to extract the consolidated activations for the memory samples which are then utilized to constrain the update of the working model so that new knowledge is obtained whilst the decision boundary is aligned with the semantic memories.
This prevents rapid changes in the parameter space as new tasks are learned.
Furthermore, aligning the working model’s decision boundary with the semantic memories serves two goals: (i) helps in retaining and consolidating information and (ii) leads to a smoother adaptation of decision boundary.
3. F ORMULATION
CLS-ER involves training a working model f ( . ; θ w ) on a data stream D sampled from a non-iid distribution.
Two additional EMA-weighted models are maintained as semantic memories: plastic model f ( . ; θ P ) and the stable model f ( . ; θ S ) .
Finally, reservoir sampling (Vitter, 1985) is employed to maintain a small episodic memory M .
At each training step, the working model receives the training batch X b from the data stream and retrieves a random batch of exemplars X m from the episodic memory.
This is then followed by the retrieval of optimal semantic information, i.e. the structural knowledge encoded in the semantic memories which account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks.
The semantic memories are designed so that the plastic model has higher performance on recent tasks whereas the stable model prioritizes retaining information on the older tasks.
Therefore, we would prefer to use the logits from the stable model Z S for older exemplars and the plastic model Z P for recent exemplars.
As CLS-ER is a general incremental learning method, instead of using a hard threshold or task information, we opt for a simple task-agnostic approach of using the performance of the semantic memories on the exemplars as a selection criterion that empirically works well.
For each exemplar, we select the replay logits Z based on which model has the highest softmax score for the ground-truth class (line 5-6 in Algorithm 1).
The selected replay logits from the semantic memories are then used to enforce a consistency loss on the working model so that it does not deviate from the already learned experiences.
Hence, the working model is updated with a combination of the cross-entropy loss on the union of the data stream and episodic memory samples, X , and the consistency loss on the exemplars X m ,
[Equation]
Input: Data stream
, Learning rate η , Consistency
Update rates
Decay
Initialize: θ W = θ P = θ S
M ←− {} 1: while Training do 2: ( X b , Y b ) ∼ D 3: ( X m , Y m ) ∼ M 4: ( X, Y ) = { ( X b , Y b ) , ( X m , Y m ) } 5:
Z P , Z S ←− f ( X m ; θ P ) , f ( X m ; θ S ) (cid:46) Select optimal semantic memory 6:
Z ←− Z P if σ ( Z P ) ( Y m ) >
σ ( Z S )
( Y m ) else Z S 7:
L = L CE ( σ ( f ( X ; θ W )) , Y ) + λ L MSE ( f ( X m ; θ W ) , Z ) (cid:46) Update working model 8: θ W ←− θ W − η ∇ θ W L 9: a, b ∼ U (0 , 1) (cid:46) Update semantic memories 10: θ P ←− α p θ P + (1 − α P )
θ W if a < r P else θ P 11: θ S
←− α S θ S + (1 − α S )
θ W
if b < r S else θ S 12:
M ←− Reservoir ( M , ( X b , Y b ))
(cid:46) Update episodic memory (Algorithm 2) return θ W , θ P , θ S where σ is the softmax function, λ the regularization parameter, and L MSE the mean squared error loss used as consistency term.
After updating the working model, we stochastically update the plastic and stable models with rates r P and r S (note that r P ≥ r S so that the plastic model is updated more frequently).
Using a stochastic rather than a deterministic approach is more biologically plausible (Maass, 2014) which reduces the overlap in the snapshots of the working model and leads to more diversity in semantic memories.
The semantic memories are updated by taking an exponential moving average of the working model’s weights (Tarvainen & Valpola, 2017) with decay parameters α P and α S , respectively.
[Equation]
Note that α P ≤ α S so that the plastic model mimics the rapid adaptation of information while the stable model mimics slow acquisition of structured knowledge.
For inference, we use the stable model as it retains long-term memory across the tasks, consolidates structural knowledge, and learns efﬁcient representations for generalization (Figure 1).
Details of our proposed method are provided in Algorithm 1.
E XPERIMENTAL S ETUP
To ensure a fair comparison of different CL methods under uniform experimental settings, we extended the Mammoth framework (Buzzega et al., 2020a) and unless stated otherwise, we follow the same training scheme (learning rate, batch sizes of incoming data and memory buffer, and the number of training epochs) as them for each of the evaluation settings.
To ﬁnd the optimal hyperparameters for CLS-ER, we run a grid search over λ , α S , α P , r S , and r P on a small validation set.
Sections C.4 and E show that our method is not highly sensitive to the particular choice of hyperparameters and different settings can attain similar performance.
Also, because of the complementary nature of the components, we can often ﬁx a set of parameters (e.g. λ , α S , α P and r S ) and only ﬁnetune the remaining parameters (e.g. r P ) which facilitates hyperparameter tuning signiﬁcantly.
Following Buzzega et al.
(2020a), we employ a fully connected network with two hidden layers, each with 100 ReLU units on all the variants of the MNIST dataset and ResNet-18 (He et al., 2015) network without pretraining for the other datasets.
In all of our settings, we use the Stochastic Gradient Descent optimizer.
We use random horizontal ﬂip and random crop on both the stream and buffer samples for S-CIFAR-10, S-Tiny-ImageNet, and GCIL-CIFAR-100.
The selected hyperparameters for each of the settings are provided in Table S4.
Note that for the vast majority of datasets, we use uniform settings (lr, epochs, batch size, memory batch size, and lambda ) across the different buffer sizes and requires only slight modiﬁcations in the other hyperparameters which shows that our method does not require extensive ﬁnetuning for different memory budgets.
For each of our
[Table]
that the authors were unable to run).
experiments, we ﬁx the order of the classes and report the average and one standard deviation of the mean test accuracy of all the tasks across 10 runs with different initializations.
Section E provides further implementation details.
E MPIRICAL E VALUATION
There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; van de Ven & Tolias, 2019).
It is therefore of utmost importance to conduct an extensive and robust evaluation over different CL settings to gauge the versatility of the method.
Details of the datasets used in each CL setting are provided in Section A.
We compare our method with the state-of-the-art rehearsal-based approaches on various CL settings and memory budgets under uniform experimental settings.
SGD refers to standard training and JOINT provides an upper bound given by training all tasks jointly.
Class Incremental Learning (Class-IL): refers to the CL scenario where new classes are added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks.
Class-IL measures how well the method can learn general representations, accumulate, consolidate, and transfer the acquired knowledge to learn efﬁcient representations and decision boundaries for all the classes seen so far.
Table 1 provides the comparison with six rehearsal-based approaches on Class-IL settings with varying datasets and task length complexities.
CLS-ER provides the highest performance in all of these scenarios.
In particular, as the dataset complexity and number of tasks increase from S-MNIST to S-Tiny-ImageNet, the performance gap between CLS-ER and DER++ increases considerably.
Especially, with a smaller memory budget, CLS-ER is able to retain more information than other methods.
In the most challenging setting, S-Tiny-ImageNet with 200 buffer size, CLS-ER provides a percentage gain of 163% and 104% over the baseline ER and the current state-of-the-art DER++, respectively.
The results demonstrate the capability of CLS-ER to efﬁciently accumulate and retain knowledge over longer sequences under complex and memory restrictive scenarios.
[Table] [Table]
We believe that the performance gains over DER++ highlight a key component of an efﬁcient CL agent: the ability to consolidate previously acquired knowledge.
DER++ fails to account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks.
Therefore, constraining the model to match the sub-optimal logits might hamper the consolidation of knowledge.
This becomes more prominent as the number of classes in each task, the sequence length, and the cross-task resemblance increase.
For instance, for DER++, replaying a sample from
Task-1 when training on S-Tiny-ImageNet Task-10, the reference logit values which are used to enforce the consistency are from a model representation state which has not considered how to distinguish the 20 classes in Task-1 from 80 additional classes which are visually and semantically similar.
It stands to reason that the optimal representation space and subsequently the decision boundaries for the classes in Task-1 would drift considerably when required to distinguish between 80 additional classes as well.
Therefore, the local information provided by the sub-optimal saved logits in DER++ fails to provide the global context required for consolidating knowledge.
CLS-ER, on the other hand, extracts logits from the semantic memories which consolidate knowledge across the tasks, and hence the working model receives more optimal feedback.
Domain Incremental Learning (Domain-IL): refers to the CL scenario where the classes remain the same in subsequent tasks but the input distribution changes.
We consider R-MNIST where each task contains digits rotated by a ﬁxed angle and P-MNIST which applies a ﬁxed random permutation to the pixels for each task.
Table 1 shows that CLS-ER provides generalization gains under both settings, particularly for lower memory budget, and performs on par with DER++ on 5120 buffer size.
We attribute this to the consolidated soft targets from the semantic memories which provide relational information about the classes from a global context compared to the local information in
DER++.
This enables our method to maintain the similarity structure across sequences effectively.
General Incremental Learning (GIL): Class-IL and Domain-IL fail to assimilate the challenges in the real-world setting where the task boundaries are blurry, and classes can reappear and have different distributions.
The CL method has to consider the sample efﬁciency, challenge of imbalanced data, and efﬁcient knowledge transfer in addition to preventing catastrophic forgetting.
We consider two GIL settings: MNIST-360 (Buzzega et al., 2020a) exposes the model to both sharp (changes in class) and smooth (rotation of digits) distribution shifts.
This requires the CL method to tackle the challenges of class-IL as well as domain-IL.
The Generalized Class Incremental Learning (GCIL) (Mi et al., 2020) is the closest to the real-world scenario as it utilizes probabilistic modeling to sample the classes and data distributions in each task.
The number of classes in each task is not ﬁxed, the classes can overlap and the sample size for each class can vary.
Table 2 shows that CLS-ER provides considerable performance gains on the challenging MNIST360, particularly with a low memory budget.
Similarly, Table 3 demonstrates the effectiveness of
CLS-ER on GCIL-CIFAR-100 under both uniform and imbalanced class samples.
Both of these settings involve recurring classes in subsequent sequences which makes the transfer of knowledge from previous occurrences important.
The performance gap between CLS-ER and DER++ in the recurring classes setting alludes to another shortcoming of saving logits from the previous state.
[Figure]
CLS-ER effectively mitigates the bias to the recent tasks and provides a more uniform probability of being predicted for the classes over the tasks even very early ones.
Consider the case where class c appears in sequence (Seq)-1 with 20 samples, and then subsequently in Seq-5 with 200 samples.
In the following sequences, DER++ uses exemplars from class c saved in Seq-1 with sub-optimal logits from the model state which was attained with only 20 samples and fails to take advantage of the better learned representations with additional data in Seq-5.
CLS-ER, on the other hand, is able to take advantage of the additional samples and provide feedback from the improved learned representations.
Furthermore, note that the MNIST-based settings can be considered under the online CL setting (see section A.4) as we only pass through the data once for each task and the performance of CLS-ER on these settings demonstrates its potential as an efﬁcient method for online CL.
M ODEL C HARACTERISTICS
We analyze CLS-ER and provide some insights into the characteristics of the proposed approach which enables it to learn effectively under challenging CL scenarios.
In our subsequent analyses, we compare CLS-ER with the baseline ER and DER++.
6. C ONVERGENCE TO F LATTER M INIMA
Due to the non-convexity of the loss landscape, there can be multiple solutions to the optimization objective, however, the local geometry at the convergence point can affect the generalization of the model.
Solutions that reside in wide valleys instead of narrow crevices generalize better (Chaudhari et al., 2019; Hochreiter & Schmidhuber, 1997; Keskar et al., 2016) as the predictions do not change drastically with small perturbations.
A CL model which converges to ﬂatter minima has more ﬂexibility to explore the neighboring parameter space to optimize on the new task without drastically increasing the loss on the previous tasks.
Following the analysis in Zhang et al.
(2018), we add independent Gaussian noise of increasing strength to the parameters of the trained model and analyze the change in accuracy and loss across the training samples.
Figure 3 shows that CLS-ER is signiﬁcantly less sensitive to perturbations compared to ER and DER++.
CLS-ER also retains performance for a longer period and its performance drops more smoothly.
These results suggest that the fast and slow adaptation of information in CLS-ER can guide the optimization to wider valleys.
6. T ASK P ROBABILITIES
Because of the sequential nature of CL, an implicit bias is induced towards the current task (Wu et al., 2019).
A number of CL methods employ explicit techniques to reduce this bias (Hou et al., 2019; Wu et al., 2019), however, they utilize the task boundaries which is counterproductive for general incremental learning.
We believe that the efﬁcient knowledge consolidation in CLS-ER
[Figure]
ER
ECE
[Figure]
DER
[Figure]
ECE=16.
Gap Outputs
[Figure]
CLS
[Figure]
ER
ECE
[Figure] [Figure]
through the semantic memories can implicitly mitigate the bias towards recent tasks.
We follow the analysis performed in Buzzega et al.
(2020b) to observe the probability of each task being predicted at the end of the training.
For each sample in the test dataset, we take the softmax output and then average the probabilities of the associated classes for each task across the dataset.
We normalize the values and report the probability of each task being predicted.
Figure 3 (right plot) shows that
CLS-ER is able to maintain a more uniform prediction probability across all the tasks over a long sequence.
Figures S3 and S4 shows similar results for other buffer sizes and S-TinyImageNet.
6. M ODEL C ALIBRATION
Model calibration refers to the accuracy with which the scores provided by the model reﬂect its predictive uncertainty.
The class probabilities predicted by DNNs are uncalibrated, often tending towards over-conﬁdence which is detrimental to the reliability of the model’s prediction (Guo et al., 2017).
This is even more pronounced in CL where the models tend to be biassed towards recent tasks.
Following Guo et al.
(2017), we provide the reliability diagrams (model accuracy as a function of its prediction conﬁdence) and the Expected Calibration Error (ECE; a weighted average over the absolute difference between accuracy and conﬁdence).
Figure 4 shows the remarkable ability of
CLS-ER to provide well-calibrated models without the application of any calibration technique.
Note that these characteristics are complementary in nature: convergence to ﬂatter minima allows our method to remain in the vicinity of optimal parameters for previous tasks when adapting to the new task, this leads to more uniform performance across tasks which can improve the task probabilities, and since the model is not too biased towards the current task, the model can provide reliable prediction across the tasks which improve the calibration.
Additional characteristics analyses on different datasets and buffer sizes are provided in Appendix.
We observe that our model’s behavior is consistent across varying datasets and buffer sizes.
C ONCLUSION
We proposed a novel dual memory experience replay method based on the complementary learning systems theory in the brain.
Our method maintains long-term and short-term semantic memories which are utilized to effectively replay the neural activities of the episodic memories and align the decision boundary of the working model for efﬁcient knowledge consolidation.
We demonstrated the effectiveness of our approach on benchmark datasets as well as more challenging general incremental learning scenarios and achieved the new state-of-the-art in the vast majority of the continual learning settings.
We further showed that CLS-ER converges to ﬂatter minima, mitigates the bias towards recent tasks, and provides a well-calibrated high-performance model.
Our strong empirical results motivate further study into mimicking the complementary learning system in the brain more faithfully to enable optimal continual learning in DNNs.
A C ONTINUAL L EARNING S ETTINGS
There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; Shim et al., 2020; van de Ven & Tolias, 2019).
It is therefore of utmost importance to conduct an extensive and robust evaluation to gauge the versatility of the method.
We believe that adhering to the key desiderata as suggested in Farquhar & Gal (2018) would help the CL community immensely in moving towards a robust evaluation of methods.
An experimental protocol that trains the method on a long sequence of tasks where the boundaries between the tasks are not distinct and the tasks themselves are not disjoint and the method does not make sure of task boundaries during training or testing can be considered as adhering to all ﬁve desiderata.
Our work focuses on the aforementioned setting which can be considered as General Incremental Learning (GIL) setting.
Here, we provide a broad categorization of these evaluation protocols which test different aspects of CL.
A. C LASS I NCREMENTAL L EARNING (C LASS -IL)
Class-IL refers to the CL scenario where new classes are added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks.
Class-IL measures how well the method can learn general representations, accumulate, consolidate, and transfer the acquired knowledge to learn efﬁcient representations and decision boundaries for all the classes seen so far.
Following Buzzega et al. (2020a) ; De Lange et al. Zenke et al.
(2017), we consider the common benchmark datasets MNIST (LeCun et al., 1998) ( SMNIST ), CIFAR-10 (Krizhevsky et al., 2009)
( S-CIFAR-10 ) and Tiny-ImageNet (Pouransari & Ghili, 2015) ( S-Tiny-ImageNet ) which are split into 5, 5, and 10 tasks each including 2, 2, andclasses respectively.
These represent Class-IL settings of increasing dataset complexity as well as longer sequences.
While it is an important and challenging benchmark, it assumes that each subsequent task will have the same number of disjoint classes and have uniform samples for each class which is not representative of real-world scenarios.
We do not consider the related Task Increment
Learning (Task-IL) setting as it assumes the availability of task labels at both training and inference which cannot truly be considered as a CL task (Farquhar & Gal, 2018).
A. D OMAIN I NCREMENTAL L EARNING (D OMAIN -IL)
Domain-IL refers to the CL scenario where the classes remain the same in each subsequent task but the input distribution changes.
We consider Rotated-MNIST (Lopez-Paz & Ranzato, 2017)
(R-MNIST) where each task contains digits rotated by a ﬁxed angle between 0 and 180 degrees and
Permuted-MNIST (Kirkpatrick et al., 2016)
(P-MNIST) which applies a ﬁxed random permutation to the pixels for each task.
Though we provide the results for Permuted MNIST for completion, we share the opinion by Farquhar & Gal (2018) that it should not be considered as a benchmark dataset as it violates the cross-task resemblance desiderata and deviates from the goal of continual learning.
A. G ENERAL I NCREMENTAL L EARNING (GIL)
The aforementioned CL scenarios fail to assimilate the challenges in the real world, setting where the task boundaries are blurry and the learning agent must rather learn from a continuous stream of data where classes can reappear and have different data distributions.
The CL method must deal with the issues of sample efﬁciency, imbalanced classes, and efﬁcient transfer of knowledge in addition to preventing catastrophic forgetting.
To test the efﬁcacy of our method in this challenging setting, we consider two GIL evaluation protocols.
MNIST-360 (Buzzega et al., 2020a) models a stream of data which presents batches of two consecutive MNIST images with each sample rotated at an increasing angle and the sequence is repeated three times.
This exposes the model to both a sharp distribution shift when the class changes and a smooth rotational distribution shift.
However, the number of classes in each task and the samples are uniform.
The Generalized Class Incremental
Learning (GCIL)
(Mi et al., 2020) utilizes probabilistic modeling to sample the classes and data distributions in each task.
Hence, the number of classes in each task is not ﬁxed, the classes can overlap and the sample size for each class can vary.
Following Mi et al.
(2020), we use GCIL on CIFAR-100 (Krizhevsky et al., 2009) dataset ( GCIL-CIFAR-100 ), set the number of samples and maximum number of classes per task to 1000 and 50 respectively, number of tasks to 20, and evaluate on both uniform and longtail (imbalanced) sample distribution.
A. O NLINE C ONTINUAL L EARNING
Online continual learning refers to the challenging scenario where a stream of samples is only seen once and is non-iid (Mai et al., 2022; Aljundi et al., 2019).
The common approach in the literature is to use the single-epoch protocol where the network is trained on each task in the sequence for only one epoch and there are no additional passages over data.
As we aim to position CLS-ER as a general incremental learning method, we are also interested in the online continual learning setting.
However, similar to Buzzega et al.
(2020a), we also believe that the dataset complexity needs to be considered when setting the number of epochs to disentangle the effect of catastrophic forgetting from underﬁtting and share their suggestion that future CL works should strive for realism by designing experimental settings which are in line with the guidelines of General Continual Learning (Farquhar & Gal, 2018) which is the goal of our study rather than adopting the single-epoch protocol.
For the MNIST-based settings, we use only one epoch per task as it is sufﬁcient for the SGD baseline to learn the single task well.
And for the more complex settings, we increase the number of epochs: 50 epochs for Sequential CIFAR-10 and Sequential Tiny-ImageNet and 100 epochs for
GCIL-CIFAR-100.
We would also like to emphasize that the experiments on MNIST based settings (S-MNIST, RMNIST, P-MNIST, and MNIST-360) can be considered as online continual learning settings as we only train the network for 1 epoch, and thereby the model only sees the data for each task once.
CLSER’s performance in these settings demonstrates its potential for the challenging online continual learning setting.
B R ESERVOIR S AMPLING
Here, we provide the algorithm for the Reservoir Sampling for maintaining a ﬁxed-size memory buffer.
Reservoir sampling takes in a data stream of unknown length and assigns equal probability to each sample for being represented in the memory buffer ( M ) with a ﬁxed budget size ( B ).
Sampling and replacement are done at random and no priority is assigned to the samples being added or replaced from the memory buffer.
C A DDITIONAL R ESULTS
In this section, we provide additional experimental results and analysis of the behavior of the model.
C. CLS-ER C OMPONENTS P ERFORMANCE
CLS-ER involves the interplay between the working model and the two semantic memories: the plastic and stable models.
While we use the stable model for ﬁnal inference, here we provide the performance of each of these individual components to provide further insights into the workings of our method.
Table S1 shows the corresponding performance of the working model and plastic model for each of our experimental settings.
We can see that the stable model can effectively consolidate
[Table]
knowledge across the tasks and therefore provide the highest mean performance for the vast majority of the settings.
Figures S1 and S2 further shows how the task-wise performance (on test set) of each of the component varies as subsequent tasks are learned.
The stable model retains the performance on previous tasks while the plastic model adapts better to the recent task.
Both these models provide feedback to the working model which in turn improves the plastic and stable model.
C. T ASK P ROBABILITIES
To test the effectiveness of our method in mitigating the bias towards recent tasks, we provide the task probabilities of the models trained with different buffer sizes on S-CIFAR-10 and S-TinyImageNet.
Figures S3 and S4 show that CLS-ER consistently achieves more uniform task probabilities compared to ER and DER++ and effectively mitigates the bias towards the last task.
C. M ODEL C ALIBRATION
To further test the consistency of CLS-ER in providing well-calibrated models and the impact of the buffer size, we evaluate the calibration of models trained with different buffer sizes on S-CIFARand S-Tiny-ImageNet.
Figures S5 and S6 show that CLS-ER consistently provides better calibrated models compared to ER and DER++.
Remarkably, for both the datasets, on lower buffer sizes, the difference in Expected Calibration Error (ECE) is considerable.
This demonstrates the capability of
CLS-ER to train high-performance and reliable models under challenging conditions.
C. E FFECT OF H YPERPARAMETERS
The interaction between the three components of CLS-ER is complementary.
Table S3 shows how the performance of each component is affected under different hyperparameter settings.
We can draw the following conclusions from the results.
The performance improvement in the plastic and stable model is reﬂected in the working model and the best performance is seen in cases where both the semantic memories are performing well (albeit the focus on tasks is different).
This highlights
After Task After Task
After Task After Task
After Task
B u
ff e r 98.
Stable Model 98.
Working Model 98.
Plastic Model
After Task
After Task After Task
After Task After Task
B u
ff e r 98.
[Table]
the crucial role of both memories in enabling CLS-ER to learn efﬁciently.
For a ﬁxed r S value, the ﬁnal performance of the stable model is affected considerably by the performance of the plastic model.
The method is not highly sensitive to the particular choice of hyperparameters as different settings can attain similar performance.
Because of the complementary nature of the components, we can often ﬁx a set of parameters (e.g. λ , α S , α S and r S ) and only ﬁnetune the remaining parameters (e.g. r (cid:48) P ) which facilitates hyperparameter tuning signiﬁcantly.
D C OMPARISON WITH A S INGLE S EMANTIC M EMORY
CLS-ER employs two semantic memories as we aim to mimic the fast and slow learning mechanisms in the hippocampus and neocortex respectively.
Here we compare our method with a single semantic memory (Mean-ER) and Table S2 shows that while it still performs admirably compared to the other CL methods, the dual semantic memories in CLS-ER provides additional performance gains especially on the complex datasets under the challenging lower memory buffer settings and has a much lower variance.
We attribute this to the failure of Mean-ER in maintaining the performance on both the recent and earlier tasks together
i.e there is an inherent trade-off as tuning the semantic memory to adapt to the recent changes comes at the cost of performance on earlier tasks and vice versa.
CLS-ER efﬁciently tackles this trade-off by maintaining two specialized long-term and shortterm memories.
The performance of Mean-ER, however, provides further evidence for the beneﬁts of using consolidated information for memory replay.
Note that for a fair comparison, we use the same hyperparameter search space as CLS-ER for ﬁnding the optimal parameters for Mean-ER and report the average and 1 std of 10 runs with different initializations using the best parameters for each setting.
Table S6 provides the chosen hyperparameters.
For inference, similar to CLS-ER, we use the EMA-weighted model (semantic memory) for
Mean-ER.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
E T RAINING AND I MPLEMENTATION D ETAILS
For a fair comparison
[Figure]
we aim to keep the experimental settings close to the current state-of-theart DER++ (Buzzega et al., 2020a) as much as possible to disassociate the effect of the training
[Figure] [Figure]
schedule.
We use the same optimizer, the number of epochs, batch size
[Figure]
and memory batch size as
DER++.
For S-Tiny-ImageNet, we reduce the number of epochs to 50 from 100 used by DER++ as our method can learn efﬁciently with fewer epochs, and quickly acquiring new knowledge is preferred for CL.
Similar to DER++, we ﬁnetune the memory batch size for S-MNIST and MNIST360.
We select the hyperparameters for each of the experimental setting using a small validation set, α S , α P ∈ (0 . 99 , 0 . 999) , r S , r P ∈ (0 , 1] , λ ∈ (0 , 2] .
Table S4 provides the hyperparameters used for each of the experimental settings.
Note that for the vast majority of datasets, we use uniform settings (lr, epochs, batch size, memory batch size, and lambda ) across the different buffer sizes and
B u
ff e r
A cc u r a c y
ER
ECE=74.
Gap Outputs
DER++
ECE=26.
Gap Outputs
CLS-ER
ECE=5.
Gap Outputs 0.
B u
ff e r
A cc u r a c y
ECE=69.
Gap Outputs
ECE=14.
Gap Outputs
ECE=8.
Gap Outputs 0.
1.0 Confidence 0.
B u
ff e r
A cc u r a c y
ECE=51.
Gap Outputs 0.
1.0 Confidence
ECE=13.
Gap Outputs 0.
1.0 Confidence
ECE=12.
Gap Outputs
Buffer
Method
Class-IL
Domain-IL
S-MNIST
S-CIFARS-Tiny-ImageNet
R-MNIST
P-MNIST
[Table]
JOINT
[Table] [Table]
20±
[Table]
59.99±
[Table]
95.76±
[Table] [Table]
33±
[Table]
[Table] [Table]
60±
[Table] [Table]
62±
[Table] [Table]
92±
[Table] [Table]
66±
[Table] [Table]
70±
[Table]
Mean-ER 88.32±
[Table]
61.88±2.
17.68±1.
92.10±1.
83.28±0.
CLS
[Table]
requires only slight modiﬁcations in the other hyperparameters which shows that our method does not require extensive ﬁnetuning for different memory budgets.
E. GCIL-CIFAR-
To test our method under challenging GIL settings that better simulate the challenges of CL in the real world
[Table]
we incorporate the GCIL setting from the code provided by Mi et al.
(2020) with the continual dataset template class in the mammoth framework.
We set the number of phases (length of task sequences) to 20, with the total number of samples in each phase set to 1000 and the maximum number of classes in each phase set to 50.
We evaluate on both uniform and longtail (imbalanced) data distributions.
Since GCIL involves the probabilistic sampling of the classes and their samples in each phase, the random seed determines the complexity of the GCIL setting.
We use 10 different identical random seeds for our experiments so that each method is evaluated under identical settings and the difference between longtail and uniform is only the data distribution between the classes in each phase.
For each of our method, we use identical training scheme (lr=0.1, epochs=100, batch size=32 and memory batch size=32).
For DER++, as per the authorts suggestion, we performed hyperparameter search over α ∈ [0 . 2 , 0 . 3] and beta ∈ [0 .
with step size of 0.1.
Table S5 provides the parameters chosen for each of the method under the different settings.
E. P ERTURBATION A NALYSIS
For the perturbation analysis, we used the code and checkpoints provided by Buzzega et al.
(2020a) for DER++ and ER.
We would like to express our gratitude to the authors for their support and for making the mammoth framework available for the research community which provides a framework for a fair comparison of different CL methods under uniform experimental conditions.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
63.74±1.
71.01±1.
68.46±1.
60.96±1.
68.31±1.
70.05±2.
63.06±1.
69.90±2.
69.57±1.
61.25±1.
69.36±1.
68.99±2.
61.61±2.
68.81±2.
71.21±0.
63.08±0.
70.99±0.
71.26±1.
62.33±0.
71.03±1.
69.00±0.
61.38±0.
68.69±0.
70.19±1.
61.39±2.
69.81±1.
73.72±0.
62.07±0.
70.18±0.
71.60±2.
61.11±2.
69.15±1.
74.18±0.
63.32±0.
71.08±2.
74.90±0.
62.35±2.
71.58±0.
74.52±1.
62.59±2.
70.90±2.
75.27±1.
62.00±1.
71.27±1.
74.61±0.
63.47±1.
70.49±0.
76.03±0.
63.63±1.
71.42±1.
72.59±1.
61.81±1.
72.02±1.
[Table]
[Table] [Table]
Uniform
CLS-ER λ = 0 .
15 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
λ
= 0 .
15 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
λ
= 0 .
15 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
DER++ α = 0 .
2 , β = 0 .
α = 0 .
2 , β = 0 .
α = 0 .
3 , β = 0 .
Longtail
CLS-ER λ = 0 .
1 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
λ
= 0 .
1 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
λ
= 0 .
1 , α S = 0 .
999 , α P = 0 .
999 , r S
= 0 .
6 , r P = 0 .
DER++ α = 0 .
2 , β = 0 .
α = 0 .
2 , β = 0 .
α = 0 .
3 , β = 0 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]