Original: E RROR S ENSITIVITY M ODULATION BASED E XPERIENCE R EPLAY : M ITIGATING A BRUPT R EPRESENTATION D RIFT IN C ONTINUAL L EARNING
Revised: E RROR S ENSITIVITY M ODULATION BASED E XPERIENCE R EPLAY : M ITIGATING A BRUPT R EPRESENTATION D RIFT IN C ONTINUAL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Humans excel at lifelong learning, as the brain has evolved to be robust to distribution shifts and noise in our ever-changing environment.
Revised: Humans excel at lifelong learning, as the brain has evolved to be robust to distribution shifts and noise in our ever-changing environment.

Original: Deep neural networks (DNNs), however, exhibit catastrophic forgetting and the learned representations drift drastically as they encounter a new task.
Revised: Deep neural networks (DNNs), however, exhibit catastrophic forgetting and the learned representations drift drastically as they encounter a new task.

Original: This alludes to a different errorbased learning mechanism in the brain.
Revised: This alludes to a different errorbased learning mechanism in the brain.

Original: Unlike DNNs, where learning scales linearly with the magnitude of the error, the sensitivity to errors in the brain decreases as a function of their magnitude.
Revised: Unlike DNNs, where learning scales linearly with the magnitude of the error, the sensitivity to errors in the brain decreases as a function of their magnitude.

Original: To this end, we propose ESMER which employs a principled mechanism to modulate error sensitivity in a dual-memory rehearsalbased system.
Revised: To this end, we propose ESMER which employs a principled mechanism to modulate error sensitivity in a dual-memory rehearsalbased system.

Original: Concretely, it maintains a memory of past errors and utilizes it to modify the learning dynamics so that the model learns more from small consistent errors compared to large sudden errors.
Revised: Concretely, it maintains a memory of past errors and uses it to modify the learning dynamics so that the model learns more from small consistent errors compared to large sudden errors.

Original: We also propose Error-Sensitive Reservoir Sampling to maintain episodic memory, which leverages the error history to pre-select low-loss samples as candidates for the buffer, which are better suited for retaining information.
Revised: We also propose Error-Sensitive Reservoir Sampling to maintain episodic memory, which leverages the error history to pre-select low-loss samples as candidates for the buffer, which are better suited for retaining information.

Original: Empirical results show that ESMER effectively reduces forgetting and abrupt drift in representations at the task boundary by gradually adapting to the new task while consolidating knowledge.
Revised: Empirical results show that ESMER effectively reduces forgetting and abrupt drift in representations at the task boundary by gradually adapting to the new task while consolidating knowledge.

Original: Remarkably, it also enables the model to learn under high levels of label noise, which is ubiquitous in real-world  streams.
Revised: Remarkably, it also enables the model to learn under high levels of label noise, which is ubiquitous in real-world data streams.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The human brain has evolved to engage with and learn from an ever-changing and noisy environment, enabling humans to excel at lifelong learning.
Revised: The human brain has evolved to engage with and learn from an ever-changing and noisy environment, enabling humans to excel at lifelong learning.

Original: This requires it to be robust to varying degrees of distribution shifts and noise to acquire, consolidate, and transfer knowledge under uncertainty.
Revised: This requires it to be robust to varying degrees of distribution shifts and noise to acquire, consolidate, and transfer knowledge under uncertainty.

Original: DNNs, on the other hand, are inherently designed for batch learning from a static data distribution and therefore exhibit catastrophic forgetting (McCloskey & Cohen, 1989) of previous tasks when learning tasks sequentially from a continuous stream of data.
Revised: DNNs, on the other hand, are inherently designed for batch learning from a static data distribution and therefore exhibit catastrophic forgetting (McCloskey & Cohen, 1989) of previous tasks when learning tasks sequentially from a continuous stream of data.

Original: The significant gap between the lifelong learning capabilities of humans and DNNs suggests that the brain relies on fundamentally different error-based learning mechanisms.
Revised: The significant gap between the lifelong learning capabilities of humans and DNNs suggests that the brain relies on fundamentally different error-based learning mechanisms.

Original: Among the different approaches to enabling continual learning (CL) in DNNs (Parisi et al., 2019), methods inspired by replay of past activations in the brain have shown promise in reducing forgetting in challenging and more realistic scenarios (Hayes et al., 2021; Farquhar & Gal, 2018; van de Ven & Tolias, 2019).
Revised: Among the different approaches to enabling continual learning (CL) in DNNs (Parisi et al., 2019), methods inspired by replay of past activations in the brain have shown promise in reducing forgetting in challenging and more realistic scenarios (Hayes et al., 2021; Farquhar & Gal, 2018; van de Ven & Tolias, 2019).

Original: They, however, struggle to approximate the joint distribution of tasks with a small buffer, and the model may undergo a drastic drift in representations when there is a considerable distribution shift, leading to forgetting.
Revised: They, however, struggle to approximate the joint distribution of tasks with a small buffer, and the model may undergo a drastic drift in representations when there is a considerable distribution shift, leading to forgetting.

Original: In particular, when a new set of classes is introduced, the new samples are poorly dispersed in the representation space, and initial model updates significantly perturb the representations of previously learned classes (Caccia et al., 2021).
Revised: In particular, when a new set of classes is introduced, the new samples are poorly dispersed in the representation space, and initial model updates significantly perturb the representations of previously learned classes (Caccia et al., 2021).

Original: This is even more pronounced in the lower buffer regime, where it is increasingly challenging for the model to recover from the initial disruption.
Revised: This is even more pronounced in the lower buffer regime, where it is increasingly challenging for the model to recover from the initial disruption.

Original: Therefore, it is critical for a CL agent to mitigate the abrupt drift in representations and gradually adapt to the new task.
Revised: Therefore, it is critical for a CL agent to mitigate the abrupt drift in representations and gradually adapt to the new task.

Original: To this end, we look deeper into the dynamics of error-based learning in the brain.
Revised: To this end, we look deeper into the dynamics of error-based learning in the brain.

Original: Evidence suggests that different characteristics of the error, including its size, affect how the learning process occurs  in the brain (Criscimagna-Hemminger et al., 2010).
Revised: Evidence suggests that different characteristics of the error, including its size, affect how the learning process occurs [Figure] in the brain (Criscimagna-Hemminger et al., 2010).

Original: In particular, sensitivity to errors decreases as a function of their magnitude, causing the brain to learn more from small errors compared to large errors (Marko et al., 2012; Castro et al., 2014).
Revised: In particular, sensitivity to errors decreases as a function of their magnitude, causing the brain to learn more from small errors compared to large errors (Marko et al., 2012; Castro et al., 2014).

Original: This sensitivity is modulated through a principled mechanism that takes into account the history of past errors (Herzfeld et al., 2014) which suggests that the brain maintains an additional memory of errors.
Revised: This sensitivity is modulated through a principled mechanism that takes into account the history of past errors (Herzfeld et al., 2014) which suggests that the brain maintains an additional memory of errors.

Original: The robustness of the brain to high degrees of distribution shifts and its proficiency in learning under uncertainty and noise may be attributed to the principled modulation of error sensitivity and the consequent learning from low errors.
Revised: The robustness of the brain to high degrees of distribution shifts and its proficiency in learning under uncertainty and noise may be attributed to the principled modulation of error sensitivity and the consequent learning from low errors.

Original: DNNs, on the other hand, lack any mechanism to modulate the error sensitivity and learning scales linearly with the error size.
Revised: DNNs, on the other hand, lack any mechanism to modulate the error sensitivity and learning scales linearly with the error size.

Original: This is particularly troublesome for CL, where abrupt distribution shifts initially cause a considerable spike in errors.
Revised: This is particularly troublesome for CL, where abrupt distribution shifts initially cause a considerable spike in errors.

Original: These significantly larger errors associated with the samples from unobserved classes dominate the gradient updates and cause disruption of previously learned representations, especially in the absence of sufficient memory samples.
Revised: These significantly larger errors associated with the samples from unobserved classes dominate the gradient updates and cause disruption of previously learned representations, especially in the absence of sufficient memory samples.

Original: To this end, we propose an Error Sensitivity Modulation based Experience Replaymethod (ESMER)  that employs a principled mechanism to modulatethe sensitivity to errors based on its consistency withthe memory of past errors (Figure 1).
Revised: To this end, we propose an Error Sensitivity Modulation based Experience Replay (ESMER) method that employs a principled mechanism to modulate sensitivity to errors based on its consistency with memory of past errors (Figure 1).

Original: Concretely, our method maintains a memory of errors along the training trajectory and utilizes it to adjust the contribution of each incoming sample to learning based on how far they are from the mean statistics of error memory.
Revised: Concretely, our method maintains a memory of errors along the training trajectory and utilizes it to adjust the contribution of each incoming sample to learning based on how far they are from the mean statistics of error memory.

Original: This allows the model to learn more from small consistent errors compared to large sudden errors, thus gradually adapting to the new task and mitigating the abrupt drift in representations at the task boundary.
Revised: This allows the model to learn more from small consistent errors compared to large sudden errors, thus gradually adapting to the new task and mitigating the abrupt drift in representations at the task boundary.

Revised: To keep the error memory stable, task boundary information is utilized to prevent sudden changes.

Original: Additionally, we propose an Error-Sensitive Reservoir Sampling approach for maintaining the buffer, which utilizes the error memory to pre-select low-loss samples from the current batch as candidates for being represented in the buffer.
Revised: Additionally, we propose an ErrorSensitive Reservoir Sampling approach for maintaining the buffer, which utilizes the error memory to pre-select low-loss samples from the current batch as candidates for being represented in the buffer.

Original: It ensures that only the incoming samples that have been well learned are added to the buffer that are better suited to retain information and do not cause representation drift when replaying them.
Revised: It ensures that only the incoming samples that have been well learned are added to the buffer that are better suited to retain information and do not cause representation drift when replaying them.

Original: The proposed sampling approach also ensures higher-quality representative samples in memory by filtering out outliers and noisy labels, which can degrade performance.
Revised: The proposed sampling approach also ensures higher-quality representative samples in memory by filtering out outliers and noisy labels, which can degrade performance.

Original: Another salient component of the learning machinery of the brain is the efficient utilization of multiple memory systems that operate on different timescales (Hassabis et al., 2017; Kumaran et al., 2016).
Revised: Another salient component of the learning machinery of the brain is the efficient use of multiple memory systems that operate on different timescales (Hassabis et al., 2017; Kumaran et al., 2016).

Original: Furthermore, replay of previous neural activation patterns is considered to facilitate memory formation and consolidation (Walker & Stickgold, 2004).
Revised: Furthermore, replay of previous neural activation patterns is considered to facilitate memory formation and consolidation (Walker & Stickgold, 2004).

Original: These components may play a role in facilitating lifelong learning in the brain and addressing the challenges of distribution shifts.
Revised: These components may play a role in facilitating lifelong learning in the brain and addressing the challenges of distribution shifts.

Original: Therefore, ESMER also employs experience replay in a dual memory system.
Revised: Therefore, ESMER also employs experience replay in a dual memory system.

Original: In addition to episodic memory, we maintain a semantic memory, which gradually aggregates the knowledge encoded in the weights of the working model and builds consolidated representations, which can generalize across the tasks.
Revised: In addition to episodic memory, we main- tain a semantic memory, which gradually aggregates the knowledge encoded in the weights of the working model and builds consolidated representations, which can generalize across the tasks.

Original: The two memories interact to enforce consistency in the functional space and encourage the model to adapt its representations and decision boundary to the new classes while preserving the relational structure of previous classes.
Revised: The two memories interact to enforce consistency in the functional space and encourage the model to adapt its representations and decision boundary to the new classes while preserving the relational structure of previous classes.

Original: The gradual aggregation of knowledge in semantic memory and the enforced relational consistency further aid in mitigating the abrupt change in representations.
Revised: The gradual aggregation of knowledge in semantic memory and the enforced relational consistency further aid in mitigating the abrupt change in representations.

Original: We extensively evaluate our method against state-of-the-art rehearsal-based approaches under various CL settings, which simulate different complexities in the real world including distribution shift, recurring classes, data imbalance, and varying degrees of noisy labels.
Revised: We extensively evaluate our method against state-of-the-art rehearsal-based approaches under various CL settings, which simulate different complexities in the real world including distribution shift, recurring classes, data imbalance, and varying degrees of noisy labels.

Original: ESMER provides considerable performance improvement in all the scenarios, especially with a lower memory budget, which demonstrates the versatility of our approach.
Revised: ESMER provides considerable performance improvement in all the scenarios, especially with a lower memory budget, which demonstrates the versatility of our approach.

Original: On the challenging Seq-CIFAR10 with 200 buffer size and 50% label corruption, ESMER provides more than 116% gain compared to the baseline ER method.
Revised: On the challenging Seq-CIFAR10 with 200 buffer size and 50% label noise, ESMER provides more than 116% gain compared to the baseline ER method.

Original: Furthermore, we show that it mitigates the abrupt representation drift, is more robust to label noise, and reduces the task recency bias, which leads to uniform and stable performance across tasks.
Revised: Furthermore, we show that it mitigates the abrupt representation drift, is more robust to label noise, and reduces the task recency bias, which leads to uniform and stable performance across tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Different approaches have been proposed to address the problem of catastrophic forgetting in
Revised: Different approaches have been proposed to address the problem of catastrophic forgetting in

Original: DNNs (De Lange et al., 2021).
Revised: DNNs (De Lange et al., 2021).

Original: Among them, rehearsal-based methods (Hayes et al., 2021) that use episodic memory for continual rehearsal of samples from previous tasks have shown promise in challenging CL scenarios (Farquhar & Gal, 2018).
Revised: Among them, rehearsal-based methods (Hayes et al., 2021) that use episodic memory for continual rehearsal of samples from previous tasks have shown promise in challenging CL scenarios (Farquhar & Gal, 2018).

Original: ER (Riemer et al., 2018) performs interleaved training of the new task and  memory sample to approximate the joint distribution of tasks.
Revised: The base Experience Replay (ER; Riemer et al. (2018)) performs interleaved training of the new task and the memory sample to approximate the joint distribution of tasks.

Original: DER++ (Buzzega et al., 2020) applies an additional distillation loss by storing the output logits together with the samples and enforcing consistency in the output space.
Revised: Dark Experience Replay (DER++; Buzzega et al. (2020)) applies an additional distillation loss by storing the output logits together with the samples and enforcing consistency in the output space.

Original: Although promising, these [Figure] Episodic
Revised: Although promising, these methods struggle to approximate the joint distribution of tasks.

Original: Memory Memory of
Revised: Caccia et al.

Original: [Figure]
Revised: (2021) show that the task transition causes a drastic drift in the learned representations of the previous classes seen and propose an asymmetric cross-entropy loss (ER-ACE) on the incoming samples that only considers logits of the new classes. However, an optimal approach to replaying and constraining the model update to retain knowledge and mitigate representation drift remains an open question.

Original: Another promising direction is the utilization of multiple memory systems (Wang et al., 2022; Pham et al., 2021).
Revised: Another promising direction is the utilization of multiple memory systems (Wang et al., 2022; Pham et al., 2021).

Original: Notably, CLS-ER (Arani et al., 2021) mimics the interplay between fast and slow learning systems by maintaining two semantic memories that aggregate the weights of the model at different rates using an exponential moving average.
Revised: Notably, CLS-ER (Arani et al., 2021) mimics the interplay between fast and slow learning systems by maintaining two semantic memories that aggregate the weights of the model at different rates using an exponential moving average.

Original: It is still unclear, however, what the optimal approach is to build and utilize multiple memory systems.
Revised: It is still unclear, however, what the optimal approach is to build and utilize multiple memory systems.

Original: Our method focuses on experience replay in a dual memory architecture and demonstrates the benefits of mimicking error sensitivity modulation in the brain.
Revised: Our method focuses on experience replay in a dual-memory architecture and demonstrates the benefits of mimicking error sensitivity modulation in the brain.

Original: We also advocate for a more comprehensive evaluation that covers the different challenges involved in lifelong learning in the real world.
Revised: We also advocate for a more comprehensive evaluation that covers the different challenges involved in lifelong learning in the real world.

Original: In particular, we also evaluate on noisy
Revised: In particular, we also evaluate on noisy

Original: CL setting in which the agent has to learn continuously from a noisy data stream.
Revised: CL setting in which the agent has to learn continuously from a noisy data stream.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We first motivate our study with an overview of the issue of abrupt representation drift in DNNs in the CL setting, followed by inspiration from the learning dynamics in the brain.
Revised: We first motivate our study with an overview of the issue of abrupt representation drift in DNNs in the CL setting, followed by inspiration from the learning dynamics in the brain.

Original: Finally, we present the details of the different components of our method.
Revised: Finally, we present the details of the different components of our method.

Original: 3. A BRUPT REPRESENTATION DRIFT IN DNN S
Revised: 3. A BRUPT REPRESENTATION DRIFT IN DNN S

Original: The sequential learning of tasks in CL potentially exposes the model to varying degrees of distribution shifts in the input and output space.
Revised: The sequential learning of tasks in CL potentially exposes the model to varying degrees of distribution shifts in the input and output space.

Original: In particular, the introduction of previously unobserved classes at the task boundary may lead to an abrupt representation drift, leading to forgetting.
Revised: In particular, the introduction of previously unobserved classes at the task boundary may lead to an abrupt representation drift, leading to forgetting.

Original: Caccia et al.
Revised: Caccia et al.

Original: (2021) show that the samples of the new classes are poorly dispersed and lie near and within the representation space of the previous classes.
Revised: (2021) show that the samples of the new classes are poorly dispersed and lie near and within the representation space of the previous classes.

Original: Therefore, the initial parameter updates cause a significant perturbation of the representations of the previous classes.
Revised: Therefore, the initial parameter updates cause a significant perturbation of the representations of the previous classes.

Original: This is exacerbated by the inherent imbalance between the samples of the new classes in the current task and the stored samples of previous classes in memory, especially in the lower buffer size regime.
Revised: This is exacerbated by the inherent imbalance between the samples of the new classes in the current task and the stored samples of previous classes in memory, especially in the lower buffer size regime.

Original: Caccia et al.
Revised: Caccia et al.

Original: (2021) further show that while the model is able to recover somewhat from the disruptive parameter updates at the task transition with larger buffer sizes, the model fails to recover when the buffer size is small.
Revised: (2021) further show that while the model is able to recover somewhat from the disruptive parameter updates at the task transition with larger buffer sizes, the model fails to recover when the buffer size is small.

Original: Moreover, learning scales linearly with the size of the error in standard training.
Revised: Moreover, learning scales linearly with the size of the error in standard training.

Original: Hence, the considerably higher cross-entropy loss from the unobserved classes dominates learning and biases the gradients towards the new classes.
Revised: Hence, the considerably higher cross-entropy loss from the unobserved classes dominates learning and biases the gradients towards the new classes.

Original: These insights suggest that there is a need for a different learning paradigm tailored for CL that is more robust to abrupt distribution shifts.
Revised: These insights suggest that there is a need for a different learning paradigm tailored for CL that is more robust to abrupt distribution shifts.

Original: 3. I NSPIRATION FROM THE BRAIN
Revised: 3. I NSPIRATION FROM THE BRAIN

Original: The human brain, on the other hand, has evolved to be robust to distribution shifts and to continually learn and interact with a dynamic environment.
Revised: The human brain, on the other hand, has evolved to be robust to distribution shifts and to continually learn and interact with a dynamic environment.

Original: This is enabled by a complex set of mechanisms and interactions of multiple memory systems.
Revised: This is enabled by a complex set of mechanisms and interactions of multiple memory systems.

Original: In particular, we probe the dynamics of error-based learning in the brain to draw some insights.
Revised: In particular, we probe the dynamics of error-based learning in the brain to draw some insights.

Original: Unlike DNNs where learning scales linearly with error size, evidence suggests that the brain modulates its sensitivity to error as a function of error magnitude, so it learns more from small consistent errors compared to large errors (Marko et al., 2012; Castro et al., 2014; Smith & Shadmehr, 2004).
Revised: Unlike DNNs where learning scales linearly with error size, evidence suggests that the brain modulates its sensitivity to error as a function of error magnitude, so it learns more from small consistent errors compared to large errors (Marko et al., 2012; Castro et al., 2014; Smith & Shadmehr, 2004).

Original: To enable such a mechanism, the brain appears to maintain an additional memory of errors that informs error sensitivity modulation (Herzfeld et al., 2014).
Revised: To enable such a mechanism, the brain appears to maintain an additional memory of errors that informs error sensitivity modulation (Herzfeld et al., 2014).

Original: Another salient element of the learning machinery of the brain is the efficient use of multiple memory systems that operate on different timescales (Hassabis et al., 2017; Kumaran et al., 2016).
Revised: Another salient element of the learning machinery of the brain is the efficient use of multiple memory systems that operate on different timescales (Hassabis et al., 2017; Kumaran et al., 2016).

Original: Moreover, replay of previous neural activation patterns is considered to facilitate memory formation and consolidation (Walker & Stickgold, 2004; McClelland et al., 1995).
Revised: Moreover, replay of previous neural activation patterns is considered to facilitate memory formation and consolidation (Walker & Stickgold, 2004; McClelland et al., 1995).

Original: These elements could facilitate in enabling lifelong learning in the brain and addressing the challenges of distribution shifts.
Revised: These elements could facilitate in enabling lifelong learning in the brain and addressing the challenges of distribution shifts.

Original: 3. P ROPOSED A PPROACH
Revised: 3. P ROPOSED A PPROACH

Original: We hypothesize that error sensitivity modulation and the interactions of multiple memory systems that operate on different timescales may be crucial for a continual learning agent.
Revised: We hypothesize that error sensitivity modulation and the interactions of multiple memory systems that operate on different timescales may be crucial for a continual learning agent.

Original: To this end, our method aims to incorporate a principled mechanism to modulate error sensitivity based on the history of errors in a dual memory experience replay mechanism.
Revised: To this end, our method aims to incorporate a principled mechanism to modulate error sensitivity based on the history of errors in a dual memory experience replay mechanism.

Original: It involves training a working model θ w on a sequence of tasks, while maintaining two additional memories: an instance-based fixed-size episodic memory M , which stores input samples from previous tasks, and a parametric semantic memory θ s , which gradually aggregates knowledge and builds structured representations.
Revised: It involves training a working model θ w on a sequence of tasks, while maintaining two additional memories: an instance-based fixed-size episodic memory M , which stores input samples from previous tasks, and a parametric stable model θ s , which gradually aggregates knowledge and builds structured representations.

Original: The CL setting considered here involves learning a sequence of T i.i.d.
Revised: The CL setting considered here involves learning a sequence of T i.i.d.

Original: tasks from a non-stationary data stream D τ ∈ ( D 1 , ..., D T ) where the goal is to approximate the joint distribution of all the tasks and distinguish between the observed classes without access to the task identity at inference.
Revised: tasks from a non-stationary data stream D τ ∈ ( D 1 , ..., D T ) where the goal is to approximate the joint distribution of all the tasks and distinguish between the observed classes without access to the task identity at inference.

Original: In each training step, the model has access to a random batch of labeled incoming samples drawn from the current task ( x τ , y τ ) ∼ D τ and memory samples drawn from episodic memory ( x m , y m ) ∼ M .
Revised: In each training step, the model has access to a random batch of labeled incoming samples drawn from the current task ( x τ , y τ ) ∼ D τ and memory samples drawn from episodic memory ( x m , y m ) ∼ M .

Original: 3.3. L EARNING FROM L OW -L OSS
Revised: 3.3. L EARNING FROM L OW -L OSS

Original: To mimic the principled mechanism of error sensitivity modulation in the brain, we maintain a memory of errors along the training trajectory and use it to determine how much to learn from each sample.
Revised: To mimic the principled mechanism of error sensitivity modulation in the brain, we maintain a memory of errors along the training trajectory and use it to determine how much to learn from each sample.

Original: To this end, we maintain error memory using the momentum update of the mean crossentropy loss µ ℓ on the batch of samples from the current task.
Revised: To this end, we maintain error memory using the momentum update of the mean cross-entropy loss µ ℓ on the batch of samples from the current task.

Original: As semantic memory is intended to consolidate knowledge and generalize well across tasks, we use it to assess how well the new samples are placed in the consolidated decision boundary and representation space.
Revised: As stable model is intended to consolidate knowledge and generalize well across tasks, we use it to assess how well the new samples are placed in the consolidated decision boundary and representation space.

Original: This also avoids the confirmation bias that can arise from the use of the same learning model.
Revised: This also avoids the confirmation bias that can arise from the use of the same learning model.

Original: For each sample i in  current task batch,  cross-entropy loss L ce is evaluated using semantic memory: [Equation]
Revised: For each sample i in the current task batch, the cross-entropy loss L ce is evaluated using stable model:

Original: [Equation]
Revised: [Equation]

Original: Subsequently, the weight assigned to each sample for the supervised loss is determined by the distance between the sample loss and the average running estimate µ ℓ :
Revised: Subsequently, the weight assigned to each sample for the supervised loss is determined by the distance between the sample loss and the average running estimate µ ℓ :

Original: [Equation]
Revised: [Equation]

Original: where β controls the margin for a sample to be considered a low-loss.
Revised: where β controls the margin for a sample to be considered a low-loss.

Original: This weighting strategy essentially reduces weight
Revised: This weighting strategy essentially reduces weight

Original: λ i as sample loss moves away from the mean estimate, and consequently reducing learning from high-loss samples so that the model learns more from samples with low-loss.
Revised: λ i as sample loss moves away from the mean estimate, and consequently reducing learning from high-loss samples so that the model learns more from samples with low-loss.

Original: The modulated error-sensitive loss for the working model is then given by the weighted sum of all sample losses in the current task batch:
Revised: The modulated error-sensitive loss for the working model is then given by the weighted sum of all sample losses in the current task batch:

Original: L τw = (cid:80) | x b | i λ i l is .
Revised: [Equation]

Original: This simple approach to reducing the contributions of large errors can effectively reduce the abrupt drift in representations at the task boundary and allow the model to gradually adapt to the new task.
Revised: This simple approach to reducing the contributions of large errors can effectively reduce the abrupt drift in representations at the task boundary and allow the model to gradually adapt to the new task.

Original: For instance, when previously unseen classes are observed, the loss for their samples would be much higher than the running estimate; thus, by reducing the weights of these samples, the model can gradually adapt to the new classes without disrupting the learned representations of previously seen classes.
Revised: For instance, when previously unseen classes are observed, the loss for their samples would be much higher than the running estimate; thus, by reducing the weights of these samples, the model can gradually adapt to the new classes without disrupting the learned representations of previously seen classes.

Original: This implicitly accounts for the inherent imbalance between new task samples and past tasks sample in memory by giving higher weight to samples in memory, which will most likely have a low-loss as they have been learned.
Revised: This implicitly accounts for the inherent imbalance between new task samples and past tasks sample in memory by giving higher weight to samples in memory, which will most likely have a low-loss as they have been learned.

Original: The memory of errors is preserved using a momentum update of loss on the task batch using semantic memory.
Revised: The memory of errors is preserved using a momentum update of loss on the task batch on stable model.

Original: To prevent sudden changes in the estimate at the task boundary, we have a task warm-up period during which the running estimate is not updated.
Revised: To prevent sudden changes when the task switches, we have a task warm-up period during which the running estimate is not updated.

Revised: This necessitates the task boundary information during training.

Original: To further keep the error memory stable and make it robust to outliers in the batch, we evaluate the batch statistics on sample losses which lie within one std of batch loss mean:
Revised: To further keep the error memory stable and make it robust to outliers in the batch, we evaluate the batch statistics on sample losses which lie within one std of batch loss mean:

Original: [Equation]
Revised: [Equation]

Original: where µ s and σ s are the mean and std of task losses in semantic memory, l s .
Revised: where µ s and σ s are the mean and std of task losses in stable model, l s .

Original: The mean of the filtered sample losses l fs , is then used to update the memory of errors:
Revised: The mean of the filtered sample losses l fs , is then used to update the memory of errors:

Original: [Equation]
Revised: [Equation]

Original: where α ℓ is the decay parameter that controls how quickly the momentum estimate adapts to current values.
Revised: where α ℓ is the decay parameter that controls how quickly the momentum estimate adapts to current values.

Original: This provides us with an efficient approach to preserve a memory of errors that informs the modulation of error sensitivity.
Revised: This provides us with an efficient approach to preserve a memory of errors that informs the modulation of error sensitivity.

Original: [Equation] EMORY [Equation] YSTEM
Revised: 3.3. D UAL M EMORY S YSTEM

Original: To mimic the interplay of multiple memory systems in the brain and employ different learning timescales, our method maintains a parametric semantic memory and a fixed-size episodic memory.
Revised: To mimic the interplay of multiple memory systems in the brain and employ different learning timescales, our method maintains a parametric semantic memory and a fixed-size episodic memory.

Original: Episodic Memory: enables the consolidation of knowledge through interleaved learning of samples from previous tasks.
Revised: Episodic Memory enables the consolidation of knowledge through interleaved learning of samples from previous tasks.

Original: We propose Error-Sensitive Reservoir Sampling that utilizes the memory of errors to inform Reservoir Sampling (Vitter, 1985).
Revised: We propose Error-Sensitive Reservoir Sampling that utilizes the memory of errors to inform Reservoir Sampling (Vitter, 1985).

Original: Importantly, we perform a pre-selection of candidates for the memory buffer, whereby only the low-loss samples in the current batch of task samples are passed to the buffer for selection:
Revised: Importantly, we perform a pre-selection of candidates for the memory buffer, whereby only the low-loss samples in the current batch of task samples are passed to the buffer for selection:

Original: [Equation]
Revised: [Equation]

Original: This ensures that only the learned samples from the current batch are added to the buffer, helping to better preserve the information and further reduce the abrupt drift in the learned representations.
Revised: This ensures that only the learned samples from the current batch are added to the buffer, thereby preserving the information and reducing the abrupt drift in the learned representations.

Original: It can also facilitate the approximation of the distribution of the data stream by filtering out outliers and noise.
Revised: It can also facilitate the approximation of the distribution of the data stream by filtering out outliers and noise.

Original: Semantic Memory: is  a parametric model  that slowly aggregates and consolidates the knowledge encoded in the weights (Krishnan et al., 2019) of the working model.
Revised: Semantic Memory is maintained using a parametric model (stable model) that slowly aggregates and consolidates the knowledge encoded in the weights (Krishnan et al., 2019) of the working model.

Original: Following Arani et al.
Revised: Following Arani et al.

Original: (2021), we initialize the semantic memory with the weights of the working memory, which afterward keeps a momentum update of that:
Revised: (2021), we initialize the stable model with the weights of the working memory, which afterward keeps a momentum update of that:

Original: [Equation]
Revised: [Equation]

Original: where α is the decay parameter that controls how quickly the semantic model adapts to the working model and r is the update rate to enable stochastic update throughout the training trajectory.
Revised: where α is the decay parameter that controls how quickly the semantic model adapts to the working model and r is the update rate to enable stochastic update throughout the training trajectory.

Original: The semantic memory interacts with episodic memory to implement a dual memory replay mechanism to facilitate knowledge consolidation.
Revised: Stable model interacts with episodic memory to implement a dual memory replay mechanism to facilitate knowledge consolidation.

Original: It involves extracting semantic information fromthe buffer samples, and  the relational knowledge encoded in the output logitsis utilized to enforce consistency in the functional space.
Revised: It extracts semantic information from buffer samples and uses the relational knowledge encoded in the output logits to enforce consistency in the functional space.

Original: This semantic consistency loss encourages the working model to[Table] adapt the representation andthe decision boundary while preserving the relational structure of previous classes.
Revised: This semantic consistency loss encourages the working model to adapt the representation and decision boundary while preserving the relational structure of previous classes.

Original: The loss on the memory samples is thus given by the combination of cross-entropy loss and semantic consistency loss:
Revised: The loss on the memory samples is thus given by the combination of cross-entropy loss and semantic consistency loss:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where the mean squared loss is used as the semantic consistency loss L sc and γ controls the strength of consistency loss.
Revised: where the mean squared loss is used as the semantic consistency loss L sc and γ controls the strength of consistency loss.

Original: The overall loss of the working model is the sum of the loss on the new task samples and the buffer samples, that is, L w =
Revised: The overall loss of the working model is the sum of the loss on the new task samples and the buffer samples, that is, L w =

Original: L τw + L mw .
Revised: L τw + L mw .

Original: The slow aggregation of knowledge in semantic memory further aids in mitigating the abrupt change in representations.
Revised: Slow aggregation of knowledge in stable model further aids in mitigating the abrupt change in representations.

Original: Even after potentially disruptive initial updates at the task boundary, semantic memory retains the representations of the previous classes, and the enforced consistency loss encourages the working model to retain the relational structure among the previous classes and not deviate too much from them.
Revised: Even after potentially disruptive initial updates at the task boundary, stable model retains the representations of the previous classes, and the enforced consistency loss encourages the working model to retain the relational structure among the previous classes and not deviate too much from them.

Original: Therefore, the two components complement each other, and the modulation of error sensitivity coupled with the slow acquisition of semantic knowledge reduces the drift in parameters and hence forgetting.
Revised: Therefore, the two components complement each other, and the modulation of error sensitivity coupled with the slow acquisition of semantic knowledge reduces the drift in parameters and hence forgetting.

Original: For inference, we use semantic memory, as it builds consolidated representations that generalize well acrossthe tasks (Figure S1 and Table S3 in Appendix).
Revised: For inference, we use stable model, as it builds consolidated representations that generalize well across tasks (Figure S1 and Table S3).

Original:  Algorithm 1 in Appendix provides more details and flow of the method.
Revised: See Algorithm 1 for more details.

Original:  C ONTINUAL L EARNING S ETTINGS
Revised:  C ONTINUAL L EARNING S ETTINGS

Original: To faithfully gauge progress in the field, it is imperative to evaluate methods in a number of representative experimental settings that measure how well they address the fundamental challenges of CL in the real world.
Revised: To faithfully gauge progress in the field, it is imperative to evaluate methods on representative experimental settings that measure how well they address the key challenges of CL in the real world.

Original: To this end, we evaluate scenarios that adhere to the proposed desideratas (Farquhar & Gal, 2018) and present a different set of challenges.
Revised: To this end, we evaluate scenarios that adhere to the proposed desideratas (Farquhar & Gal, 2018) and present a different set of challenges.

Original:  Class-IL  presents the most challenging scenarioamong the three common settings (van de Ven & Tolias, 2019) where each task presents a new set of disjoint classes and the agent is required to distinguish between all observed classes without access to the task identity.
Revised: Class Incremental Learning ( Class-IL ) presents the most challenging scenario (van de Ven & Tolias, 2019) where each task presents a new set of disjoint classes and the agent is required to distinguish between all observed classes without access to the task identity.

Original: However, the Class-IL setting assumes that the number of classes across the different tasks remains the same, classes do not reappear, and that the training samplesfor different classes are well balanced.
Revised: However, the Class-IL setting assumes that the number of classes per task remains constant, classes do not reappear, and that the training samples are well balanced across classes.

Original: Generalized Class-IL (GCIL)
Revised: Generalized Class-IL (GCIL)

Original: (Mi et al., 2020) alleviates these limitations and provides a more realistic setting where the number of classes, the appearing classes and their sample sizes for each task are sampled from probabilistic distributions.
Revised: (Mi et al., 2020) alleviates these limitations and offers a more realistic setting where the number of classes, the appearing classes and their sample sizes for each task are sampled from probabilistic distributions.

Original: We further emphasize that noise is ubiquitous in the real world and, therefore, the learning agent is required not only to avoid forgetting but also to learn under noisy labels.
Revised: We further emphasize that noise is ubiquitous in the real world; therefore, the learning agent is required not only to avoid forgetting, but also to learn under noisy labels.

Original: We consider the NoisyClass-IL setting, which adds varying degrees of symmetric noise in the Class-IL setting.
Revised: We consider the Noisy-Class-IL setting, which adds varying degrees of symmetric noise in the Class-IL setting.

Original: Details of the datasets used in each setting are provided in Section A.2 in Appendix. [Table] [Table]
Revised: Details of the datasets used in each setting are provided in Section A.3.

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: We compare our method with strong rehearsal-based methods with single memory (ER, DER++,
Revised: We compare our method with strong rehearsal-based methods with single memory (ER, DER++,

Original: ER-ACE) and multiple memories (CLS-ER) under different CL settings and uniform experimental settings (details in Section A.1 in Appendix).
Revised: ER-ACE) and multiple memories (CLS-ER) under different CL settings and uniform experimental settings (details in Section A.1 in Appendix).

Original: Table 1 provides the results for the Class-IL and GCIL settings with different data complexity and memory constraints. 
Revised: Table 1 provides the results for the Class-IL and GCIL settings with different data complexity and memory constraints. [Table] [Table]

Original: Class-IL requires the method to transfer and consolidate knowledge while also tackling the abrupt representation drift at the task boundary when new classes are observed.
Revised: Class-IL requires the method to transfer and consolidate knowledge while also tackling the abrupt representation drift at the task boundary when new classes are observed.

Original: The consistent performance improvement in both considered datasets demonstrates the proficiency of ESMER in consolidating knowledge with a low memory budget.
Revised: The consistent performance improvement in both considered datasets demonstrates the proficiency of ESMER in consolidating knowledge with a low memory budget.

Original: Generally, we observe that multiple memory systems perform considerably better under these challenging settings.
Revised: Generally, we observe that multiple-memory systems perform considerably better in these challenging settings.

Original: Note that Seq-CIFAR100 represents a considerably more complex setting in terms of both the recognition task among semantically similar classes and the potential representation drift in the split setting, which introduces 20 unobserved classes with high semantic similarity to previous classes.
Revised: Note that Seq-CIFAR100 is a considerably more complex setting in terms of both the recognition task among semantically similar classes and the potential representation drift in the split setting, which introduces 20 unobserved classes with high semantic similarity to previous classes.

Original: The improvement of ESMER over the well-performing CLS-ER in this challenging setting suggests thaterror sensitivity modulation in our  method successfully mitigates representation drift and reduces interference among similar tasks.
Revised: The improvement of ESMER over the well-performing CLS-ER in this challenging setting suggests that our error-sensitivity modulation method successfully mitigates representation drift and reduces interference among similar tasks.

Original: The performance of our method under extremely small buffer sizes in Table 2, which exacerbates the abrupt representation drift, further supports the claim.
Revised: The performance of our method under extremely small buffer sizes in Table 2, which exacerbates the abrupt representation drift, further supports the claim.

Original: The improvement over CLS-ER is promising given thatour method utilizes one semantic memory, whereas CLS-ER utilizes two that aggregate knowledge at different timescales.
Revised: This gain is promising, given that CLS-ER uses two semantic memories with different update frequencies, while our method uses only one.

Original: Table 1 also provides the results for the GCIL setting, which further requires the model to tackle the challenges of class imbalance, sample efficiency, and learn from multiple occurrences of an abject.
Revised: Table 1 also provides the results for the GCIL setting, which further requires the model to tackle the challenges of class imbalance, sample efficiency, and learn from multiple occurrences of an abject.

Original: ESMER provides a consistent improvement in both uniform and longtail settings.
Revised: ESMER provides a consistent improvement in both uniform and longtail settings.

Original: The error sensitivity modulation and consequent gradual adaptation to unobserved classes can allow the model to deal with the class imbalance and learn over multiple occurrences well.
Revised: The error sensitivity modulation and consequent gradual adaptation to unobserved classes can allow the model to deal with the class imbalance and learn over multiple occurrences well.

Original: Considering a new task with a mix of unobserved and recurring classes, the loss of samples from the recurring classes would most likely be low, as they have been learned earlier, while the loss on unobserved classes would be significantly higher.
Revised: Considering a new task with a mix of unobserved and recurring classes, the loss of samples from the recurring classes would most likely be low, as they have been learned earlier, while the loss on unobserved classes would be significantly higher.

Original: Consequently, the contribution of unobserved classes would be down-weighted, allowing the model to focus and adapt the representations of the recurring class with reduced interference and representation drift from the unobserved classes.
Revised: Consequently, the contribution of unobserved classes would be down-weighted, allowing the model to focus and adapt the representations of the recurring class with reduced interference and representation drift from the unobserved classes.

Original: Learning more from low errors also provides more consistent learning and can improve sample efficiency by avoiding potentially suboptimal gradient directions caused by outliers.
Revised: Learning more from low errors also provides more consistent learning and can improve sample efficiency by avoiding potentially suboptimal gradient directions caused by outliers.

Original: Finally, Table 2 provides the results under the Noisy-Class-IL setting where the agent is required to concurrently tackle the challenges of learning from noisy labels.
Revised: Finally, Table 3 provides the results under the Noisy-Class-IL setting where the agent is required to concurrently tackle the challenges of learning from noisy labels.

Original: Our strong empirical results demonstrate the superiority of ESMER in tackling the unique challenges of this setting.
Revised: Our strong empirical results demonstrate the superiority of ESMER in tackling the unique challenges of this setting.

Original: Learning from low errors implicitly makes our model more robust to label corruption and reduces memorization, since the noisy labels would correspond to higher errors.
Revised: Learning from low errors implicitly makes our model more robust to label noise and reduces memorization, since the noisy labels would correspond to higher errors.

Original: Furthermore, the error-sensitive reservoir sampling approach ensures a higher purity of the memory buffer, which plays a critical role in noisy CL as buffer samples are replayed multiple times, further aggravating their harmful effect. 
Revised: Furthermore, the error-sensitive reservoir sampling approach ensures a higher purity of the memory buffer, which plays a critical role in noisy CL as buffer samples are replayed multiple times, further aggravating their harmful effect. [Table]

Original:  W HY DOES ESMER WORK ?
Revised:  W HY DOES ESMER WORK ?

Original: Here, we attempt to provide insights into what enables the gains with ESMER.
Revised: Here, we attempt to provide insights into what enables the gains with ESMER.

Original: We perform our analysis on Seq-CIFAR10 with 200 buffer size and compare it with ER, DER++ and CLSER.
Revised: We perform our analysis on Seq-CIFAR10 with 200 buffer size and compare it with ER, DER++ and CLSER.

Original: Ablation Study: We systematically remove the salient components of our method and evaluate the effect onits performance in Table 3.
Revised: Ablation Study: We systematically remove the salient components of our method and evaluate their effect on performance in Table 4.

Original: Error-sensitivity modulation provides considerable gains in both low-buffer and noisy label settings.
Revised: Error-sensitivity modulation provides considerable gains in both low-buffer and noisy label settings.

Original: Furthermore, the gradual accumulation of knowledge in semantic memory proves to be a highly proficient approach for consolidating knowledge.
Revised: Furthermore, the gradual accumulation of knowledge in stable model proves to be a highly proficient approach to consolidating knowledge.

Original: Finally, the proposed error-sensitive reservoir sampling technique provides further generalization gains, particularly in the Noisy-Class-IL setting, where maintaining a clean buffer is critical.
Revised: Finally, the proposed error-sensitive reservoir sampling technique provides further generalization gains, particularly in the Noisy-Class-IL setting, where maintaining a clean buffer is critical.

Original: Importantly, these components complement each other so that the combination provides the maximum gains.
Revised: Importantly, these components complement each other so that the combination provides the maximum gains.

Original: Drift in Representations: To assesswhether ESMER can mitigate the abrupt drift in representations at the task boundary, we monitor the performance of the model on Task 1 as it learns Task 2.
Revised: Drift in Representations: To assess ESMER in mitigating the abrupt drift in representations at the task boundary, we monitor the performance of the model on Task 1 as it learns Task 2.

Original: Figure 2 (a) shows that ESMER successfully maintains its performance, while ER suffers a sudden performance drop and fails to recover from earlier disruptive updates.
Revised: Figure 2 (a) shows that ESMER successfully maintains its performance, while ER suffers a sudden performance drop and fails to recover from earlier disruptive updates.

Original: Furthermore, while the working model in ESMER undergoes  reduced representation drift after the task warmup period, the semantic memory retains the performance and provides a stable reference for constraining the update of the working model, helping it recover from the disruption.
Revised: Furthermore, while the working model in ESMER undergoes a reduced representation drift after the task warmup period, the stable model retains the performance and provides a stable reference to constrain the update of the working model, helping it recover from the disruption.

Original: This suggests that error-sensitivity modulation, coupled with gradual adaptation in semantic memory, provides a promising approach to mitigate abrupt representation drift.
Revised: This suggests that error-sensitivity modulation, coupled with gradual adaptation in stable model, is a promising approach to mitigate abrupt representation drift.

Original: Recency Bias: The sequential learning of tasks with standard SGD in CL induces a recency bias in the model, making it much more likely to predict recently learned classes (Hou et al., 2019).

Original: To evaluate the task recency bias of the different methods, we calculate the average probability of predicting each task at the end of training by averaging the softmax outputs of the classes associated with each task on the test set.

Original: Figure 2 (b) shows that ESMER provides more uniform predictions across tasks.

Original: Generally, methods employing multiple memories are better in reducing the recency bias, and the dynamics of low-loss learning in our method further reduces the bias.

Original: Figure 3 further show that ESMER maintains a better balance between the plasticity and stability of the model.

Original: Robustness to Noisy Labels: We see considerable gains with ESMER in the challenging NoisyClass-IL setting.

Original: Here, we probe into what enables our method to be more robust to label corruption.

Original: One of the key challenges in learning under noisy labels is the tendency of DNNs to memorize in-

Original: [Figure]

Original: q

Original: [Figure]

Original: r a i n i n g 98.

Original: ER

Original: T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98. DER++ T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98. CLS-ER
Revised: Robustness to Noisy Labels: We probe into what enables ESMER to be more robust to label noise.

Original: T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98. ESMER [Figure] correct labels (Arpit et al., 2017) that adversely affect their performance (Sukhbaatar et al., 2014).
Revised: One of the key challenges in learning under noisy labels is the tendency of DNNs to memorize incorrect labels (Arpit et al., 2017) that adversely affect their performance (Sukhbaatar et al., 2014).

Original: To evaluate the degree of memorization, we track the performance of the model on noisy and clean samples during training.
Revised: To evaluate the degree of memorization, we track the performance of the model on noisy and clean samples during training.

Original: Figure 4 (a) shows that ESMER significantly reduces memorization, leading to improved performance.
Revised: Figure 3 (a) shows that ESMER significantly reduces memorization, leading to better performance.

Original: Furthermore, the quality of the samples in the memory buffer can play a critical role in the performance of the model, as the memory samples are replayed multiple times.
Revised: Furthermore, the quality of the samples in the memory buffer can play a critical role in the performance of the model, as the memory samples are replayed multiple times.

Original: Figure 4 (b) compares the percentage of noisy labels in the buffer with our proposed error-sensitive reservoir sampling and the typical reservoir sampling approach.
Revised: Figure 3 (b) compares the percentage of noisy labels in the buffer with our proposed error-sensitive reservoir sampling and the typical reservoir sampling approach.

Original: Our approach considerably reduces the probability of adding a noisy sample in episodic memory and maintains a higher quality of the buffer.
Revised: Our approach considerably reduces the probability of adding a noisy sample in episodic memory and maintains a higher quality of the [Figure] [Figure] ER

Revised: T a s k 1 T a s k 2 T a s k 3 T a s k 4 T a s k98.

Revised: ESMER buffer.

Original: Note that both features are a by-product of mimicking the error sensitivity modulation in the brain rather than  explicitly designed for the setting.
Revised: Note that both features are a byproduct of mimicking error-sensitivity modulation in the brain rather than being explicitly designed for the setting.

Revised: Recency Bias: The sequential learning of tasks with standard SGD in CL induces a recency bias in the model, making it much more likely to predict newly learned classes (Hou et al., 2019).

Revised: To evaluate the task recency bias of the different methods, we calculate the average probability of predicting each task at the end of training by averaging the softmax outputs of the classes associated with each task on the test set.

Revised: Figure 2 (b) shows that ESMER provides more uniform predictions across tasks.

Revised: Generally, methods that employ multiple memories are better at reducing the recency bias, and the dynamics of low-loss learning in our method further reduces the bias.

Revised: Figure 4 further shows that ESMER maintains a better balance between plasticity and stability of the model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Inspired by the dynamics of error-based learning and multiple memory systems in the brain, we proposed ESMER, a novel approach for error sensitivity modulation based experience replay in a dual memory system.
Revised: Inspired by the dynamics of error-based learning and multiple memory systems in the brain, we proposed ESMER, a novel approach for error sensitivity modulation based experience replay in a dual memory system.

Original: The method maintains a memory of errors along the training trajectory, which is used to modulate the contribution of incoming samples towards learning such that the model learns more from low-errors.
Revised: The method maintains a memory of errors along the training trajectory, which is used to modulate the contribution of incoming samples towards learning such that the model learns more from low-errors.

Original: Additionally, semantic memory is maintained that gradually aggregates the knowledge encoded in the weights of the working model using momentum update.
Revised: Additionally, semantic memory is maintained that gradually aggregates the knowledge encoded in the weights of the working model using momentum update.

Original: We also proposed an error-sensitive reservoir sampling approach that prioritizes the representation of low-loss samples in the memory buffer, which are better suited for retaining knowledge.
Revised: We also proposed an error-sensitive reservoir sampling approach that prioritizes the representation of low-loss samples in the memory buffer, which are better suited for retaining knowledge.

Original: Our empirical evaluation demonstrates the effectiveness of our approach in challenging and realistic
Revised: Our empirical evaluation demonstrates the effectiveness of our approach in challenging and realistic

Original: CL settings.
Revised: CL settings.

Original: ESMER mitigates abrupt drift in representations, provides higher robustness to noisy labels, and provides more uniform performance across tasks.
Revised: ESMER mitigates abrupt drift in representations, provides higher robustness to noisy labels, and provides more uniform performance across tasks.

Original: Our findings suggest that a principled mechanism for learning from low errors in a dual memory system provides a promising approach for mitigating abrupt representation drift in CL that warrants further study.
Revised: Our findings suggest that a principled mechanism for learning from low errors in a dual-memory system provides a promising approach to mitigating abrupt representation drift in CL that warrants further study.

Original: A A PPENDIX A. E XPERIMENTAL D ETAILS
Revised: A. E XPERIMENTAL D ETAILS

Original: We conducted all our experiments under common settings to disentangle the performance improvement of the different CL methods from the training settings (Mirzadeh et al., 2020).
Revised: We conducted all our experiments under common settings to disentangle the performance improvement of the different CL methods from the training settings (Mirzadeh et al., 2020).

Original: Following (Arani et al., 2021; Buzzega et al., 2020), we use ResNet-18 network and train with SGD optimizer.
Revised: Following (Arani et al., 2021; Buzzega et al., 2020), we use ResNet-18 network and train with SGD optimizer.

Original: For all our experiments, we train for 50 epochs with a batch size of 32 for both incoming samples and memory and apply random crop and horizontal flip data augmentations.
Revised: For all our experiments, we train for 50 epochs with a batch size of 32 for both incoming samples and memory and apply random crop and horizontal flip data augmentations.

Original: Note that we store non-augmented images in the buffer and apply the data augmentations to the memory samples for replay.
Revised: Note that we store non-augmented images in the buffer and apply the data augmentations to the memory samples for replay.

Original: We report the mean and std of 3 runs.
Revised: We report the mean and std of 3 runs.

Original: A.1. H YPERPARAMETER T UNING
Revised: A.1. H YPERPARAMETER T UNING

Original: We use a small validation set to select the hyperparameters for all methods.
Revised: We use a small validation set to select the hyperparameters for all methods.

Original: For ESMER, we use α l = 0 .
Revised: For ESMER, we use α l = 0 .

Original: 99 , α l = 0 .
Revised: 99 , α l = 0 .

Original: 999 , and tune γ ∈ { 0 . 15 , 2 . 0 } , β ∈ [0 . 8 , 3 . 0] and r ∈ [0 , 1] .
Revised: 999 , and tune γ ∈ { 0 . 15 , 2 . 0 } , β ∈ [0 . 8 , 3 . 0] and r ∈ [0 , 1] .

Original: Note that we did not conduct an exhaustive parameter search for each experiment, and it is possible that a better hyperparameter search can further improve performance.
Revised: Note that we did not conduct an exhaustive parameter search for each experiment, and it is possible that a better hyperparameter search can further improve performance.

Original: The hyperparameters used for each experiment are provided in Table S1.
Revised: The hyperparameters used for each experiment are provided in Table S1.

Original: For the baselines, we use the results from (Arani et al., 2021) if similar settings are provided; otherwise we conduct a grid search over the hyperparameter values  in the  paper.
Revised: For the baselines, we use the results from (Arani et al., 2021) if similar settings are provided; otherwise, we conduct a grid search over the hyperparameter values used in the original paper.

Original: For ER and ERACE, we search η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } .
Revised: For ER and ER-ACE, we search η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } .

Original: For DER++ we search over η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } , α ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } and β ∈ { 0 . 5 , 1 . 0 } .
Revised: For DER++ we search over η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } , α ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } and β ∈ { 0 . 5 , 1 . 0 } .

Original: Finally, for CLS-ER, we search over α s = 0 .
Revised: Finally, for CLS-ER, we search over α s = 0 .

Original: 999 , α p = 0 .
Revised: 999 , α p = 0 .

Original: 999 , λ ∈ { 0 . 15 , 2 . 0 } , r p ∈ [0 , 1] and r s ∈ [0 , 1] .
Revised: 999 , λ ∈ { 0 . 15 , 2 . 0 } , r p ∈ [0 , 1] and r s ∈ [0 , 1] .

Original: The parameters selected for the baselines in each setting are provided in Table S2.
Revised: The parameters selected for the baselines in each setting are provided in Table S2.

Original: A. D ATASETS
Revised: A. C OMPARISON WITH N EUROMODULATION

Revised: Metaplasticity (Abraham, 2008) and Neuromodulation (Doya, 2002) refers to different mechanisms to modulate the plasticity rate of individual synapses (strengths / weights of the incoming and outgoing connections in DNNs) (Kudithipudi et al., 2022; Bailey et al., 2000). These form the biological underpinning behind the popular regularization-based approaches where the goal is to identify synapses that are important for previous tasks and penalize changes to them while learning a new task (Zenke et al., 2017; Kirkpatrick et al., 2017).

Revised: Neuromodulation has also been explored in conjunction with local learning (Madireddy et al., 2020) which shows promise.

Revised: The error sensitivity modulation in ESMER differs substantially from these methods, as instead of modulating the plasticity rate of individual synapses, we modulate the contribution of each sample

Revised: [Table]

Revised: in the batch based on its error magnitude.

Revised: Our work aims to bring about a salient characteristic of the learning dynamics of the brain: it learns more from small consistent errors compared to large errors.

Revised: This is not the case for standard ANNs, where learning scales linearly with the error size.

Revised: Therefore, the error sensitivity modulation in ESMER specifically focuses on utilizing a memory of errors to modulate the contribution of samples to learning based on their error magnitude, so that the model learns more from small errors.

Revised: Please note that our approach is complementary to metaplasticity and neuromodulation-based approaches, and the interaction of these biologically plausible mechanisms is an interesting avenue of research.

Revised: A. D ATASETS

Original: Here we provide details of the datasets used in each of the considered CL setting.
Revised: Here we provide details of the datasets used in each of the considered CL setting.

Original: A.2. C LASS -IL
Revised: A.3. C LASS -IL

Original: For Class-IL, we use the common sequential versions of CIFAR10 (Seq-CIFAR10) and CIFAR (Seq-CIFAR100) datasets whereby the classes are split into five tasks each containing a disjoint set of 2 and 20 classes, respectively.
Revised: For Class-IL, we use the common sequential versions of CIFAR10 (Seq-CIFAR10) and CIFAR (Seq-CIFAR100) datasets whereby the classes are split into five tasks each containing a disjoint set of 2 and 20 classes, respectively.

Original: We do not vary the order of classes, and so, for instance, Task 1 in
Revised: We do not vary the order of classes, and so, for instance, Task 1 in

Original: Seq-CIFAR10 always contains the first two classes.
Revised: Seq-CIFAR10 always contains the first two classes.

Revised: Note that our method does not utilize separate classification heads or subnets.

Revised: For completion, we also evaluate the performance of the same model trained for Class-IL under the Task-IL setting where the model assumes availability of task labels at inference.

Revised: We use the task label to narrow down predictions to the subset of output logits belonging to classes belonging to the task.

Original: A.2. GCIL
Revised: A.3. GCIL

Original: Following (Mi et al., 2020; Arani et al., 2021), we apply the GCIL setting on CIFAR100 dataset.
Revised: Following (Mi et al., 2020; Arani et al., 2021), we apply the GCIL setting on CIFAR100 dataset.

Original: The setting involves training 20 tasks sequentially, and for each task, the number and appearing classes are sampled from a probability distribution but capped at a maximum of 50 and subsequentlysamples are drawn from either a uniform (Unif) or longtail sample distributions.
Revised: The setting involves training 20 tasks sequentially, and for each task, the number and appearing classes are sampled from a probability distribution but capped at a maximum of 50 and subsequentlysamples are drawn from either a uniform (Unif) or longtail sample distributions.

Original: For further details, see (Mi et al., 2020).
Revised: For further details, see (Mi et al., 2020).

Original: Importantly, we found that even for a fixed dataset seed, the corresponding tasks in the Longtail and Unif settings differ not only in the sample distribution (as expected) but also significantly in the number of classes drawn and the appearing classes.
Revised: Importantly, we found that even for a fixed dataset seed, the corresponding tasks in the Longtail and Unif settings differ not only in the sample distribution (as expected) but also significantly in the number of classes drawn and the appearing classes.

Original: For instance, with the dataset seed set to 1993, the last task in the Longtail setting contains 50 classes, whereas the corresponding task in the Unif setting contains only 17 classes.
Revised: For instance, with the dataset seed set to 1993, the last task in the Longtail setting contains 50 classes, whereas the corresponding task in the Unif setting contains only 17 classes.

Original: This makes the former task significantly easier.
Revised: This makes the former task significantly easier.

Original:  We fix this in our implementation by drawing 20 random numbers (after setting the random seed to 1993) when the dataset is initialized and setting the same unique seed before sampling each task.
Revised: [Table] We fix this in our implementation by drawing 20 random numbers (after setting the random seed to 1993) when the dataset is initialized and setting the same unique seed before sampling each task.

Original: This ensures that the same number of classes and the appearing classes (if not all the samples for a particular class are seen already) are drawn for the two settings.
Revised: This ensures that the same number of classes and the appearing classes (if not all the samples for a particular class are seen already) are drawn for the two settings.

Original: This provides a better evaluation of the effect of the difference in data distributions between the two variants.
Revised: This provides a better evaluation of the effect of the difference in data distributions between the two variants.

Original: This leads to a significant difference in baseline performance compared to those reported in (Arani et al., 2021).
Revised: This leads to a significant difference in baseline performance compared to those reported in (Arani et al., 2021).

Original: We also perform the hyperparameter for all the considered methods and find that, for

Original: Input and Params: error memory params α ℓ and β ; semantic memory params α and r ; semantic consistency weight γ ; data stream D ; learning rate η

Original: Initialize: l µ = 0 , M ←− {} , θ s = θ w 1: while Training do 2:

Original: Sample batch from task stream, ( x τ , y τ ) ∼ D τ , and episodic memory, ( x m , y m ) ∼ M 3:

Original: Evaluate loss of task samples on semantic memory, l is (Eq. 1) 4:

Original: Get the weights of task samples, λ i (Eq. 2) 5:

Original: Evaluate the sensitivity-modulated task loss, L τw = (cid:80) | x b | i λ i · l is 6:

Original: Evaluate the cross-entropy and consistency loss on memory samples, L mw (Eq. 7) 7:

Original: Combine overall loss:

Original: L w = L τw + L mw 8: Update parameters of working model: θ w ←− θ w − η ∇ θ w L w 9: Update the semantic memory: θ s ← αθ s + (1 − α ) θ w , if r > a ∼ U (0 , 1) (Eq. 6) 10:
Revised: We also perform the hyperparameter for all the considered methods and find that, for the modified setting, our choice of parameters performs considerably better than the ones used in (Arani et al., 2021).

Original: Filter candidates for episodic memory, ( x c , y c ) (Eq. 5) 11:

Original: Update episodic memory: M ←− Reservoir ( M , ( x c , y c )) 12:

Original: Remove outliers from task batch loss: l fs (Eq. 3) 13:

Original: Update error memory if epoch > task warm-up period: µ ℓ (Eq. 4) return θ s the modified setting, our choice of parameters performs considerably better than the ones used in (Arani et al., 2021).

Original: We report the results on more optimal hyperparameters provided in Table S2. 
Revised: We report the results on more optimal hyperparameters provided in Table S2. [Table]

Original: A.2. N OISY -C LASS -IL
Revised: A.3. N OISY -C LASS -IL

Original: Noisy Class-IL aims to evaluate the robustness of a learning agent to label noise.
Revised: Noisy Class-IL aims to evaluate the robustness of a learning agent to label noise.

Original: Since noise is ubiquitous in real-world streams, ideally a CL method should be able to tackle the challenges of learning under noisy labels and CL concurrently.
Revised: Since noise is ubiquitous in real-world streams, ideally a CL method should be able to tackle the challenges of learning under noisy labels and CL concurrently.

Original: This requires them to avoid memorization and forgetting.
Revised: This requires them to avoid memorization and forgetting.

Original: We consider varying degrees of symmetric label corruption on Seq-CIFAR10 dataset, Noisy-Seq-CIFAR10 .
Revised: We consider varying degrees of symmetric label noise on Seq-CIFAR10 dataset, NoisySeq-CIFAR10 .

Original: For each task, we sample random labels from the classes in the task and replace a fraction of the original labels with the random ones.
Revised: For each task, we sample random labels from the classes in the task and replace a fraction of the original labels with the random ones.

Original: Note that, under this setting, the random label may correspond to the original label.
Revised: Note that, under this setting, the random label may correspond to the original label.

Original: A. W ORKING M ODEL P ERFORMANCE
Revised: A. W ORKING M ODEL P ERFORMANCE

Original: To assess how well semantic memory aggregates knowledge across the task, we compare its performance with the performance of the working model across all datasets and memory constraints in Table S3.
Revised: To assess how well stable model aggregates knowledge across the task, we compare its performance with the performance of the working model across all datasets and memory constraints in Table S3.

Original: Semantic memory considerably provides higher performance.
Revised: Stable model considerably provides higher performance.

Original: To better understand, we look at the task-wise performance on the test set at the end of each task in Figure S1.
Revised: To better understand, we look at the taskwise performance on the test set at the end of each task in Figure S1.

Original: Semantic memory retains more knowledge of earlier tasks and consolidates information to build representations which generalize acrossthe tasks.
Revised: Stable model retains more knowledge of previous tasks and consolidates information to build representations that generalize across tasks.

Original: A. E FFECT OF H YPERPARAMETERS
Revised: A. E FFECT OF H YPERPARAMETERS

Original: Tuning ESMER involves setting the parameters for two complementary components: error sensitivity modulation ( β , α l ) and semantic memory ( r , α , γ ).
Revised: Tuning ESMER involves setting the hyperparameters for two complementary components: error sensitivity modulation ( β , α l ) and stable model ( r , α , γ ).

Original: To understand the interaction of these components and the sensitivity of our method to hyperparameters, we look at the performance of the model with different sets of values for the most pertinent parameters ( β , r ).
Revised: To understand the interaction of these components and the sensitivity of our method to hyperparameters, we look at the performance of the model with different sets of values for the most pertinent parameters ( β , r ).

Original: Table S4 shows that the loss margin, β , and the update frequency, r complement each other and, generally, a higher r requires a lower β .
Revised: Table S4 shows that the loss margin, β , and the update frequency, r complement each other and, generally, a higher r requires a lower β .

Original: Furthermore, a different set of values can provide similar performance, which significantly facilitates hyperparameter tuning.
Revised: Furthermore, a different set of values can provide similar performance, which significantly facilitates hyperparameter tuning.

Original: As can be seen in Table S1, the majority of parameters ( α s , α ℓ and γ ) can be fixed, and the optimal parameters do not vary much for different memory budgets.[Table] [Figure] [Table]
Revised: As can be seen in Table S1, the majority of parameters ( α s , α ℓ and γ ) can be fixed, and the optimal parameters do not vary much for different memory budgets.

Original: B u ff e r =

Original: [Table]

Original: [Table]
Revised: [Table]

Original: p =0.7, r s =0.

Original: Seq-CIFAR

Original: ER η =0.

Original: ER-ACE η =0.

Original: DER++ η =0.03, α =0.5, β =0.

Original: CLS-ER η =0.1 λ

Original: =

Original: 0.15, r p =0.3, r s =0.

Original: ER η =0.

Original: ER-ACE η =0.

Original: DER++ η =0.03, α =0.2, β =0.

Original: CLS-ER η =0.1 λ

Original: =

Original: 0.15, r p =0.3, r s =0.

Original: ER η =0.

Original: ER-ACE η =0.

Original: DER++ η =0.03, α =0.2, β =0. CLS-ER η =0.1 λ = 0.15, r p =0.3, r s =0.
Revised: [Figure]

Original: Noisy-Seq-CIFAR

Original: ER η =0.

Original: ER-ACE η =0.

Original: DER++ η =0.03, α =0.1, β =0.

Original: CLS-ER η =0.1 λ

Original: =

Original: 0.15, r p =0.3, r s =0.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: 65.37 ±0.

Original: 55.80 ±2.

Original: 62.73 ±2.

Original: 51.43 ±3.

Original: 59.63 ±1. 55.96 ±1. 59.29 ±1.55.20 ±0.
Revised: ±1.

Original: 57.49 ±2.

Original: 52.98 ±3.

Original: 59.06 ±2.

Original: 62.15 ±1.62.47 ±0. 64.34 ±0. 63.64 ±0.
Revised: 64.40 ±1.

Original: 64.00 ±0.

Original: 64.99 ±2.

Original: 63.82 ±0.

Original: 66.79 ±1.

Original: 63.78 ±1.

Original: 69.16 ±0.65.18 ±3. 64.81 ±1. 61.70 ±
Revised: 66.36 ±0.

Original: [Table]

Original: 67.37 ±1.

Original: 64.40 ±1.

Original: 66.36 ±0.

Original: 62.93 ±0.
Revised: 62.93 ±0.

Original: [Table]

Original: T T T T
Revised: T a s k

Original: T 98.

Original: T

Original: T

Original: T

Original: T

Original: T 75.

Original: T

Original: T

Original: T

Original: T

Original: T 76.

Original: T a s k

Original: P r o b a b ili t y
Revised: P r o b a b ili t y

Original: B u
Revised: B u

Original: ff e r
Revised: ff e r

Original: Seq-CIFAR
Revised: Seq-CIFAR

Original: Seq-CIFAR
Revised: Seq-CIFAR

Original: Task 1 Task 2 Task
Revised: Task 1 Task 2 Task

Revised: [Figure]

Original: Task 4 Task
Revised: Task 4 Task

Revised: [Figure]

Original: ER
Revised: ER

Original: DER++
Revised: DER++

Original: ER-ACE
Revised: ER-ACE

Original: CLS-ER
Revised: CLS-ER

Original: ESMER 0.
Revised: ESMER 0.

Original: T a s k
Revised: T a s k

Original: P r o b a b ili t y
Revised: P r o b a b ili t y

Original: B u
Revised: B u

Original: ff e r
Revised: ff e r

Original: ER
Revised: ER

Original: DER++
Revised: DER++

Original: ER-ACE
Revised: ER-ACE

Original: CLS-ER
Revised: CLS-

Revised: [Figure]

Original: ESMER
Revised: ESMER

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Table] T T

Revised: T

Revised: T

Revised: T 98.

Revised: T

Revised: T

Revised: T

Revised: T

Revised: T 75.

Revised: T

Revised: T

Revised: T

Revised: T

Revised: T 76.