Paper: May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels (ICLR-2024)

Reviewer comment (Experiments):
The authors advocate the use of forgetting as a selection criterion, yet its effectiveness may be limited to specific types of label noise. It appears inadequate for more complex noise categories, for instance, pairflip-45 and instance label noise, let alone for real-world datasets. A more extensive experimental analysis is essential to thoroughly investigate the 'forgetting' phenomenon's applicability and limitations in various noise scenarios.

Author response:
Following the reviewers' suggestion we conduct additional analysis on the *real-world* noisy dataset, Food-101N, as done in [PuriDivER]. The results show that our method produces an increase of $4.52$% accuracy points w.r.t. our second-best PuriDivER.ME and of $5.40$% accuracy points when 'fine-tuning on the buffer' is enabled for both methods,  with our method being only $5.28$% accuracy point under the upper-bound (Joint).

| **Benchmark**         | **Food-101N** |
|------------------------|---------------|
| Joint                  | $39.91$       |
| **PuriDivER.ME**       | $25.34$       |
|   *w. buffer fit.*     | $29.23$       |
| **OURs**               | $29.86$       |
|   *w. buffer fit.*     | **$34.63$**   |
    
 We also note that our work comprises an initial empirical evaluation of the behaviour of clean and noisy data in the continual learning scenario, out of which the method is then built. On this premise, and in line with the current literature [PudiDivER,SPR], instance-independent label noise allows us to carry out such analyses. Moreover, we want to stress that we closely follow the settings of the other CNL methods and therefore conduct experiments suitable for comparisons. For what concerns the other types of noise, we firstly want to remark that the **asymmetric** noise setting, also called *class-dependent* label noise, is indeed an approximation of real-world corruption patterns. Specifically, in the CIFAR-100 dataset, images are grouped based on their relative superclasses, thus each image comes with a "fine" label (specific class) and a "coarse" label (superclass). The asymmetric noise injection alters labels within the same superclass. This results in sample ambiguity occurring only between similar classes as it would in a **realistic** scenario. Secondly, instance-dependent noise has not yet been extensively investigated and only a few studies address it, owing to its complex modelling [ALNL].

Reviewer comment (Experiments):
The experiments are confined to small datasets, with only 10 classes from WebVision utilized, which casts doubt on the scalability of the proposed methods. For a dataset of 50k, joint learning seems more appropriate, and the unimpressive outcomes in Table 1 underscore the proposed methods' shortcomings. Similarly, the results in Table 2 are too inconclusive for any solid verification. The authors should report the results of other label noise methods such as DivideMix on Joint.

Author response:
Following the reviewers' suggestion we conduct additional analysis on the *real-world* noisy dataset, Food-101N, as done in [PuriDivER]. The results show that our method produces an increase of $4.52$% accuracy points w.r.t. our second-best PuriDivER.ME and of $5.40$% accuracy points when 'fine-tuning on the buffer' is enabled for both methods,  with our method being only $5.28$% accuracy point under the upper-bound (Joint).

| **Benchmark**         | **Food-101N** |
|------------------------|---------------|
| Joint                  | $39.91$       |
| **PuriDivER.ME**       | $25.34$       |
|   *w. buffer fit.*     | $29.23$       |
| **OURs**               | $29.86$       |
|   *w. buffer fit.*     | **$34.63$**   |
    
 We also note that our work comprises an initial empirical evaluation of the behaviour of clean and noisy data in the continual learning scenario, out of which the method is then built. On this premise, and in line with the current literature [PudiDivER,SPR], instance-independent label noise allows us to carry out such analyses. Moreover, we want to stress that we closely follow the settings of the other CNL methods and therefore conduct experiments suitable for comparisons. For what concerns the other types of noise, we firstly want to remark that the **asymmetric** noise setting, also called *class-dependent* label noise, is indeed an approximation of real-world corruption patterns. Specifically, in the CIFAR-100 dataset, images are grouped based on their relative superclasses, thus each image comes with a "fine" label (specific class) and a "coarse" label (superclass). The asymmetric noise injection alters labels within the same superclass. This results in sample ambiguity occurring only between similar classes as it would in a **realistic** scenario. Secondly, instance-dependent noise has not yet been extensively investigated and only a few studies address it, owing to its complex modelling [ALNL].

We are unsure about what makes the reviewer refer to our results as *unimpressive*, whereas the other reviewers valued our work for the *novelty*, the *comprehensive experiments* (**2XeP**), the *effectiveness* (**Jj1M**) of the method and the *thoroughness* of the analysis (**aFAe**).

Reviewer comment (Novelty):
I believe the alternate replay is an interesting design, but the novelty of the proposed method is still limited. The strategies used in this paper are simple and very common techniques, such as selecting clean samples based on loss thresholds and MixMatch. It seems like a direct combination of existing technologies, with no essential innovation in methodology, especially for continual learning.

Author response:
**Weakness 1**:   
We struggle to understand the point raised by the reviewer; they recognize the novelty of Alternate Replay (the main contribution of our work) but still find the contribution not good enough. 

As the vast majority of current LNL works, our work combines existing techniques to bring new ideas and methodologies (i.e. our **AER** and **ABS**). While we borrow and draw inspiration from well-established techniques in the literature about noisy labels, they do not constitute our primary contributions. 

Indeed, as appreciated by reviewer 2XeP, our work rearranges experience replay to handle noisy label problems, introducing a novel bi-fold sampling strategy which favours the presence of clean data from both the past and current tasks. Such a strategy exploits the fact that mislabeled examples are forgotten faster than regular ones and introduces buffer forgetting to separate clean and noisy in-memory samples. 

Notably, we are the first to address incremental learning under the *noisy* and *offline* setup, whereas the current existing works focus on the online setup. For example, PuriDivER cannot adequately fit complex distributions when only one epoch is allowed, as holds for FOOD-101N.

Reviewer comment (Experiments):
It is difficult to adapt to more complex noise distribution by selecting samples directly based on the loss threshold. More advanced methods should be considered [1, 2] instead of still adjusting the threshold manually, which is not helpful for either CL or noisy label problems.

Author response:
**Weakness 2**:  

The methods suggested by the reviewer rely on meta-learning to learn a label correction mechanism. To do so, these approaches need a small *clean validation set*. 
We argue that such a requirement might clash with our setting, tailored for realistic scenarios. Indeed, several real-world noisy datasets do not come with annotations regarding which examples are "clean" and which are not.
Hence, we believe that the direct application of [1],[2] is slightly out-of-scope for our work; we will modify our manuscript to include the suggested works in the related materials.

Regarding the simplicity of our sample selection technique, we have a different perspective and see it as a strength of our approach. 
We emphasise that our algorithm has a single hyper-parameter (the learning rate) and does not require any other tuning (we use the same threshold for each dataset).   

Nonetheless, to assess the merits/validity of our choice, we conducted new experiments by fitting the more "complex" Gaussian Mixture Model (GMM) on per-sample loss values to divide training samples into labelled *vs* unlabeled sets. 
We equip our approach with this selection technique and report here a comparison with our original "simple" approach (we do so for both synthetic - CIFAR-100 - and real label noise - FOOD-101N - datasets).  For the experiments with synthetic noise, since we have information regarding which targets are noisy or not, we can report the buffer cleanliness percentages at the end of training for both selection techniques. 

**Final Average Accuracy**
| Benchmark   | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. FOOD-101N	|
|-------------|----------------|----------------|----------------|
| Noise rate  |  *asymm*  20            | *asymm* 40             | *real-world* | 
| **OURs**  | **$41.24\pm0.40$** | **$29.2\pm0.91$** | $29.86\pm0.35$ |
| OURs w. **GMM** sample selection   | $39.25\pm0.60$    | $26.8\pm0.90$      | $23.41\pm0.47$|  
  

**Final percentage of clean samples in the buffer**
| Benchmark   | Seq. CIFAR-100 | Seq. CIFAR-100 |
|-------------|----------------|----------------|
| Noise rate  | *asymm*   20            |  *asymm* 40             |
| **OURs**  | **0.74** | **0.96** |
| OURs w. **GMM** sample selection   | 0.66   |  0.89  |

With the results in the tables above, we show that the simple sample selection technique adopted proves more effective at separating clean samples. Nevertheless, sample selection via GMM remains a valuable option to avoid additional hyperparameters.

Reviewer comment (Experiments):
PuriDivER is designed to handle online CL and blurry tasks with noisy labels. Therefore, it may not be suitable for offline CL. However, both online CL and blurry tasks are more challenging and realistic settings, and I’m curious how well the proposed method AER works under these settings as well. In particular, how to adapt AER to online CL where the distribution changes rapidly?

Author response:
**Weakness 3**:   
Despite the reviewer considering the offline setting unrealistic and negligibly important, we would like to clarify that our experimental setting aligns with many notable works [PASS, DER, LIP, ICARL, LUCIR, CO2L, LSIL, PODNET]. 
In this respect, we advocate for what has been said in [DER]: when only one epoch is allowed on the current task, even the SGD baseline fails at fitting it with adequate accuracy, especially with complex datasets such as CIFAR-100 and FOOD-101N. Therefore, the resulting performance - and in turn the comparisons among different approaches - can be difficult to read, as the effects of catastrophic forgetting and those linked to underfitting interleave here. 

Since in our work Alternated Replay is activated/deactivated every other epoch, the method does not fit the online training scenario. 

[PASS]: Zhu et al. Prototype augmentation and self-supervision for incremental learning. CVPR 21.  
[DER]: Buzzega et al. Dark experience for general continual learning: a strong, simple baseline. aNeurIPS 20.  
[LIP]: Bonicelli et al. On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning. aNeurIPS 21.  
[ICARL]: Rebuffi et al. icarl: Incremental classifier and representation learning. CVPR 17.  
[LUCIR]: Hou et al. Learning a unified classifier incrementally via rebalancing. CVPR 19.  
[CO2L]: Cha et al. Co2L: Cooperative continual learning for lifelong visual recognition. ICCV 21.  
[LSIL]: Wu et al. Large Scale Incremental Learning CVPR 19.  
[PODNET]: Douillard et al. 8: Pooled Outputs Distillation for Small-Tasks Incremental Learning ECCV 20.

Reviewer comment (Experiments):
The comparison methods cannot demonstrate the real effectiveness of AER and ABS. Since few methods focus on offline CL with noisy labels, the authors should conduct more detailed comparative experiments. For example, using the same ABS strategy, compare the performance of AER and DividMix to verify the excellence of AER. In addition, using the same AER strategy, the performance of ABS and PuriDivER sampling methods should also be compared.

Author response:
**Weakness 4**:   
We here report some comparative experiments on the contribution of our AER and ABS on the two methods cited by the reviewer (for both CIFAR-100 and NTU-60 datasets). 

We observe from the following tables that for both DivideERMix and PuriDivER, the addition of AER and ABS has always a positive impact on performance. The increase is more noticeable in PuriDivER than in DividERmix, thus supporting our choice of having it as our main competitor.

| **Benchmark**          |  **Seq. CIFAR-100**     | **Seq. CIFAR-100** |    **Seq. CIFAR-100**   |
|------------------------|-----|---------------------|-----|
| **Noise rate (symm)**  | 20% | 40%                 | 60% |
| **OURs** *w. AER w. ABS* | $47.58$ | $41.58$ | $30.13$ |
| **DividERMix** | $29.21$ | $22.41$ | $14.21$ |
|   *w. AER* | $32.63$ | $24.52$ | $14.35$ |
|   *w. ABS* | $32.95$ | $25.35$ | $14.43$ |
| **PuriDivER** | $27.53$ | $24.36$ | $17.81$ |
|   *w. AER* | $28.90$ | $24.68$ | $23.00$ |  

| **Benchmark**                  | **Seq. NTU-60** | **Seq. NTU-60** |
|--------------------------------|-|-|
| **Noise rate (symm)**          | 20% | 40% |
|  **OURs** *w. AER w. ABS* | $49.59$ | $48.18$ |
| **DividERMix**                | $32.61$ | $20.23$ |
|   *w. AER*                    | $32.98$ | $26.39$ |
|   *w. ABS*                    | $38.49$ | $24.92$ |
| **PuriDivER**              | $39.33$ | $38.86$ |
|   *w. AER*                    | $47.39$ | $39.56$ |

Reviewer comment (Experiments):
The main concern is about the performance of the algorithm. The authors conducted experiments on four datasets, three of which were synthesized, and one contained natural noise (WebVision). The discussions and improvements primarily focused on the synthesized datasets. Regarding the WebVision dataset, it remains unclear why the authors selected only 10 classes out of the available 1000 classes, and the criteria for selecting these 10 classes are also not clear. According to Table 2, the AER + ABS approach did not outperform PuriDivER. The proposed method marginally outperformed PuriDivER only with the implementation of consolidation and MixMatch, which were adopted from other papers. It is of great concern that the algorithm can only function effectively in the synthetic scenario.

Author response:
We appreciate the reviewer's suggestion and comments about both strenghts and weaknesses, inviting further discussion to enhance the paper and its evaluation.  
**Weakness 1**:
* Similarly to the approaches in Continual Learning under Noisy Labels (CLN) studies [PuriDivER, SPR], we opt for selecting classes from the WebVision dataset by choosing *the ones containing the largest number of images each*. This decision is motivated by observations from the PuriDivER experiments on the WebVision and Food-101 datasets presented in the original paper, where the method noticeably underfits. Our emphasis is on addressing *forgetting* rather than the underfitting resulting from having numerous classes with varying cardinalities, given the continual learning scenario we are tackling. Following this and other comments from the reviewers, we have incorporated an experiment with **Food-101N**, and now we evaluate two datasets with *real-world noise* We would like to include these additional results in the main paper should the reviewers reconsider their ratings.
| Benchmark   | Food-101N |
|---------------------|-----------|
| Joint   | $39.91$   |
| PuriDivER.ME  | $25.34$   |
|  w. buffer fit. | $29.23$   |
| OURs    | $29.86$  |
|  w. buffer fit. | **34.63** |
   * [SPR]: Kim, Chris Dongjoo, et al. "Continual Learning on Noisy Data Streams via Self-Purified Replay." In ICCV. 2021.  
   [PuriDivER]: Bang, Jihwan, et al. "Online continual learning on a contaminated data stream with blurry task boundaries." In CVPR. 2022. 
* It is important to highlight that PuriDivER inherently includes the buffer fitting stage as a vital aspect of the method, while our AER+ABS does not. Consequently, a fair comparison should be drawn between our method, incorporating the buffer fitting stage, and theirs. It is noteworthy that we have meticulously tailored Puridiver.ME to our offline setting, with consistently superior results compared to PuriDivER. Additionally, we decided to also include the performance of PuriDivER.ME with the *buffer fitting stage disabled*. Results are reported in the following table and we argue these provide a fair comparison between our method and PuriDivER.ME.
Furthermore, the results on the newly introduced real-world noisy dataset (Food-101N) validate the efficacy of our method even in the absence of buffer fitting.  

| Benchmark   | Seq. CIFAR-100 | Seq. CIFAR-100 |
|-------------|----------------|----------------|
| | symm  | asymm |
| Noise rate | 60  | 40   |
| OURs  | $\mathbf{28.60\pm0.55}$ | $\mathbf{31.32\pm1.35}$ |
| PuriDivER.ME (w/o buff. fit)   | $15.56\pm0.46$ | $23.22\pm0.50$ |    

* As for the evaluations they consider lacking, these are precisely documented in the **supplementary materials** provided alongside the main paper. As evident from the results in Tab. E (reported again here), the introduction of each additional feature contributes to a performance improvement, particularly noteworthy when incorporating AER and ABS individually, as well as in combination.  

 Benchmark | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | |
|-----------|----------------|----------------|----------------|----------------|----------------|
| ER-ACE    | $\alpha$  | AER       | ABS       | **FAA**  |
| &check;   |           |           |         | 11.65      |
| &check;    | &check;    |           |       | 19.97      |
| &check;   | &check;    | &check;   |      | 24.19      |
| &check;   | &check;   | &check;   | &check;   | **26.34**  |
| &check;   | &check;    |           | &check;   | 21.68      |  

* We would like to clarify that the consolidation phase of the Puridiver method is not part of our approach. A single experiment was presented in the supplementary material for the purpose of comparison, validating the robustness of our consolidation. Nonetheless, we use this term to denote the technique employed by both PuriDivER and PuriDivER.ME to distinguish between clean and noisy data, subsequently applying pseudo-labeling in a DivideMix fashion.  

* We concur with the reviewer's observation that "The single or multiple training iterations are decided by the offline/online setting, and not something that an algorithm could choose." The references to offline or online settings in the main paper are made to highlight that some competitors were tailored for online Continual Learning (CL), whereas our method is explicitly designed for offline CL. We will enhance the clarity on this aspect throughout the entire paper, with the hope that the reviewer will reconsider their evaluation.

* Following the suggestion of the reviewer, we compare experiments already reported in the main paper of AER and ABS applied to DER++ with PuriDivER relying on DER++, which we name PuriDivDER++. Results are reported in the following table 

| Benchmark | Seq. CIFAR-100 |
|-------------|----------------|
| | symm|
| Noise rate  | 20 |
| DER++ | 35.09 |
|  w.OURs | **43.25** | 
| PuriDivDER++| 35.58|

Reviewer comment (Experiments):
An ablation study of AER and ABS is needed.

Author response:
As for the evaluations they consider lacking, these are precisely documented in the **supplementary materials** provided alongside the main paper. As evident from the results in Tab. E (reported again here), the introduction of each additional feature contributes to a performance improvement, particularly noteworthy when incorporating AER and ABS individually, as well as in combination.  

 Benchmark | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | |
|-----------|----------------|----------------|----------------|----------------|----------------|
| ER-ACE    | $\alpha$  | AER       | ABS       | **FAA**  |
| &check;   |           |           |         | 11.65      |
| &check;    | &check;    |           |       | 19.97      |
| &check;   | &check;    | &check;   |      | 24.19      |
| &check;   | &check;   | &check;   | &check;   | **26.34**  |
| &check;   | &check;    |           | &check;   | 21.68      |

Reviewer comment (Writing):
It is unclear whether there is an explanation of PuriDivEr's consolidation.

Author response:
We would like to clarify that the consolidation phase of the Puridiver method is not part of our approach. A single experiment was presented in the supplementary material for the purpose of comparison, validating the robustness of our consolidation. Nonetheless, we use this term to denote the technique employed by both PuriDivER and PuriDivER.ME to distinguish between clean and noisy data, subsequently applying pseudo-labeling in a DivideMix fashion.

Reviewer comment (Experiments):
It would be beneficial if the authors also compared the proposed method to PuriDivER when applying the algorithm to DER++.

Author response:
Following the suggestion of the reviewer, we compare experiments already reported in the main paper of AER and ABS applied to DER++ with PuriDivER relying on DER++, which we name PuriDivDER++. Results are reported in the following table 

| Benchmark | Seq. CIFAR-100 |
|-------------|----------------|
| | symm|
| Noise rate  | 20 |
| DER++ | 35.09 |
|  w.OURs | **43.25** | 
| PuriDivDER++| 35.58|

Reviewer comment (Presentation):
Some motivation and details of the algorithm are not clear. In Figure 1, the authors aim to demonstrate the change in loss of noisy data with and without replay to highlight that noisy data tends to exhibit greater forgetting. However, it remains unclear in the figure which task the noisy data originates from. Moreover, the depiction of forgetting would be clearer by comparing the accuracy (loss) of the same data from different tasks, rather than comparing the same task with different data or comparing the same data at different stages.

Author response:
We wish to emphasize that the caption of the same figure already specifies that the data originates from the second task. Plotting the same figure at the first task, where the model exclusively replays data from the same task, would have demonstrated behaviours already well-documented and extensively shown in the literature [SSF,toneva2018ans]. Furthermore, our intention was to illustrate loss separation in the continual scenario and that with no countermeasures it becomes impossible to distinguish clean from noisy samples from the **current** task. Therefore, the second task provides the first favourable opportunity to demonstrate that loss separation between clean and noisy data also occurs in a continual learning scenario. We avoid comparing the same data from different tasks as each rehearsal process involves storing data in the buffer upon encountering samples from the current task. Therefore, with Fig. 1(b) we aim to demonstrate how we can leverage forgetting during the current task to more effectively distinguish between noisy and clean data. Overall, Fig. 1 serves to illustrate the intuition behind our AER.

Reviewer comment (Theory):
It is not clear if Eq(1) can represent the optimization goal. If understood correctly, Eq(1) means finding a model to fit the noisy distribution, even when given a wrong label. The loss function yields a smaller value when the model also predicts the wrong label. However, the algorithm seems to eliminate the influence of the noisy label and encourages the model to predict the correct label.

Author response:
We appreciate the reviewer's suggestion about and build upon it to **re-arrange Sec. 3.1** such that the optimization goal is not to encourage the model to learn the wrong labels, which is not our desiderata.

Reviewer comment (Writing):
In Eq(4), it is unclear whether the score function \( s(x) \) is for task \( t \). In this case, why would the buffer contain the current task data?

Author response:
The *reservoir* mechanism employed in most rehearsal methods literature is crafted to produce an i.i.d. representation of the original distribution and insert data from the current task into the buffer while it is in progress. Consequently, we adhere to the same methodology.

Reviewer comment (Theory):
In the motivation of ABS, it is unclear why \( \mathcal{L} \geq 0 \) causes \( s(x) \) to favour samples from the current task.

Author response:
In ABS, the scores are defined as  
$ s(x) = L(x), \text{if}  \quad x \in D_t$  
$\qquad \quad  -L(x) \text{if} \quad x \in D_{< t}$

Such a score denotes the chance for a given example to be replaced. However, before sampling from $s$ we need to normalize the scores, thus we define a normalization factor $z$ = $\sum_{i} s(x_i)$, $\forall$ $x_i$ $\in$ M and compute the normalized scores of current and past task buffer samples as 

$ s(x) = p_{curr}(x) = \frac{L(x)}{z}, \text{if}  \quad x \in D_t$  
$\qquad \quad  p_{past}(x) = -\frac{L(x)}{z} \text{if} \quad x \in D_{< t}$  
thus, since  $\mathcal{L}(\mathbf{x}) \geq 0,  \forall \mathbf{x}$, it follows that $\frac{\mathcal{L}(\mathbf{x})}{z} \geq -\frac{\mathcal{L}(\mathbf{x})}{z}$ .

We thank the reviewer for their concern and following their feedback we will include this detailed explanation in the final revision.

Reviewer comment (Writing):
In the ABS part, it is unclear what \( p_{curr} \) and \( p_{past} \) are.

Author response:
In ABS, the scores are defined as  
$ s(x) = L(x), \text{if}  \quad x \in D_t$  
$\qquad \quad  -L(x) \text{if} \quad x \in D_{< t}$

Such a score denotes the chance for a given example to be replaced. However, before sampling from $s$ we need to normalize the scores, thus we define a normalization factor $z$ = $\sum_{i} s(x_i)$, $\forall$ $x_i$ $\in$ M and compute the normalized scores of current and past task buffer samples as 

$ s(x) = p_{curr}(x) = \frac{L(x)}{z}, \text{if}  \quad x \in D_t$  
$\qquad \quad  p_{past}(x) = -\frac{L(x)}{z} \text{if} \quad x \in D_{< t}$  
thus, since  $\mathcal{L}(\mathbf{x}) \geq 0,  \forall \mathbf{x}$, it follows that $\frac{\mathcal{L}(\mathbf{x})}{z} \geq -\frac{\mathcal{L}(\mathbf{x})}{z}$ .

We thank the reviewer for their concern and following their feedback we will include this detailed explanation in the final revision.

Reviewer comment (Writing):
The presentation of the part of the method requires more details. For example, what does the task D_t = (X_t, Y_t) mean? Does it mean a dataset or distribution? And what does D_t ∩ M mean in eq.(4)?

Author response:
**Weakness 1**: In our paper, $D_t$ was the set of data belonging to the specific $t^{th}$ task, containing $(X_t , Y_t)$ tuples of image and label. Consequently, in Eq. (4),  $D_t \cap M$ is the intersection between current task data and buffer data, so $ (x_t,y_t) \sim D_t \cap M$ stands for data belonging to both sets.
We acknowledge that the $\textit{problem setting}$ is crucial for method comprehension, and we understand the importance of providing detailed information for those who have no knowledge about the continual learning setting. Following the reviewer's suggestion, we therefore rearranged Sec. 3.1 to enhance comprehensibility and now treat $D_t$ as a distribution. Specifically, the new version of the sentence is now  "We formalize the problem of **Continual Learning under Noisy labels** (**CLN**) as learning from a sequence of $T$ tasks. During each task $t \in \{0, 1, \dots, T\}$, input samples $\mathbf{X_t}$ and their corresponding annotations $\mathbf{Y_t}$ are drawn from an i.i.d. distribution $\mathcal{D}_t$."

Reviewer comment (Evaluation):
The experiments only provide an average accuracy for all the tasks and do not prove the proposed method can reduce the forgetting of past tasks directly.

Author response:
**Weakness 2**: We have opted for the most commonly used forgetting metric in the Continual Learning field (see Eq.(1) of the submitted supplementary material), applying this measure to each conducted experiment. As mentioned in Sec. 4.1 of the main paper, we kindly invite the readers to examine the **supplementary materials** provided alongside the main paper, as the evaluations they consider lacking are documented there. Moreover, as reported in Tab. D of the supplementary material, our method outperforms the others in terms of **Final Forgetting** metrics.

Reviewer comment (Experiments):
How does the size of the buffer impact the performance?

Author response:
**Question (b)**: Regarding the question about how the size of the buffer impacts the performances, we argue that in a scenario where it is crucial to maintain a clean buffer for accurate memory of past tasks, having a very large buffer, as is common in cases aimed at mitigating forgetting, might have a small impact on performance due to the poisoning effect of noise in the data stored. Instead, in such a scenario it is far more effective to extract a smaller amount of clean samples. To demonstrate this, we conduct a small evaluation of the hardest scenarios of the main table to investigate the effect of different buffer sizes. Such an evaluation shows that the second-best methods we evaluated (i.e., our ad-hoc version of PuriDivER, PuriDivER.ME), cannot effectively take advantage of the increased buffer, whereas our proposal can, and remains competitive across the board. The Final Average Accuracy results to assess the just-mentioned evaluations are reported in the following table.  
| Benchmark          | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 | Seq. CIFAR-100 |
|--------------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|
| Buffer size        | 500 | 500 | 500 | 2000 | 2000 | 2000 | 5000 | 5000 | 5000 |
| | symm | symm | asymm | symm | symm | asymm | symm | symm | asymm |
| Noise rate         | 40% | 60% | 40% | 40% | 60% | 40% | 40% | 60% | 40% |
| PuriDivER.ME       | $23.72\pm0.48$ | $18.87\pm0.83$ | $20.40\pm0.98$ | $37.61\pm0.85$ | $27.18\pm0.76$ | $30.22\pm0.74$ | $39.48\pm1.2$ | $22.62\pm0.71$ | $32.07\pm1.18$ |
| OURs               | $27.48\pm0.35$ | $17.50\pm0.55$ | $21.62\pm0.48$ | $38.64\pm0.57$ | $26.34\pm0.85$ | $29.26\pm0.91$ | $45.12\pm0.30$ | $33.80\pm0.37$ | $\mathbf{35.38\pm1.31}$ |
| w. buffer fit.     | $\mathbf{30.32\pm0.97}$ | $\mathbf{19.24\pm0.74}$ | $\mathbf{22.66\pm0.80}$ | $\mathbf{41.58\pm0.63}$ | $\mathbf{30.13\pm0.98}$ | $\mathbf{31.49\pm0.38}$ | $\mathbf{47.63\pm0.66}$ | $\mathbf{35.46\pm0.53}$ | $34.61\pm1.04$ |

Reviewer comment (Experiments):
When the length of a sequence is increased, more informative data is added and how to preserve the information from the first several tasks with a smaller size of data in the buffer?

Author response:
**Question (d)**:Most rehearsal methods rely on the $reservoir$ mechanism, which is designed to yield an i.i.d. representation of the original distribution. This, combined with our selection strategy, which guarantees the data is both **clean** and **class-balanced**, leads to strong performances of the method even as the number of tasks increases.