Paper: Time-Sensitive Replay for Continual Learning (ICLR-2024)

Reviewer comment (Reproducibility):
The code is not provided which makes judgment about reproducibility challenging.

Author response:
A link to an anonymous repository containing our code is provided on Page 2 in the final paragraph of the Introduction and in Section E.1 on Page 15.

Reviewer comment (Experiments):
Experiments are performed in synthetic settings using benchmarks that are common but do not have a natural notion of time. Given the proposed setting, why haven't experiments been performed on suitable real-world datasets that have a notion of time?

Author response:
We wanted to remain consistent with theoretical CL work, many of which compare with image datasets constructed to be introduced over time. It would be interesting future work to extend our method for datasets that have time information and where time is important, but doing so would likely require a recurrent architecture that in our opinion falls beyond the scope of this work.

Reviewer comment (Experiments):
Continual learning has a rich literature and there are many works on the topic. What is the justification for selecting the methods that have been used in the paper? Do they have SOTA performance? It is very common in CL literature to provide an extensive comparison.

Author response:
We selected models with state-of-the-art performance for both replay and regularisation. However, we appreciate and agree with the comment that TSR and TF-TSR should be compared with more recent methods, and have provided additional experiments in the revised paper. We provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR).

Reviewer comment (Theory):
Even if we have such validation datasets, evaluating the performance on all previous tasks using the validation data at each time step will introduce a lot of computational complexity, especially when the number of tasks is large. The detection complexity can be much higher than the replay complexity. Even though an analysis of the theoretical complexity is provided in the appendix, it is unclear if the proposed algorithm can achieve better computational complexity than other replay-based methods in theory.

Author response:
Evaluations are required at each batch to monitor for prediction drift over previous tasks. On Pages 13 and 14 of the Appendix, we discuss the theoretical complexity of this process. We appreciate that it is unclear how the replay must be reduced to provide benefit despite consistent validation tests, and so have made revisions in the paper. Furthermore, we have provided additional discussion in this section about the complexity of forward-pass validation tests. We describe the process of sampling from the validation buffer to reduce this complexity further. Moreover, on Page 9, we empirically compare the training time of TSR and TF-TSR with existing baselines and find that despite the expense required to monitor performance over previous tasks, our methods nonetheless achieve a lesser training time.

Reviewer comment (Evaluation):
In the empirical evaluation comparison, even if the proposed method has a shorter training duration, the final accuracy can be, for example, 2\% lower than CEC on MNIST. This somehow indicates that the condition to determine when to replay is not so helpful. In other words, the proposed algorithm does not tell us correctly when to replay, and more data replay clearly improves the performance as in CEC.

Author response:
Even if the mean accuracy may sometimes be lower for our experiments than other full replay models, our results always fall within the error of baseline evaluations and there is never a statistically significant difference in our results. We have made some revisions in our experiments section to emphasise this.

Reviewer comment (Experiments):
Only two baselines (among which only one is replay-based) are considered in the paper, which are also relatively outdated. Considering that there are many advanced replay-based CL methods out there, more baseline methods should be investigated. Specifically, the question here is that if we can select the replay samples wisely, do we still need to care about when to replay? Maybe the answer is still yes, but this investigation will further justify the contribution of this paper.

Author response:
We appreciate and agree with the comment that TSR and TF-TSR should be compared with more recent methods. We provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR). These additional results demonstrate that TF-TSR achieves a classification accuracy that is not significantly different to current state-of-the-art methods with lower time requirements. An opportunity for future work may be to investigate whether sampling algorithms may enhance TSR.

Reviewer comment (Writing):
The writing can be improved and there are multiple typos, e.g., $l_i^T$ should be $l_j^T$ in calculating the moving average, $l_k^{T_1} $ should be $l_k^{T_i}$ in "Model Update and Validation".

Author response:
We agree with the reviewer that there are a couple of oversights in our writing. We have corrected these oversights in the revised paper.

Reviewer comment (Experiments):
A considerable limitation of the paper is that it ignores cases where the data distribution changes continuously over time. In such cases, task boundaries cannot be defined or detected. In my opinion, most real-world applications of continual learning do not have task boundaries, since the distribution is constantly changing (e.g., weather, climate, stock market, hospital occupancy, etc.).

Author response:
> A considerable limitation of the paper is that it ignores cases where the data distribution changes continuously over time. In such cases, task boundaries cannot be defined or detected. In my opinion, most real-world applications of continual learning do not have task boundaries, since the distribution is constantly changing.

We agree that it is important to consider changing distributions in a data stream for continual learning settings. We intended that distribution changes are represented as tasks in our task-sequential setting, where examples may abruptly shift throughout the data stream. We also employed synthetic datasets to adjust the distributions and how distributions change over each benchmark image dataset. These synthetic datasets are described on Page 6 and used on Pages 8 and 9, where we observe that TF-TSR can successfully detect tasks even under these more challenging settings. We appreciate that this should be made clear in the paper, and so have made revisions to the definition of a task on Page 3 and have added a further description of synthetic datasets in Page 18 in the Appendix.

Reviewer comment (Experiments):
I am not sure if the comparison of TSR and TF-TSR with other baselines is fair. TSR requires the storage of validation data, while other methods do not necessarily do so, and I could not find any part of the text that explains whether this discrepancy was taken into account in the evaluation.

Author response:
> Can you explain if and how you took into account the size of the validation memory in the evaluation?

We agree that a pre-established validation buffer severely impacts the practicality of the TSR method. Similarly to TSR’s reliance on a knowledge of task information, in most real-world cases, it is unrealistic to expect a priori samples of a data stream. Both of these limitations are addressed in the TF-TSR method. Page 5 of our paper contains a paragraph on the TF-TSR Validation and Replay Buffer changes. Here, we acknowledge that a validation set cannot be expected prior to training, so validation examples must be stored in memory as new data arrives. In this instance, there is no additional information or data required by TF-TSR prior to training. We appreciate that this aspect should be highlighted more in the paper, and so we have made adjustments in this section. We provide further experiments varying the size of the replay buffer in the Appendix on Page 18.

Reviewer comment (Experiments):
Why did you not compare with other very simple and efficient baselines (e.g., simple experience replay)?

Author response:
> Why did you not compare with other very simple and efficient baselines (e.g., simple experience replay)?

We thank the reviewer for pointing out this oversight. Our perspective was that comparing with more recent and superior replay methods was sufficient. We have provided additional experiments to remedy this. In addition to our current baselines, we provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR). These additional results demonstrate that TF-TSR achieves a classification accuracy that is not significantly different to current state-of-the-art methods with lower time requirements.