Paper: Active Continual Learning: On Balancing Knowledge Retention and Learnability (ICLR-2024)

Reviewer comment (Presentation):
The choice of some visualizations in the paper is very odd. For example, in Figures 3 and 4, a dashed red line is selected to represent the performance on the "full labelled dataset". But, why is a line used to connect in between methods (x axis)? What is the meaning of this? Similarly, all the dots representing different strategies make it very difficult to grasp what is the actual performance of each of the selected methods with each of those strategies. I would strongly suggest to find a much better representation.

Author response:
Thanks for the comment. We aimed to establish the baseline to compare between CL with full labelled dataset and CL with AL - active continual learning (ACL). We re-plot Figure 3 and 4 to report the relative performance of ACL and the baseline. We hope they are better visualised.

Reviewer comment (Experiments):
Although the selected datasets are CL benchmarks, these are also the easiest ones. I would have expected to see experimental results on more challenging datasets such as CIFAR-100, some version of ImageNet (tiny-ImageNet, mini-ImageNet), etc. Furthermore, from Figure 3 and 4, it seems that for the slightly more challenging datasets and tasks (P-MNIST, CIFAR-10 CIL, TIL), all the ACL methods perform substantially badly, therefore raising questions on the actual effectiveness of CL combined with active learning, and the significance of the proposed approach.

Author response:
We would like to emphasize that our goal is not to show that ACL is better than CL. In fact, we propose a practical scenario where there is a need to label data for CL. Our work empirically show that simply combining existing CL and AL is not optimal due to the trade-off between forgetting and learnability. We hope that our research will give rise to new avenues of research, prompting the community to further research regarding the development of ACL methods capable of effectively harmonizing the dual objectives of low forgetting and quick learnability.

Reviewer comment (Experiments):
A final remarkable weakness that I see in this paper is the limited number of tasks in the experiments. I would expect that active selection of examples would be significantly more difficult as the number of tasks increases, and therefore would have expected to see results along these lines.

Author response:
We evaluate on both text classification and image classification tasks which cover different CL scenarios (domain-IL, class-IL and task-IL) and having different number of tasks: MNIST (5 tasks), CIFAR-10 (5 tasks), ASC (19 tasks) and 20News (10 tasks).

Reviewer comment (Novelty):
The paper does not seem to have a high novel approach, but mostly evaluate the current AL and CL approach.

Author response:
We would like to emphasize that our contributions extend well beyond introducing a formulation for ACL: 
- We introduce the analysis of the forgetting-learning profile to study the trade-off between low forgetting and quick learnability objectives in the context of ACL.
- While this trade-off has been previously studied in the CL literature, our empirical findings confirm the existence of this trade-off in the simple combination of the AL and CL methods. 
- We conduct comprehensive experiments on combining several representative CL and AL methods in domain, class, and task incremental learning. The findings provide insights into the benefit of ACL (sequential vs independent labelling) and the selection of AL and CL methods in each CL learning scenario.

Our findings give rise to new avenues of research, prompting the community to further research regarding the development of ACL methods capable of effectively harmonizing the dual objectives of low forgetting and quick learnability.

Reviewer comment (Experiments):
The evaluation seems only on P-MNIST, S-MNIST, CIFAR-10, maybe small sets.

Author response:
We evaluation on both text classification and image classification tasks which covering different CL scenarios (domain-IL, class-IL and task-IL) and having different number of tasks: MNIST (5 tasks), CIFAR-10 (5 tasks), ASC (19 tasks) and 20News (10 tasks).