Paper: OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning (ICLR-2024)

Reviewer comment (Experiments):
The main weakness of the paper is that important comparison results in the use of pretrained ViT networks are omitted. The significance of the contribution in this paper would be much easier to appraise if these are added by the authors.

Author response:
**W1.1 First, there are two omitted relevant papers that use alternative methods to prompting.**

Thank you for pointing it out those relevant and contemporaneous works, we add the discussion of those two papers [Zhang et al., 2023][McDonnell et al., 2023] in the related works section.
Moreover, we would like to point out one experimental setting in their papers, even though they are using ViT as the backbone but they are actually using different pre-trained checkpoints from ours. Thus, it makes harder to directly compare the performance to their results.

Given the time of rebuttal, it is beyond our computing resource to re-run all experiments based on another checkpoint for ViT given that we have conducted the suggested experiments on more datasets and more ablation studies.
Nonetheless, we think how to combine the virtual outliers to prototype-based method will be an interesting research direction.

References:

- Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, Yunchao Wei, SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model, ICCV 2023
- Mark D. McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, Anton van den Hengel. RanPAC: Random Projections and Pre-trained Models for Continual Learning, NeurIPS 2023.

Reviewer comment (Experiments):
There are two omitted relevant papers that use alternative methods to prompting, both producing higher final average accuracies using pre-trained ViT models than any reported here. The first of these is the SLCA method [1]. The second is the RanPAC method [3] that builds on the work of the paper by Zhou et al [2], which is already cited in the paper. In the current paper, OVOR on ImageNet-R achieves 75.6% for T=10, and CIFAR-100 achieves 86.7% for T=10. In ref [1], ImageNet-R achieves 77% for T=10 and CIFAR-100 achieves 91.53% for T=10. In ref [3], ImageNet-R achieves 78.1% for T=10 and for CIFAR100 achieves 92.2% for T=10 (T=5 and T=20 results can also be seen in Table A4 of [3]).

Author response:
**W1.1 First, there are two omitted relevant papers that use alternative methods to prompting.**

Thank you for pointing it out those relevant and contemporaneous works, we add the discussion of those two papers [Zhang et al., 2023][McDonnell et al., 2023] in the related works section.
Moreover, we would like to point out one experimental setting in their papers, even though they are using ViT as the backbone but they are actually using different pre-trained checkpoints from ours. Thus, it makes harder to directly compare the performance to their results.

Given the time of rebuttal, it is beyond our computing resource to re-run all experiments based on another checkpoint for ViT given that we have conducted the suggested experiments on more datasets and more ablation studies.
Nonetheless, we think how to combine the virtual outliers to prototype-based method will be an interesting research direction.

References:

- Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, Yunchao Wei, SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model, ICCV 2023
- Mark D. McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, Anton van den Hengel. RanPAC: Random Projections and Pre-trained Models for Continual Learning, NeurIPS 2023.

Reviewer comment (Experiments):
A clear distinction between refs [1,2,3] and the current paper is that [1,2,3] report results on many more datasets, whereas here only two are used: split CIFAR-100 and split ImageNet-R. The current paper's contribution and standing relative to refs [1,2,3] would be much clearer if results for OVOR are reported on at least 4 datasets. For example, [1] also reports results on split CUB and split Cars, while [2] and [3] each report on 7 datasets for CIL in total. There is no good reason to not demonstrate OVOR's performance on at least 4, and preferably 6 datasets such as those in [1,2,3]. This is particularly important given that table 2 shows that OnePrompt alone produces lower final average accuracy than CODA-P, and only the combination of OnePrompt with VOR does better than CODA-P.

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Experiments):
The recommendation to add more datasets is independent of any date comparisons between papers, even if [3] and possibly [1] are deemed "contemporaneous" according to unclear FAQ rules. Zhou et al's other datasets could have been used to add more dataset comparisons to help the reader understand whether there truly is a trend for the method to outperform CODA-Prompt, DualPrompt, etc.

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Experiments):
The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR. While ablations for OnePrompt are shown for ImageNet-R in Table 2, there is no data for CIFAR-100. To understand the contributions of OnePrompt and VOR both individually and combined, data for ablations on more than just a single dataset are needed.

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Experiments):
Given the authors state "The proposed virtual outlier regularization (VOR) is agnostic to any prompt-based CIL," it is of high interest to know how well VOR combines with CODA-P, especially since CODA-P without VOR outperforms OnePrompt without VOR.

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Theory):
The authors then make a statement about privacy concerns, which is a totally independent point from the distinction between class prototypes and rehearsal memory. The reviewer is skeptical about privacy concerns regarding a class-mean vector, as it seems improbable that original samples could be extracted from such a vector aggregated over many samples.

Author response:
**W3. On page 3, in the second paragraph, the authors refer to the prototype methods...**

Thank you for the remarks, we removed the pertinent two sentences in the revision, because it seems that they are giving the impression that we do not intend to (to justify that prompt-based methods are preferred over prototype-based methods).
Nonetheless, we would like to discuss more thoroughly about certain parts in the reviewer's comments.

First of all, the reviewer mentioned that rehearsal means "retaining copy of full original data sample", but for works that learn a generative model from past data [Ostapenko et al., 2019][van der Van et al., 2020], a.k.a. pseudo-replay or deep generative replay, they are still considered as rehearsal-based methods [Smith et al., 2023a][Mai et al., 2022].
Of course, this by no means imply that prototype-based methods are not rehearsal-free, and it is also true that the prototypes are not used along with new inputs to update the model.
To us it is more like a spectrum, depending on how much information about past data is stored, with original replay-based on one end and having only classifier weights on the other, and between them there are pseudo-replay (storing generative models), SLCA (storing class distributions), prototype-based (storing class means).
We updated the discussion of prototype-based method in the related works.

Next, regarding the privacy concerns, we totally agree that class-means contain very little information compared to replay buffers.
The original samples cannot be extracted from class means alone, but that may not be the case when we have class means along with the encoder weights.
In fact, there exists work [Haim et al., 2022] that tries to reconstruct original data from the classifier, though it is done on simpler networks and smaller datasets, hence there is a long way for this to become a pressing issue for networks and datasets as large and complex as we are using now.
Nonetheless, it demonstrates the possibility of extracting training samples, even without prototypes.
And certainly storing prototypes would provide extra information on top of that.
Hence in our view, similar as aforementioned, there is a spectrum with these methods, where storing prototypes provides more information about past data than without, which makes it easier to achieve better CL performance, but also for reconstruction of past data samples.

Reference:

- Oleksiy Ostapenko, Mihai Marian Puscas, Tassilo Klein, Patrick J ̈ahnichen, and Moin Nabi.
Learning to remember: A synaptic plasticity driven framework for continual learning. In
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA,
USA, June 16-20, 2019, pp. 11321–11329.
- Gido M. van de Ven, Hava T. Siegelmann, and Andreas Savas Tolias. Brain-inspired replay for
continual learning with artificial neural networks. Nature Communications, 11, 2020.
- James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim,
Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de-
composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909–11919,
June 2023a.
- Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. On-
line continual learning in image classification: An empirical survey. Neurocomputing, 469:
28–51, 2022.
- Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, Michal Irani. Reconstructing Training Data from Trained Neural Networks, NeurIPS 2022

Reviewer comment (Novelty):
The inclusion of these statements appears to be an attempt to justify why a prompt-based CIL method should be preferred over a prototype method like Zhou et al. The reviewer believes such justification is unnecessary. Regardless of whether prototype methods may or may not achieve better results than prompting methods, the current paper's advantages over past prompting methods remain potentially important because (i) there may be benefits from combining prompting and prototype methods, and (ii) future improvements in ViT transformer networks may benefit more from prompting than from prototype methods.

Author response:
**W3. On page 3, in the second paragraph, the authors refer to the prototype methods...**

Thank you for the remarks, we removed the pertinent two sentences in the revision, because it seems that they are giving the impression that we do not intend to (to justify that prompt-based methods are preferred over prototype-based methods).
Nonetheless, we would like to discuss more thoroughly about certain parts in the reviewer's comments.

First of all, the reviewer mentioned that rehearsal means "retaining copy of full original data sample", but for works that learn a generative model from past data [Ostapenko et al., 2019][van der Van et al., 2020], a.k.a. pseudo-replay or deep generative replay, they are still considered as rehearsal-based methods [Smith et al., 2023a][Mai et al., 2022].
Of course, this by no means imply that prototype-based methods are not rehearsal-free, and it is also true that the prototypes are not used along with new inputs to update the model.
To us it is more like a spectrum, depending on how much information about past data is stored, with original replay-based on one end and having only classifier weights on the other, and between them there are pseudo-replay (storing generative models), SLCA (storing class distributions), prototype-based (storing class means).
We updated the discussion of prototype-based method in the related works.

Next, regarding the privacy concerns, we totally agree that class-means contain very little information compared to replay buffers.
The original samples cannot be extracted from class means alone, but that may not be the case when we have class means along with the encoder weights.
In fact, there exists work [Haim et al., 2022] that tries to reconstruct original data from the classifier, though it is done on simpler networks and smaller datasets, hence there is a long way for this to become a pressing issue for networks and datasets as large and complex as we are using now.
Nonetheless, it demonstrates the possibility of extracting training samples, even without prototypes.
And certainly storing prototypes would provide extra information on top of that.
Hence in our view, similar as aforementioned, there is a spectrum with these methods, where storing prototypes provides more information about past data than without, which makes it easier to achieve better CL performance, but also for reconstruction of past data samples.

Reference:

- Oleksiy Ostapenko, Mihai Marian Puscas, Tassilo Klein, Patrick J ̈ahnichen, and Moin Nabi.
Learning to remember: A synaptic plasticity driven framework for continual learning. In
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA,
USA, June 16-20, 2019, pp. 11321–11329.
- Gido M. van de Ven, Hava T. Siegelmann, and Andreas Savas Tolias. Brain-inspired replay for
continual learning with artificial neural networks. Nature Communications, 11, 2020.
- James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim,
Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de-
composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909–11919,
June 2023a.
- Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. On-
line continual learning in image classification: An empirical survey. Neurocomputing, 469:
28–51, 2022.
- Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, Michal Irani. Reconstructing Training Data from Trained Neural Networks, NeurIPS 2022

Reviewer comment (Writing):
Minor point: the paper's introduction assumes the reader understands that the approaches in the paper apply only to pre-trained ViT networks. In the larger Continual Learning community, this is still unusual, as most CL methods focus on training CNNs such as ResNets and mitigating forgetting without the benefits of a pre-trained model. The paper would benefit from clarifying these points early on.

Author response:
**W5. The paper’s intro assumes the reader understands well that the
approaches in the paper apply only to pre-trained ViT networks. For the larger
Continual Learning community, this is still unusual as most methods for CL
focus on training CNNs such as ResNets, and on mitigating forgetting when the
benefits of a pre-trained model are not available.**

Thanks for the suggestion, we have made changes in Section 2 and 3.2 when CODA-Prompt, DualPrompt and L2P are mentioned.
Nonetheless, we would like to clarify that VOR can also possibly be applied to networks other than ViT.
As it uses features produced by the encoder to regularize the classifier head, there is no dependency on network architecture of the encoder, or whether it is pre-trained.

Reviewer comment (Experiments):
Summary requirement for reconsidering rating: The authors need to (i) provide a clearer comparison of OVOR against other methods on more datasets, ideally including comparison with refs [1,3], and (ii) show thorough ablations compared with the best competing prompting method, CODA-P (CODA-P with and without VOR, and OnePrompt with and without VOR), all on at least 4 datasets.

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Experiments):
Question: What is the performance of VOR when combined with L2P, DualPrompt, and CodaPrompt, and how does that compare with OVOR?

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Experiments):
Question: What is the performance of OVOR, and of ablations (VOR alone and OnePrompt alone), on at least 2 more datasets?

Author response:
**W1.2 The current paper's contribution and standing relative to refs would be much clearer if results for OVOR are reported on at least 4 datasets.** 

**W2. The paper does not thoroughly explore ablation results with regards to the combination of OnePrompt or other prompting methods with VOR.**

Thanks for the suggestion, we address these two points together (and two questions). We further compare the performance of VOR over the prompt-based method, including L2P, DualPrompt, CODA-P and OnePrompt on ImageNet-R, CIFAR-100, CUB-200 and ImageNet-A, and the results are shown in the following table. The proposed VOR consistently improve all prompt-based methods on all datasets, which shows the VOR could be used in many types of datasets. We updated the results in Table 2 of the revision. 

**Table**: The CIL accuracy on four datasets under the 10-task setting. The number in the parenthesis denotes the results with VOR.

| Method | ImageNet-R | CIFAR-100 | CUB-200 | ImageNet-A |
| --- | --- | --- | --- | --- |
| L2P | 69.31 (71.14) |  82.43 (83.87) | 70.62 (76.33) | 40.16 (41.27) |
| DualPrompt | 71.47 (73.27) | 83.28 (84.85) | 71.85 (78.44) | 44.85 (45.45) |
| CODA-P | 74.87 (76.82) | 86.41 (86.53) | 72.74 (78.72) | 50.20 (51.04) |
| OnePrompt | 73.34 (75.61) | 85.60 (86.68) | 72.19 (78.12) | 47.36 (48.49) |

Reviewer comment (Theory):
In multiple parts of the paper, it is mentioned that prompt-based approaches require a pool of task-specific prompts. Although I understand the idea behind it, this is only partially true, as the authors say in section 3.2. They explain that the selection depends on a function; ideally, one would expect this feature to be able to be separated by task, but it is not a requirement of the methods.

Author response:
It is true that the prompt selection can be non-task-specific, e.g. CODA-Prompt uses an attention mechanism to obtain a weighted combination of all prompts.
Nonetheless, the learning of those prompts is not. Again in CODA-Prompt, the prompt pool divided into a number of partitions, and each partition will only be updated by data from one task, and frozen afterwards.
To say in very high level way, what benefit having multiple prompts brings is the ability to isolate knowledge from different tasks, otherwise it can viewed as a single prompt updated continuously, just with more parameters.
This is why we refer them as "task-specifc" prompt pool.

Reviewer comment (Presentation):
The structure of the paper needs to be clarified. For example, why present Virtual Outlier Regularization first rather than OnePrompt? At least for me, OnePrompt is like a simple baseline. It's a good one, as it shows in the results, but only a base method to be compared.

Author response:
The inter-task confusion is the most important issue in all rehearsal-free methods, and the proposed VOR is a general solution for the prompt-based methods (as shown in Table 2); thus, the VOR is our main contribution, and then we introduce it first.
After having VOR to help different prompt-based methods, we further explore different prompt-based methods instead of directly adopting their prompt-based methods, and we find that the simple OnePrompt achieved superior performance.
With this regard, we believe that it is better if OnePrompt is not presented before VOR, breaking the aforementioned narrative.

Reviewer comment (Experiments):
Have you run experiments when training only the classifier? Together with using the approach mentioned in Eq 3.

Author response:
[Smith et al., 2023] has conducted this experiment in their paper, where it is called "FT++".
We quote those numbers in the below table and compare to OVOR.
When only the classifier is trained, the resulting accuracy is significantly worse than OVOR.


**Table**: Performance between FT++ and OVOR on different task numbers of ImageNet-R and CIFAR100.

| | 5-task ImageNet-R | 10-task ImageNet-R | 20-task ImageNet-R | 10-task CIFAR100 |
|---|---|---|---|---|
| FT++ | 60.42 | 48.93 | 35.98 | 49.91 |
| OVOR | 76.79 | 75.61 | 73.13 | 86.68 |

Reference:

- James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim,
Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de-
composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909–11919,
June 2023.

Reviewer comment (Evaluation):
Is there an explanation for why the VOR increases forgetting? As shown in Table 2.

Author response:
In general, a method with higher accuracy might increase forgetting since the method have more room to forget.
Although VOR increases forgetting, its increases in accuracy are larger than forgetting.
As shown in the below table, where we report the start and final accuracy for each task on 10-task ImageNet-R.
OVOR on average has higher starting accuracy than OnePrompt, which is used to compute forgetting, resulting in higher forgetting.
Despite of this, OVOR is generally having higher final accuracy for each task as well.

**Table**: Starting and final accuracy of OnePrompt and OVOR for each task on 10-task ImageNet-R.

| Task index |1|2|3|4|5|6|7|8|9|10|
|---|---|---|---|---|---|---|---|---|---|---|
|OnePrompt (start)|91.90\%|79.75\%|83.18\%|72.89\%|81.29\%|79.56\%|72.17\%|75.31\%|73.94\%|71.13\%|
|OnePrompt (final)|82.14\%|69.00\%|77.94\%|69.10\%|75.00\%|75.75\%|71.27\%|73.90\%|72.57\%|71.13\%|
|OVOR (start)|92.27\%|82.68\%|83.73\%|78.86\%|85.00\%|84.31\%|75.58\%|82.23\%|73.08\%|71.13\%|
|OVOR (final)|79.93\%|68.72\%|78.12\%|74.05\%|79.68\%|81.30\%|74.15\%|81.60\%|72.57\%|71.13\%|

Reviewer comment (Experiments):
In Section 5, you mentioned that for OnePrompt, you insert the prompts into the first 5 layers. Did you do an ablation of this hyperparameter?

Author response:
Thanks for suggesting this ablation study. To fairly compare with previous methods, we follow the design of DualPrompt to insert the prompts without task-specific prompts. 
Here, we experimented with different configurations of prompts insertion and the results on 10-task ImageNet-R are shown in the following table (Table B.16 in the revision).

**Table**: Ablation study of different prompt lengths and locations for OnePrompt.

| Length 5 (start, end) | Length 20 (start, end) | Accuracy |
|:---:|:---:|:---:|
|(1,2)|(3,5)|73.34\%|
|(1,12)| - |73.92\%|
| - |(1,12)|74.69\%|
|(3,4)|(3,5)|73.46\%|
|(5,6)|(3,5)|73.41\%|
|(6,7)|(3,5)|73.89\%|
|(1,2)|(2,4)|72.89\%|
|(1,2)|(4,6)|73.55\%|
|(1,2)|(5,7)|73.93\%|

The default option, which is the first row, is outperformed by many others, but except for the second and third one (inserting same-length prompts to all layers), the gains are negligible.
The first row is used as the default because it is same as DualPrompt, and it is easier to compare with DualPrompt when OnePrompt has the exactly same prompt length and position.
In the revision, we further add the results of the second row as OnePrompt-Deep in Table 3.

Reviewer comment (Evaluation):
How costly is the VOR approach?

Author response:
The virtual outlier only needs to be generated once for each task (not each epoch).
As the exact computations depend on the size of the dataset, we measure the cost of VOR by the required extra training time of one task. For 10-task ImageNet-R, with VOR, it only takes extra 1.9\% increase in training time.

Reviewer comment (Presentation):
Do you have the results for OnePrompt for Table 4? In the last 2 paragraphs, you mentioned a comparison between OnePrompt and other methods, however, I don't see it in the tables. Are you referring to OVOR?

Author response:
Yes, we have update that in Table 3 of the revision (Table 4 in the original paper).

Reviewer comment (Theory):
The paper lacks intuition on how the single prompt method works equally or better than the multiple prompt methods.

Author response:
Thank you for pointing it out where readers may find confusing, let us elaborate more on our intuition why OnePrompt can outperform DualPrompt, or on the other hand, why DualPrompt fails to provide better results.  
First, let us recap the design of DualPrompt.  
The core design of DualPrompt is have a prompt set, where each prompt has an 1-to-1 correspondence to one task.  
DualPrompt also design the prompt keys that are used to select proper prompt and the keys are likely converged to the mean representation of one task based on its minimization function.   
During inference, DualPrompt selects a prompt by finding the closest prompt key to the query image and then inserts the prompts to the ViT.  
Then, selecting the correct prompt ideally should be important to classification result for DualPrompt, otherwise it would contradict with the intent of the design.  
Nonetheless, given the composition of one task is purely random, there is no semantic relationship between mean representation of one task to a query image; thus, the matching rate of prompt is pretty low in DualPrompt. In their paper, on the 10-task ImageNet-R dataset, it only achieved **55.8\%** prompt matching rate on all test samples after trained with all tasks. Given the low match rate in DualPrompt, it would be limited how much the task-specific prompt mechanism can help the classification accuracy.

Reviewer comment (Theory):
Is there any mathematical explanation/proof as to why the outlier method works well in tightening the decision boundaries, based on the Gaussian mechanism employed?

Author response:
As illustrated in Figure 1, the virtual outliers are used to prevent the classifier from producing high scores for samples in far away regions from training data in the feature space. Our idea is inspired by [Kim et al. 2022] which shows that if the feature encoder is more robust to out-of-distribution (OOD) samples, it reduces the inter-task confusion of CIL in general. On the contrary, instead of having the robust feature encoder, we use the virtual outlier samples to regularize the classifier head instead. The Gaussian noise added on the outer samples during the virtual outlier synthesis can be viewed as to guide the decision boundary of classifier head to have a margin from the outer samples. A theoretical proof of how the virtual outliers could tighten the decision boundary is one of our future direction.  

Reference:

- Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. "A Theoretical Study on Solving Continual Learning" NeurIPS 2022