Paper: One-stage Prompt-based Continual Learning (ICLR-2024)

Reviewer comment (Experiments):
Assumption on the pretrained network. One limitation of existing PCL methods is that they rely on supervised pretraining on ImageNet1k, which has a large impact on the model performance but can be infeasible for a general CL task. Does this method still work without using supervised pretraining? For instance, what if replacing the supervised pretraining model with an unsupervised pretraining model (e.g. DINO [1])?

Author response:
Thank you for the valuable feedback. We think the reviewer's suggestion is interesting and important for PCL works. In response to this suggestion, we use DINO pre-trained weights instead of ImageNet-1k pre-trained weights, and compare the performance across ER, LwF, and PCL works. We set the other experimental settings as identical.

| Method (DINO pre-trained method) |    Acc (%)    | Forgetting Score |
|----------------------------------|:-------------:|:----------------:|
| ER                               |  60.43 ± 1.16 |   13.30 ± 0.12   |
| LwF                              | 62.73 ± 1.13  |    4.32 ± 0.63   |
| L2P                              |  60.32 ± 0.56 |    2.30 ± 0.11   |
| DualPrompt                       |  61.77 ± 0.61 |    2.31 ± 0.23   |
| CodaPrompt                       |  67.61 ± 0.19 |    2.23 ± 0.29   |
| OS-Prompt                        |  67.52 ± 0.34 |    2.32 ± 0.13   |
| OS-Prompt++                      |  67.92 ± 0.42 |    2.19 ± 0.26   |

The non-PCL methods (ER and LwF) demonstrate similar or even superior performance compared to DualPropmt and L2P. This contrasts with the results obtained using the ImageNet-1k pre-trained model, where PCL methods outperformed non-PCL methods. This observation suggests that the backbone model plays a crucial role in PCL. However, CodaPrompt and our OS-Prompt outperform the other methods by a significant margin. This indicates that our method continues to perform well even without utilizing supervised pertaining. 

We have included the experiments and analysis related to the above observations in the Appendix.

Reviewer comment (Evaluation):
Lack of clarity in experimental results. In particular, Figure 2 is not very clear. Is the token embedding distance depicted in this Figure related to the current task? What are the distances between token embeddings for tasks 1 to t-1 when task t is learned? Such an illustration could provide more insights into how the method alleviates catastrophic forgetting.

Author response:
Thanks for the insightful suggestion. In response to the reviewer's feedback, we have restructured Fig. 2 in our manuscript. This figure depicts the measurement of distances between token embeddings of previous tasks when a new task is learned. We conducted experiments on 9 pairs, assessing the distance between token embeddings of task 1 and those after tasks {3, 5, 7, 9}; the distance between token embeddings of tasks 3 and those after tasks {5, 7, 9}; and the distance between token embeddings of tasks 5 and those after tasks {7, 9}. The results reveal that token embeddings from early layers show minimal changes during training. Consequently, utilizing token embeddings from these earlier layers (layers  1~5) provides a stable and consistent representation throughout the continual learning stages.

Reviewer comment (Experiments):
Potential Scalability Concerns: While the OS-Prompt framework demonstrates efficiency improvements on benchmarks like CIFAR-100 and ImageNet-R, the paper does not provide insights into how this method scales with larger, more complex datasets. This leaves questions about its applicability in broader, real-world scenarios where data variability and volume might be significantly higher.

Author response:
We appreciate the insightful suggestion from the reviewer. In response to their input, we conducted experiments on DomainNet [R1], a large-scale domain adaptation dataset consisting of around 0.6 million images distributed across 345 categories. Each domain involves 40,000 images for training and 8,000 images for testing. Our continual domain adaptation task was created using five diverse domains from DomainNet: Clipart → Real → Infograph → Sketch → Painting. The experimental methodology closely followed the protocol established in prior PCL research. The outcomes of these experiments are presented in the table below.

| Method (Dataset: DomainNet) |    Acc (%)   | Forgetting score |
|-----------------------------|:------------:|:----------------:|
| UppderBound                 |     79.65    |         -        |
| FT                          | 18.00 ± 0.26 |   43.55 ± 0.27   |
| ER                          | 58.32 ± 0.47 |   26.25 ± 0.24   |
| L2P                         | 69.58 ± 0.39 |    2.25 ± 0.08   |
| Deep L2P                    | 70.54 ± 0.51 |    2.05 ± 0.07   |
| DualPrompt                  | 70.73 ± 0.49 |    2.03 ± 0.22   |
| CODA                        | 73.24 ± 0.59 |    3.46 ± 0.09   |
| OS-Prompt                   | 72.24 ± 0.13 |    2.94 ± 0.02   |
| OS-Prompt++                 | 73.32 ± 0.32 |    2.07 ± 0.06   |

Our method (OS-prompt and OS-propmt++) achieves comparable performance compared to the previous PCL methods with ~50% inference computational cost. These results affirm the practical viability of our approach, indicating its applicability in real-world scenarios. We have added the results in the Appendix.

[R1] Peng et al. Moment matching for multi-source domain adaptation. ICCV19

Reviewer comment (Evaluation):
Lack of Exploration on QR Loss Limitations: The introduction of the QR loss is innovative, but the paper could benefit from a more in-depth discussion on its potential limitations or scenarios where it might not be as effective. A deeper dive into the trade-offs associated with the QR loss would provide a more balanced view of its utility.

Author response:
Thank you for your suggestion. During the training of OS-Prompt++, in comparison to OS-Prompt, the inclusion of QR loss does enhance accuracy. However, QR loss requires a reference ViT feedforward step (which is different ViT from the backbone ViT), resulting in additional training computational cost compared to OS-Prompt. It's worth highlighting that while OS-Prompt++ has a high training computational cost compared to OS-Prompt, it remains consistent with the training cost of prior PCL works (refer to Table 3).

This computational efficiency during training becomes important in the context of addressing the online continual learning problem [R2]. This problem assumes the model needs rapid training on the given streaming data. If the training cost requires a high computational cost, the stream does not wait for the model to complete training before revealing the next data for predictions, leading to performance degradation. Therefore, an analysis of the training efficiency of OS-Prompt++ is required.

To address this issue, in Fig. 6, we provide the trade-off between FLOPs and accuracy within a reference ViT used for QR Loss. In our OS-Prompt++ default setting, we extract the reference prompt query r from the final layer of the reference ViT. This requires additional training complexity compared to OS-Prompt. To improve efficiency, instead of using the final layer’s [CLS] token embedding for prompt query r, we utilize the intermediate token embeddings from layers deeper than 5 within the reference ViT. The results show there is an accuracy and training efficiency trade-off in our QR Loss design. 

More importantly, as per the reviewer's suggestion, we have added content in the Conclusion section to point out the potential limitations of the proposed method.

[R2] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023

Reviewer comment (Experiments):
Comparative Analysis Depth: While the paper highlights the superiority of the OS-Prompt over the CodaPrompt method, it might be beneficial to see how the proposed framework fares against a wider array of contemporary methods. A more extensive comparative analysis would offer readers a comprehensive understanding of where OS-Prompt stands in the broader landscape of PCL techniques.

Author response:
Thank you for the suggestions. Within the domain of PCL techniques, our primary objective is to improve energy efficiency, setting our approach complementary to previous PCL work that primarily targeted improving the accuracy of rehearsal-free continual learning. 

While our comparison focuses on the state-of-the-art PCL method (CodaPrompt), we provide a comprehensive comparison/analysis across our method and other PCL techniques. The details are outlined below:

- We present accuracy and relative computational cost comparisons in Tables 1, 2, and 3, with a summarized overview depicted in Figure 1 in the introduction.
- [Figure 5] In addition to CodaPrompt, we have included data points for L2P and DualPrompt to further enrich the comparative analysis across these PCL methods. 
- [Table 8 in Appendix] Since our goal is orthogonal to prior work, we apply our strategy to previous PCL methods, specifically L2P and DualPrompt. This shows that our one-stage PCL can be applied to future methods incorporating advanced prompt formation strategies.

Please let us know if you think further specific experiments are required.

Reviewer comment (Experiments):
Is QR without query ViT could perform better generalization on unseen task/domain?

Author response:
We appreciate the insightful suggestion from the reviewer. This is an interesting and important concern because we use the intermediate layer’s [CLS] feature as a query, which might have less generalization power compared to the [CLS] feature from the last layer of query ViT.

To this end, we conducted experiments on DomainNet [R1], a large-scale domain adaptation dataset consisting of around 0.6 million images distributed across 345 categories. Our continual domain adaptation task was created using five diverse domains from DomainNet: Clipart → Real → Infograph → Sketch → Painting. Thus every task has a different domain image, which is a proper experiment to check the generalization ability of our method. The experimental results are presented in the table below.

| Method (Dataset: DomainNet) |    Acc (%)   | Forgetting score |
|-----------------------------|:------------:|:----------------:|
| UppderBound                 |     79.65    |         -        |
| FT                          | 18.00 ± 0.26 |   43.55 ± 0.27   |
| ER                          | 58.32 ± 0.47 |   26.25 ± 0.24   |
| L2P                         | 69.58 ± 0.39 |    2.25 ± 0.08   |
| Deep L2P                    | 70.54 ± 0.51 |    2.05 ± 0.07   |
| DualPrompt                  | 70.73 ± 0.49 |    2.03 ± 0.22   |
| CODA                        | 73.24 ± 0.59 |    3.46 ± 0.09   |
| OS-Prompt                   | 72.24 ± 0.13 |    2.94 ± 0.02   |
| OS-Prompt++                 | 73.32 ± 0.32 |    2.07 ± 0.06   |

Our method (OS-prompt and OS-propmt++) achieves comparable performance compared to the previous PCL methods with ~50% inference computational cost. These findings suggest the adaptability of our method across various domains in different stages of continual learning. 

We have added the results in the Appendix.

[R1] Peng et al. Moment matching for multi-source domain adaptation. ICCV19

Reviewer comment (Experiments):
The query pool regularization introduces extra computation in the training phase, especially as more advanced reference ViTs are used. I do not think this is particularly fair in comparison to prior work as they used less resources during training.

Author response:
Thank you for your response. We would like to provide clarification regarding your comment. The reference ViT used in OS-Prompt++ does not consist of larger models; we employ the standard ViT-B/16, which serves as the backbone architecture. Additionally, the architectures utilized for ER, LwF, and other PCL methods are all based on ViT-B/16. 

As demonstrated in our response, despite the two-step feedforward during training (shared with other PCL methods), the total FLOPs are lower compared to other continual learning methods. Compared to prompt learning, the end-to-end training method (such as ER and LwF) incurs a threefold increase in computational cost due to weight updates for the entire set of parameters. 

In summary, our method employs the same model size as others, and prompt learning proves more training-efficient than other continual learning methods, primarily due to a lower number of parameter updates.

Reviewer comment (Evaluation):
As the idea of continual learning depends much more heavily on the computation cost of training and inference due to the training phase being run continuously, I do not think matching the computation cost during the inference phase only is fully fair.

Author response:
Thank you for the valuable feedback. We want to clarify that our OS-Prompt++ (with query pool regularization) does not result in increased training costs compared to previous Prompt Continual Learning (PCL) methods, such as CodaPrompt, DualPrompt, and L2P (refer to Table 3). Instead, OS-Prompt++ maintains the same training cost as the earlier approaches. On the other hand, our efficient version of PCL, named OS-prompt, minimizes the forward computation of the PCL framework, improving computational efficiency for both training and inference.

In our work, we aim to solve 'Rehearsal-Free' continual learning, aiming to address both privacy and memory efficiency concerns, aligning with prior PCL research. Although our problem setting differs from online continual learning like [A], we agree with the reviewer that our proposed method needs a more thorough discussion of the training computation cost to provide a better understanding of the proposed approach. Following Table 1 provided in [A], we compute the relative training complexity with respect to ER or Experience Replay, which is the simple baseline with standard gradient-based training. 


| CL strategy       | Method             | Relative training complexity w.r.t ER |
|-------------------|--------------------|---------------------------------------|
| Experience Replay | ER                 | 1                                     |
| Regularization    | LwF                | 4/3                                   |
| Prompt-based      | L2P                | 1                                     |
| Prompt-based      | DualPrompt         | 1                                     |
| Prompt-based      | CodaPrompt         | 1                                     |
| Prompt-based      | OS-Prompt (Ours)   | 2/3                                   |
| Prompt-based      | OS-Prompt++ (Ours) | 1                                     |

Here, we would like to clarify the training computational cost of prompt learning. In general neural network training,  the forward-backward computational cost ratio is approximately 1:2. This is due to gradient backpropagation ($\frac{dL}{da_l}=W_{l+1}\frac{dL}{da_{l+1}}$) and weight-updating ($\frac{dL}{dW_l}=\frac{dL}{da_l+1}a_l$), where $L$ represents the loss, $W$ denotes the weight parameter, $a$ is the activation, and $l$ is the layer index.  In prompt learning, the forward-backward computational cost ratio is approximately 1:1. This is in contrast to general neural network training, as only a small fraction of the weight parameters (less than 1\%) are updated.


Bearing this in mind, we present the observations derived from the table, assuming that all methods employ the same architecture.
* Previous PCL (L2P, DualPrompt, CodaPrompt) consists of two steps; (1) the query ViT requires only feedforward without backpropagation; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in the relative training computational cost is 1 ($= \frac{Query ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$)
* Similarly, our OS-Prompt++ also consists of two steps; (1) the reference ViT with only a feedforward step to calculate QR loss; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in a relative training computational cost 1 ($= \frac{Reference ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$) with respect to ER. 
* On the other hand, our OS-Prompt requires one feedforward-backward step with prompt tuning. Therefore, relative training computaional cost becomes $\frac{2}{3}$ ($= \frac{Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$). This shows the potential of our OS-Prompt on online continual learning.

Overall, we propose two versions of the one-stage PCL method (i.e., OS-prompt and OS-prompt++), and these two options enable the users to select a suitable method depending on the problem setting. For example, for online continual learning, OS-prompt is a better option since it requires less training cost. On the other hand, for offline continual learning, one can use OS-prompt++ to maximize performance while spending more training energy. 

We have added this discussion in the Appendix. 

[A] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023

Reviewer comment (Evaluation):
Recent works such as [A] tackle this specific part of the resource constraints in continual learning and I believe the proposed method needs to discuss on the computation cost of the training phase more thoroughly. 

[A] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023

Author response:
Thank you for the valuable feedback. We want to clarify that our OS-Prompt++ (with query pool regularization) does not result in increased training costs compared to previous Prompt Continual Learning (PCL) methods, such as CodaPrompt, DualPrompt, and L2P (refer to Table 3). Instead, OS-Prompt++ maintains the same training cost as the earlier approaches. On the other hand, our efficient version of PCL, named OS-prompt, minimizes the forward computation of the PCL framework, improving computational efficiency for both training and inference.

In our work, we aim to solve 'Rehearsal-Free' continual learning, aiming to address both privacy and memory efficiency concerns, aligning with prior PCL research. Although our problem setting differs from online continual learning like [A], we agree with the reviewer that our proposed method needs a more thorough discussion of the training computation cost to provide a better understanding of the proposed approach. Following Table 1 provided in [A], we compute the relative training complexity with respect to ER or Experience Replay, which is the simple baseline with standard gradient-based training. 


| CL strategy       | Method             | Relative training complexity w.r.t ER |
|-------------------|--------------------|---------------------------------------|
| Experience Replay | ER                 | 1                                     |
| Regularization    | LwF                | 4/3                                   |
| Prompt-based      | L2P                | 1                                     |
| Prompt-based      | DualPrompt         | 1                                     |
| Prompt-based      | CodaPrompt         | 1                                     |
| Prompt-based      | OS-Prompt (Ours)   | 2/3                                   |
| Prompt-based      | OS-Prompt++ (Ours) | 1                                     |

Here, we would like to clarify the training computational cost of prompt learning. In general neural network training,  the forward-backward computational cost ratio is approximately 1:2. This is due to gradient backpropagation ($\frac{dL}{da_l}=W_{l+1}\frac{dL}{da_{l+1}}$) and weight-updating ($\frac{dL}{dW_l}=\frac{dL}{da_l+1}a_l$), where $L$ represents the loss, $W$ denotes the weight parameter, $a$ is the activation, and $l$ is the layer index.  In prompt learning, the forward-backward computational cost ratio is approximately 1:1. This is in contrast to general neural network training, as only a small fraction of the weight parameters (less than 1\%) are updated.


Bearing this in mind, we present the observations derived from the table, assuming that all methods employ the same architecture.
* Previous PCL (L2P, DualPrompt, CodaPrompt) consists of two steps; (1) the query ViT requires only feedforward without backpropagation; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in the relative training computational cost is 1 ($= \frac{Query ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$)
* Similarly, our OS-Prompt++ also consists of two steps; (1) the reference ViT with only a feedforward step to calculate QR loss; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in a relative training computational cost 1 ($= \frac{Reference ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$) with respect to ER. 
* On the other hand, our OS-Prompt requires one feedforward-backward step with prompt tuning. Therefore, relative training computaional cost becomes $\frac{2}{3}$ ($= \frac{Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$). This shows the potential of our OS-Prompt on online continual learning.

Overall, we propose two versions of the one-stage PCL method (i.e., OS-prompt and OS-prompt++), and these two options enable the users to select a suitable method depending on the problem setting. For example, for online continual learning, OS-prompt is a better option since it requires less training cost. On the other hand, for offline continual learning, one can use OS-prompt++ to maximize performance while spending more training energy. 

We have added this discussion in the Appendix. 

[A] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023