Paper: Learning without Forgetting for Vision-Language Models (ICLR-2024)

Reviewer comment (Novelty):
The paper lacks motivation and innovation: The authors suggest using CLIP for class-incremental continual learning, but it would be more interesting to see its performance on tasks like incremental captioning or retrieval. Unlike L2P, where a large pretrained model was used, CIL could have been just one application.

Author response:
We thank the reviewer for the suggestion. Just as the reviewer pointed out, we not only investigate PROOF’s strong performance in class-incremental learning in this paper. Additionally, **we have compared PROOF to other competitors in the continual cross-modal retrieval tasks in Section 5.4 and Figure 4 (a) in the initial submission**. Correspondingly, we find PROOF is a concise and unified framework that solves catastrophic forgetting in various continual learning scenarios (e.g., class-incremental learning and continual cross-modal retrieval).

Reviewer comment (Novelty):
Furthermore, the PROOF mechanism, while innovative, lacks depth. Projection networks are common in continual learning, and the new context definition isn't explored.

Author response:
We thank the reviewer for the suggestion. Since the reviewer did not give any related papers on “Projection networks are common in continual learning”, we have searched for related topics and found some, e.g., [1-3] However, among these related papers, the projection is designed on the parameter/gradient space, i.e., these works are designed to tackle forgetting by finding optimization paths through projection. By contrast, the projection in this paper is designed on the embedding space, which aims to encode task-specific information on pre-trained embeddings. Our method creates and learns parallel subspaces based on the pre-trained vision-language model, while others do not. **Hence, it seems the “projections” are obviously two different concepts.** We are happy to give further clarifications if the reviewer has other related papers for supplementary.

**Furthermore, we need to highlight that the core contribution of our paper is not how to design the projection since they are only linear layers. By contrast, our contributions lie in enabling a pre-trained vision-language model to adapt to downstream tasks continually. Such a task requires the projection expansion strategy, cross-modal fusion, and context joint adaptation. We believe these parts are not common in other papers.**

Reviewer comment (Evaluation):
The main paper lacks standard deviation in results and doesn't consider multiple runs with different class orders.

Author response:
We thank the reviewer for the suggestion. In the main paper, we follow the benchmark class-incremental learning setting [4] to shuffle classes with random seed 1993. Just as the reviewer pointed out, we not only run the paper one time in our experiments. By contrast, **we have compared PROOF to other competitors with five class orders (1993, 1994, 1995, 1996, 1997) and reported the performance curve in Supplementary Section D.1 in the initial submission**. These performance curves contain the average and standard deviation information, and results on CIFAR and ImageNet-R with multiple runs verify that our PROOF consistently outperforms other competitors.

Reviewer comment (Writing):
The paper's narration could also be improved.

Author response:
We thank the reviewer for the suggestion. In this revision, we have **made Section 3 much shorter** by deleting discussions of baseline methods and **increased the length of Section 4 (especially Section 4.2)** to fully introduce the cross-modal fusion module. Additionally, we have **added the requested experimental results and discussions in Supplementary Section D.3, E.4, E.5.** We hope the revision can improve the quality of the narration.

Reviewer comment (Presentation):
It looks like the supplementary materials are more informative and present more interesting results w.r.t. the main paper. Why did the authors exclude them from the main paper?

Author response:
We thank the reviewer for the suggestion. In the main paper, we use class-incremental learning as the most widely applicable continual learning setting and conduct experiments on various class-incremental learning datasets and protocols (main paper Section 5.2, Table 1, Figure 2). We also investigate each component’s influence with ablation studies (main paper Section 5.3, Figure 3). Finally, we also verify PROOF’s potential in other continual learning settings (e.g., continual cross-modal retrieval) and other applications (preserving zero-shot ability) in the main paper Section 5.4 and Figure 4. In the supplementary, we mainly supply additional results and implementation details that cannot be included in the main paper due to the page limit.

**However, all essential parts of the supplementary material are mentioned and discussed in the main paper.** For example, Supplementary Section B indicates PROOF can work competitively in continual cross-modal retrieval, which is discussed in the main paper Section 5.4 and Figure 4 (a). Supplementary Section D.10 discusses the results on a new TV series dataset, which is discussed in the main paper Section 5.4 and Figure 4 (b). Supplementary Section C discusses how to maintain the model’s zero-shot performance, and we have also included it in the main paper Section 5.4 and Figure 4 (c). Apart from these parts, other results in the supplementary act as the additional results of the main paper, e.g., the full results or detailed performance in Table 4. **Hence, there is no critical information excluded from the main paper.**

Reviewer comment (Writing):
The definition of W is not reported in the paper. How W is defined in the context?

Author response:
**The definition of W is clearly defined in the main paper, Section 4.2, Paragraph “How to define the context?”.** Specifically, W is the concatenation of all textual features in the label space, i.e., $W=[P_t(w_1), P_t(w_2),\dots, P_t(w_{|\mathcal{Y}_b|})]$. Among them, $P_t$ is the textual projection function (as defined in Eq. 4), and $w_i$ stands for CLIP’s text embedding of templated input “a photo of a [CLASS]” (as defined below Eq. 2).

Reviewer comment (Experiments):
Inference Mismatch - Projections are combined at inference time which may not fully match the training conditions for a specific task projection.

Author response:
We thank the reviewer for the question. We first review the training process of PROOF to illustrate why inference mismatch will not happen. Given the pre-trained vision-language model, we freeze the pre-trained weights and learn expandable projections to map image/text embeddings into task-specific space. Since we learn expandable projections as data evolves, we utilize the additive result as the final image/text embedding (i.e., $P_i(z)$ and $P_t(w)$ in Eq. 4). During training, we freeze former tasks’ projections and only train the current one. The training target is to match the additive features of image-text pair $P_i(z)$ and $P_t(w)$, and the latest projection (the current task’s projection) is optimized to serve as **residual terms** that can adjust the embedding for a holistic prediction. **Since the loss term in Eq. 7 is optimized with exemplars (i.e., a few old class instances) and the current dataset, optimizing it will make the aggregated features holistic to consider all classes.** In other words, since the target is to learn new classes and meanwhile maintain the performance of old classes, being ‘ideal’ projections for some specific task is not the best solution for all seen classes, and our learning protocol strikes a balance between learning new and remembering old. Even if additional projection terms are added after some specific tasks, optimizing Eq. 7 with exemplars and the current dataset still leads to a well-calibrated prediction considering all seen classes. 

To empirically show the forgetting degree of different tasks, we also measure the relative forgetting degree using the “Forgetting Measure” in [1][2]. Considering each task’s forgetting degree, it measures the gap between the best and final performance and averages among all tasks. Hence, methods with a lower forgetting measure show better performance in resisting catastrophic forgetting. In this revision, we have supplied the results in **Supplementary Section E.4** and report the results here. We report the results compared to three competitors, i.e., CoOp, iCaRL, and MEMO. 


| Method | Exemplar | Aircraft B0 Inc10| Cars B0 Inc10| CIFAR100 B0 Inc10| UCF B0 Inc10|
|:----|:----:|:----:|:----:|:----:|:----:| 
|CoOp| &#10003;| 28.20|	7.43	|	20.74|	10.81|
|iCaRL | &#10003;| 13.44|	4.18	|	31.45|	6.66|
|MEMO| &#10003;| 17.34|	4.77|		12.40|	9.78|
|PROOF| &#10003;| **9.41**| **3.51**| **8.02**| **4.07**|

As shown in the table, **PROOF has the lowest forgetting measure, indicating it suffers the least forgetting (less than 10%) compared to other competitors.** It verifies that the final performance and best performance are almost the same even after long training stages for PROOF. In other words, 
the optimization target makes the latest projection a residual term to **learn a unified embedding considering all seen classes, and inference mismatch can be alleviated**.

Reviewer comment (Theory):
Representation Drift - The post-attention module representations learned by the frozen projections may drift or shift slightly during new task training due to weight updates elsewhere. Small drifts can accumulate.

Author response:
We thank the reviewer for the suggestion. It must be noted that there are two main steps in our method, i.e., feature extraction and cross-modal fusion. The extraction process contains the pre-trained embedding and projection, which is designed to extract task-specific features. By contrast, the attention module is responsible for fusing cross-modal information so that features can be more discriminative. **Hence, such fusion is irrelevant to the task, which should be generalizable across tasks. In other words, even if the features shift slightly, the attention process still works to fuse and highlight task-specific information.**

On the other hand, the whole optimization flow is continuous, even if the feature representation is changing due to projections. In other words, the drift of former extracted features will result in the corresponding adaptation of the attention module. Since the model is trained with exemplars and current datasets (as defined in Eq. 7), optimizing the loss considers all essential features among all seen classes for decision. The representation drift of former stages will be compensated by the latter stages.

Finally, we must highlight an essential module in the attention process, i.e., the context prompt C. It serves as the task anchor for each task, which is specially learned for each new task. Since the cross-modal fusion process will contextualize the visual embedding with the context information, the context prompt can identify the specific task to highlight important features. As a result, the negative impact of representation drift can be alleviated in our method.

Reviewer comment (Presentation):
Section 3 is really long and has a lot of redundant information, it should be made much shorter. That space should be given to increase the length of section 4 to give a better understanding of the fusion module.

Author response:
We thank the reviewer for the suggestion. In this revision, we have **made Section 3 much shorter** by deleting discussions of baseline methods and **increased the length of Section 4 (especially Section 4.2)** to fully introduce the cross-modal fusion module. In the following part, we elaborate on the attention mechanism for cross-modal fusion.

The typical matching target of CLIP is to match the visual feature to the textual feature and output the target with the largest similarity. However, we argue that it would be beneficial to further refine these features to capture the **contextual relationship** between images and texts. For example, when the query instance is a “panda,” it is desirable to adjust the visual features to highlight the discriminative attributes such as **black eyes and ears**. Meanwhile, considering the visual embedding of a panda, the textual features should also be adapted in a coherent manner so that co-adapted visual and textual features can lead to more discriminative predictions. Similarly, when the query instance is a “cat,” features like beards and tails should be emphasized jointly for visual and textual embeddings. This adjustment process involves jointly adapting the query embedding and the context (e.g., textual information) to obtain a contextualized embedding. A desirable adjustment function should be able to relate every other component as context to conduct joint adaptation.  Correspondingly, we use the self-attention mechanism that contextualizes and fuses the query embeddings and cross-modal information.

We define the context information as $Context=[P, W]$ (we do not consider the context prompt in this part), including the visual prototypes and textual classifiers. Hence, we feed the self-attention module with $[P_i(z), Context]$ as its query ($\mathcal{Q}$), key ($\mathcal{K}$), value ($\mathcal{V}$). During the attention process, it calculates the local similarity among these inputs, and adjusts them simultaneously via the relationship to other components. During the calculation, it first projects $\mathcal{Q}$, $\mathcal{K}$, $\mathcal{V}$ into the same embedding space and calculates the relative similarity. For the visual feature $P_i(z)$, the output is denoted as $P_i(z)+\sum_k\alpha_{qk}V_{:,k}$, i.e., adding a residual term to the original input. The residual information is the weighted combination of other inputs, which corresponds to the visual prototypes and textual classifiers. The adaption is the same for other components in the attention module, and the cross-modal fusion leads to more discriminative features (as shown in the main paper, Figure 3 (b)).

Reviewer comment (Writing):
Also please make it more clear how you are using attention.

Author response:
We thank the reviewer for the suggestion. In this revision, we have **made Section 3 much shorter** by deleting discussions of baseline methods and **increased the length of Section 4 (especially Section 4.2)** to fully introduce the cross-modal fusion module. In the following part, we elaborate on the attention mechanism for cross-modal fusion.

The typical matching target of CLIP is to match the visual feature to the textual feature and output the target with the largest similarity. However, we argue that it would be beneficial to further refine these features to capture the **contextual relationship** between images and texts. For example, when the query instance is a “panda,” it is desirable to adjust the visual features to highlight the discriminative attributes such as **black eyes and ears**. Meanwhile, considering the visual embedding of a panda, the textual features should also be adapted in a coherent manner so that co-adapted visual and textual features can lead to more discriminative predictions. Similarly, when the query instance is a “cat,” features like beards and tails should be emphasized jointly for visual and textual embeddings. This adjustment process involves jointly adapting the query embedding and the context (e.g., textual information) to obtain a contextualized embedding. A desirable adjustment function should be able to relate every other component as context to conduct joint adaptation.  Correspondingly, we use the self-attention mechanism that contextualizes and fuses the query embeddings and cross-modal information.

We define the context information as $Context=[P, W]$ (we do not consider the context prompt in this part), including the visual prototypes and textual classifiers. Hence, we feed the self-attention module with $[P_i(z), Context]$ as its query ($\mathcal{Q}$), key ($\mathcal{K}$), value ($\mathcal{V}$). During the attention process, it calculates the local similarity among these inputs, and adjusts them simultaneously via the relationship to other components. During the calculation, it first projects $\mathcal{Q}$, $\mathcal{K}$, $\mathcal{V}$ into the same embedding space and calculates the relative similarity. For the visual feature $P_i(z)$, the output is denoted as $P_i(z)+\sum_k\alpha_{qk}V_{:,k}$, i.e., adding a residual term to the original input. The residual information is the weighted combination of other inputs, which corresponds to the visual prototypes and textual classifiers. The adaption is the same for other components in the attention module, and the cross-modal fusion leads to more discriminative features (as shown in the main paper, Figure 3 (b)).

Reviewer comment (Theory):
The effectiveness of alleviating forgetting is uncertain. The process involves incrementally learning image projection heads and text projection heads, which are then combined for various tasks. When new tasks are learned, the projections of previous tasks are fixed and not updated. However, during inference, the projections of all tasks are merged, which might not be ideal for test data from older tasks due to potential side effects caused by the projections from the new tasks.

Author response:
We thank the reviewer for the question and answer it from two aspects. We first review the training process of PROOF to illustrate its rationale for overcoming forgetting. Given the pre-trained vision-language model, we freeze the pre-trained weights and learn expandable projections to map image/text embeddings into task-specific space. Since we learn expandable projections as data evolves, we utilize the additive result as the final image/text embedding (i.e., $P_i(z)$ and $P_t(w)$ in Eq. 4). During training, we freeze former tasks’ projections and only train the current one. The training target is to match the additive features of image-text pair $P_i(z)$ and $P_t(w)$, and the latest projection (the current task’s projection) is optimized to serve as **residual terms** that can adjust the embedding for a holistic prediction. **Since the loss term in Eq. 7 is optimized with exemplars (i.e., a few old class instances) and the current dataset, optimizing it will make the aggregated features holistic to consider all classes.** In other words, since the target is to learn new classes and meanwhile maintain the performance of old classes, being ‘ideal’ projections for old classes is not the best solution for all seen classes, and our learning protocol strikes a balance between learning new and remembering old. 

Secondly, we also measure the relative forgetting degree using the “Forgetting Measure” in [1][2]. Considering each task’s forgetting degree, it measures the gap between the best and final performance and averages among all tasks. Hence, methods with a lower forgetting measure show better performance in resisting catastrophic forgetting. In this revision, we have supplied the results in **Supplementary Section E.4** and report the results here. We report the results compared to three competitors, i.e., CoOp, iCaRL, and MEMO. 


| Method | Exemplar | Aircraft B0 Inc10| Cars B0 Inc10| CIFAR100 B0 Inc10| UCF B0 Inc10|
|:----|:----:|:----:|:----:|:----:|:----:| 
|CoOp| &#10003;| 28.20|	7.43	|	20.74|	10.81|
|iCaRL | &#10003;| 13.44|	4.18	|	31.45|	6.66|
|MEMO| &#10003;| 17.34|	4.77|		12.40|	9.78|
|PROOF| &#10003;| **9.41**| **3.51**| **8.02**| **4.07**|

As shown in the table, **PROOF has the lowest forgetting measure, indicating it suffers the least forgetting (less than 10%) compared to other competitors.** It verifies that the final performance and best performance are almost the same even after long training stages for PROOF.

In summary, the optimization target makes the latest projection a residual term to **learn a unified embedding considering all seen classes**. Besides, experimental results also verify that PROOF shows the lowest forgetting among all competitors.

Reviewer comment (Experiments):
The extent to which contextual information is effective has not been extensively studied. The projection fusion method proposes to contextualize and merge embeddings and contextual information using self-attention. However, in the experiments, only the results of Projection & Fusion are compared with Projection & Fusion & Context Prompt, without explicitly evaluating the effectiveness of the concatenated context information in Q, K, V as [P_i(z), Context] in self-attention, or the effectiveness of the context prompt. In other words, the final context information is defined as Context = [P, W, C], but the specific contributions of W and C to the final results need further analysis.

Author response:
We thank the reviewer for the suggestion. In fact, the ablation study on the context components is extensively researched in **Supplementary Section D.5** in the submitted version. We explain the general results here and refer the reviewer to supplementary materials for more details. Specifically, we conduct experiments by varying the compositional components in the Context, i.e., Context=[P], Context=[W], Context=[P, W], Context=[P, W, C]. For these variants, we keep the other settings the same and report the incremental performance. 

From the results, we observe that using visual prototypes or textual prototypes alone yields similar performance, and the impact of adjustment is marginal. However, when visual and textual prototypes are jointly utilized as context information, the model can learn from cross-modality and achieve better performance. Lastly, the introduction of context prompts into the context further enhances the performance of PROOF, resulting in the best performance among all variations. These ablation studies verify that each component in the context helps in the final prediction.

Reviewer comment (Evaluation):
The evaluation metric used may not provide a comprehensive measure of the extent of forgetting.

Author response:
We thank the reviewer for the suggestion. As we responded in A1, we consider the forgetting measure in [1][2] to measure the task-wise forgetting degree. Results show that our proposed PROOF has the lowest forgetting among other competitors. We have included these additional results in **Supplementary Section E.4**.

[1] A Comprehensive Survey of Continual Learning: Theory, Method and Application. arXiv 2023.

[2] Online continual learning in image classification: An empirical survey. Neurocomputing 2022.

Reviewer comment (Evaluation):
To what extent the proposed method could alleviate forgetting?

Author response:
We thank the reviewer for the suggestion. As we responded in A1, we consider the forgetting measure in [1][2] to measure the task-wise forgetting degree. Results show that our proposed PROOF has the lowest forgetting among other competitors. We have included these additional results in **Supplementary Section E.4**.

[1] A Comprehensive Survey of Continual Learning: Theory, Method and Application. arXiv 2023.

[2] Online continual learning in image classification: An empirical survey. Neurocomputing 2022.