Paper: Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models (ICLR-2024)

Reviewer comment (Experiments):
prefer to learn more details of how you decide the length of soft prompt vectors, e.g., why 4 and 16, will there be more ranges to be investigated basing on the specificl tasks for VLMs?

Author response:
**A1.** Thanks for your valuable comment regarding the discussion of the impact of the length of soft prompt vectors. Our paper mainly focuses on investigating the effect of soft prompt norms on vision-language models, so we just follow the standard settings of CoOp, with a length of 16 for soft prompts. Generally, the length of soft prompts is typically determined empirically. However, we do have observed better results with certain datasets using different lengths, such as 6 and 10, compared to the standard length of 16, through experiments. This suggests that the standard settings of CoOp may not be optimal for all datasets, indicating the need to explore and determine an appropriate length of soft prompts, especially across different datasets. Therefore, we believe it is **necessary and beneficial** to try more lengths of soft prompts when dealing with various VLM-based tasks.

Reviewer comment (Experiments):
prefer to learn more investigations of combining Nemesis with existing PEFT algorithms to see if the results can be further improved or not so that other researchers can better leverage your method to their existing frameworks.

Author response:
**A2.** We appreciate your concern regarding the need for discussion on potential applicable scenarios. While our proposed method primarily focuses on benchmarking soft prompt-tuning VLMs and several downstream VLM-based tasks, including few-shot image classification, domain generalization, and base-to-new generalization, it should be noted that the benefits of the two proposed normalization losses may be not limited to these tasks alone. We speculate that they can also be applied to other parameter-efficient tuning (**PEFT**) methods, such as prompt-tuning, prefix-tuning, P-tuning, adapter-tuning, and LoRA, as well as their various downstream tasks, including vision-related tasks like image classification, object detection and semantic segmentation, as well as language-related tasks like text classification and generation. 

To verify this, we conducted preliminary experiments on a few PEFT methods and their applicable scenarios, including **visual prompt-tuning (VPT)** [1] for image classification and **prefix-tuning** [2] for paraphrase classification. Due to time constraints during the rebuttal period, we focused exclusively on testing our proposed PEN loss, which can be seamlessly and effortlessly integrated into existing PEFT methods. On the other hand, the proposed PAN loss may involve designing specific corruption experiments and adopting distinct performance comparison metrics depending on different methods and tasks. To be more specific, we should adopt various performance metrics when identifying the positions that induce the Low-Norm Effect, such as classification accuracy for classification tasks, mAP (mean Average Precision) for object detection tasks, IoU (Intersection over Union) for segmentation tasks, and BLEU (BilinguaL Evaluation Understudy) for text generation. We hope for more research to explore its applicability and adaptability in various PEFT tasks in the future. Hence, here we only report our preliminary results based on the PEN loss using various values of $\omega$, as shown in following tables.  $\omega$ is a scaling coefficient that controls the normalization strength.

| Datasets | $\omega$=0 | $\omega$=0.001 | $\omega$=0.01 | $\omega$=0.1 | $\omega$=1 | $\omega$=10 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: |
| Sun397 (Natural) | 49.86 | 49.76 | **50.04** | 48.89 | 46.67 | 45.07 |
| EuroSAT (Specialized) | 91.89 | 91.30 | 92.00 | **92.81** | 91.63 | 91.31 |
| DMLab (Structured) | 33.54 | 33.39 | 33.75 | 34.85 | **35.75** | 35.50 |

| Datasets | Performance Metrics | $\omega$=0 | $\omega$=0.0001 | $\omega$=0.001 | $\omega$=0.01 | $\omega$=0.1 | $\omega$=1 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| MRPC | F1 Score | 89.37 | **91.06** | 90.90 | 88.70 | 80.62 | 80.81 |
| MRPC | Accuracy | 85.10 | **88.00** | **88.00** | 85.10 | 69.62 | 70.90 |

The VPT using the VTAB-1k [3] benchmark, including 19 datasets categorized into three groups: Natural, Specialized, and Structured. To validate our proposed loss (the PEN loss), we select one dataset from each group. Additionally, we also evaluate the effectiveness of the PEN loss on the MRPC dataset [4], which is a popular benchmark of natural language understanding. The experiment details are based on the GitHub repository at <https://github.com/KMnP/vpt> for VPT with the ViT-B/16 backbone and <https://github.com/huggingface/peft> for prefix-tuning with the RoBERTa-Large backbone.

From presented results, the proposed PEN loss can **enhance the performance** in all conducted experiments, which is an exciting outcome. However, it is acknowledged that determining the ideal weight for the proposed loss requires further investigation, which is a worthy direction that can be explored in future work. 

All in all, these preliminary responses our research prospect discussed in the Conclusion Section (i.e. Section 6) and we are hopeful that our finds and approaches will provide new insights to PEFT methods and inspire future research in these fields.

Reviewer comment (Experiments):
could there be a combination of between soft-prompt tuning and hard-prompt tuning? (hard = explicitly use some predefined words/phrases as part of the prompts);

Author response:
**A3.** Thanks for your insightful comment considering the possibility of combination of between soft-prompt tuning and hard-prompt tuning. As far as we know, **P-tuning** [5] and **P-tuning v2** [6] have implemented this idea, which employs trainable continuous prompt embeddings in concatenation with discrete prompts, and achieved remarkable performance. We enthusiastically welcome the integration of our method, Nemesis, into P-tuning. We believe that incorporating Nemesis into P-tuning could result in remarkable results.

Reviewer comment (Experiments):
any idea of further combining existing PEFT (prompt tuning, prefix tuning, LoRA...) with your Nemesis method?

Author response:
**A4.** The response related to the potential applicable scenarios of Nemesis has provided earlier in **A2**. Our preliminary experiments have demonstrated that our proposed method Nemesis can enhance the performance of visual prompt-tuning and prefix-tuning methods. We sincerely welcome further testing of our approach on more PEFT methods and downstream tasks in the future.

Reviewer comment (Experiments):
$\beta$ can be either 0 or 1, corresponding to two variants of the proposed Nemesis method. However, there is no ablation study on the selection of $\beta$, nor is there an exploration of the potential impact of setting $\beta$ with decimal values to assign weights to the two methods.

Author response:
**A1.** We sincerely appreciate your suggestion to conduct ablation study about $\beta$ (i.e. explore the simultaneous combination of the PEN loss and the PAN loss). In response to this concern, we conducted experiments to investigate the combined effect of these two losses on model performance. To be specific, we evaluated various values of $\beta$, including 0, 0.1, 0.3, 0.5, 0.7, 0.9, and 1, where 0 and 1 represent the use of the PAN and PEN loss, respectively. The detailed results are shown in the following table. 
|  | $\beta$=0 | $\beta$=0.1 | $\beta$=0.3 | $\beta$=0.5 | $\beta$=0.7 | $\beta$=0.9 | $\beta$=1 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| 1-shot | 59.33 | 59.45 | **59.54** | 59.29 | 59.26 | 59.37 | 59.42 |
| 16-shots | 74.20 | **74.39** | 73.37 | 74.02 | 73.93 | 73.79 | 73.47 |

The results presented in the table are obtained by average results of 11 datasets used in the paper and the best performance is indicated with bold. The results display an evident pattern that combining two proposed normalization losses **improves model performance** compared to using only a single normalization loss, whether it is 1-shot or 16-shots setting. Furthermore, it can be observed that $\beta$=0.3 and $\beta$=0.1 achieve the best results, respectively. This suggests that the PAN loss plays a dominant role, while the PEN loss provides assistance, which can lead to improved performance.

We apologize for not discussing the ablation studies of $\beta$ in the paper. We understand the importance of including this information, so we have added it into the Ablation Study Section (i.e. Section 4.7) of the updated paper.

Reviewer comment (Theory):
The Position Equality Normalization (PEN) loss applies equal weight to the norms of soft prompts at all positions. While the paper does acknowledge that normalizing prompt vectors at positions unaffected by the Low-Norm Effect may not yield performance improvement, the inherent assumption of the universality of the Low-Norm Effect across positions may not hold true for all datasets or real-world scenarios. The approach could benefit from a more dynamic, adaptive mechanism.

Author response:
**A3.** We appreciate your insightful comment on our proposed PEN loss. As your statement, there is an inherent assumption of the diversity of the Low-Norm Effect across positions, which aligns with our observations during experiments. The mechanism behind this diversity is complex as the positions that induce the Low-Norm Effect can change over time during training. Hence, the PEN loss seems be not sufficiently general and adaptive because it assigns equal weight to the norms of soft prompts at all positions. Nevertheless, this does not impede the PEN loss from being employed in more dynamic and adaptive forms. For instance, it is viable to randomly normalize soft prompts at one or more positions or apply random sizes of normalization for various positions. According to our ablation studies in the paper (i.e. Table 3), it is evident that randomly normalizing partial positions outperforms normalizing all positions. Therefore, we acknowledge your comment suggesting that the PEN loss could benefit from a more dynamic adaptive mechanism.

On the other hand, it actually motivates us to propose **the PAN loss as a more dynamic and adaptive alternative** to the PEN loss. The PAN loss can effectively identify the prompt positions that induce the Low-Norm Effect and then normalize them accordingly. Furthermore, our empirical results illustrate that the PAN loss does perform better in a large shot setting. 

Of course, it is encouraged to investigate other adaptive alternatives in our future work.

Reviewer comment (Experiments):
The paper utilizes the RESCALE operation with a specific rescaling factor, τ, described as a positive real number less than 1. However, there’s no mention of how the value of τ is determined, if it's consistent across datasets, or its sensitivity. The choice of τ could have implications on the effectiveness of the Nemesis method, and without clear insight into its selection, there’s potential variability in results.

Author response:
**A4.** Thank you for your comment regarding the discussion of the rescaling factor $\tau$ used in the PAN loss. We would like to kindly point out that it seems to have been overlooked in your review. In fact, the paper did provide hyper-parameter analysis for different $\tau$ values, including 0.1, 0.5, and 0.9. Based on our results, $\tau$ = 0.9 achieved slightly superior performance on average across multiple datasets compared to the default value of 0.5 that was initially used. Generally, the value of $\tau$ **ranging from 0.1 to 0.9 did not significantly impact** the model's performance.

However, we have not tested smaller values of $\tau$, such 0.01 and 0.001, when using the PAN loss. Based on the results of our corruption experiments (i.e. Table A3), it is observed that smaller values of $\tau$ typically lead to a lower occurrence frequency of the Low-Norm Effect. This suggests that extremely small values of $\tau$ might affect the effectiveness of the PAN loss because they cannot identify more prompt positions that could induce the Low-Norm Effect during the training process. Therefore, we suggest **using 0.5 or 0.9** as the value of $\tau$ when employing the Nemesis method.

Reviewer comment (Experiments):
Given the significance of the parameter $\beta$ in differentiating between the two variants of the Nemesis method, why was an ablation study not conducted to evaluate its impact? Additionally, have you considered exploring decimal values for $\beta$ to potentially strike a balance between the effects of the PEN and PAN losses?

Author response:
**A5.** The response related to $\beta$ has provided earlier in **A1**. As suggested, we have incorporated an ablation study about $\beta$ into the Ablation Study Section (i.e. Section 4.7) of the updated paper.

Reviewer comment (Experiments):
How does the proposed Nemesis method compare with other soft-prompt tuning methods in terms of computational efficiency and scalability, especially in larger datasets or more complex tasks?

Author response:
**A6.** The response related to computation costs has provided earlier in **A2**. As suggested, we have added a subsection (i.e. Appendix A.2.7) to analyze computation costs in the revised paper. 

As for **scalability**, we have conducted preliminary experiments on a few PEFT methods and their applicable scenarios, including **visual prompt-tuning (VPT)** [2] for image classification and **prefix-tuning** [3] for paraphrase classification. Due to time constraints during the rebuttal period, we focused exclusively on testing our proposed PEN loss, which can be seamlessly and effortlessly integrated into existing PEFT methods. From presented results in **A3 of Reviewer KnC5** or **A2 of Reviewer bzG1**, the proposed PEN loss **can enhance the performance** in all conducted experiments, which is an exciting outcome. On the other hand, the proposed PAN loss may involve designing specific corruption experiments and adopting distinct performance comparison metrics depending on different methods and tasks. To be more specific, we can adopt various performance metrics when identifying the positions that induce the Low-Norm Effect, such as classification accuracy for classification tasks, mAP (mean Average Precision) for object detection tasks, IoU (Intersection over Union) for segmentation tasks, and BLEU (BilinguaL Evaluation Understudy) for text generation. We hope to conduct more research to explore its applicability and adaptability in various PEFT tasks in our future work.

Reviewer comment (Writing):
The writing of some parts of the paper are not clear enough. It is recommended that the authors check. For example, there is a discrepancy between formula 4 and the symbol definition in the previous paragraph.

Author response:
**A1.** Thank you for your attention and feedback. We have carefully reviewed the symbol definitions and have made the necessary revisions to address the issue you mentioned. In order to differentiate the subscripts of $\alpha$ between in Eq. (3) and in Eq. (4), we have revised Eq. (4) in the updated paper.

Reviewer comment (Evaluation):
The two types of losses proposed in the paper lack a correlation with practical significance, suggesting authors discuss why the two forms of normalization affect soft prompt.

Author response:
**A2.** We apologize for not explicitly discussing the specific impacts of two normalization losses on models in the paper. However, we have added a subsection in the Appendix (i.e. Appendix A.2.6) of the revised paper to discuss this. We hope that it can address your concern.

Taking the Caltech101 dataset [1] as an example, we conducted experiments to illustrate the impact of two proposed losses on soft prompts and model performance. Here, we straightforwardly shows several important observations obtained by experimental results. **Firstly**, regardless of whether it is in the 1-shot or 16-shot setting, the PEN loss demonstrates a stronger level of normalization compared to the PAN loss. On the other hand, despite the stronger normalization of the PEN loss, it does not outperform the PAN loss in the 16-shot setting. This implies that simply reducing the norm of soft prompts does not always lead to an improvement in model performance. **Secondly**, there is a notable occurrence frequency of the Low-Norm Effect in the 16-shot setting during the training process, indicating that an increasing number of training samples can facilitate the identification of prompting positions that induce the Low-Norm Effect by the PAN loss. This observation may explain why the PAN loss performs better in the 16-shot setting but is inferior to the PEN loss in the 1-shot setting. **Lastly**, the addition of two normalization losses does not seem to have a substantial influence on the original cross-entropy loss, suggesting that they do not contribute to the model performance primarily by reducing the cross-entropy loss. Instead, their benefits are likely derived from harnessing the Low-Norm Effect. The visual results can be found in the updated paper.

Reviewer comment (Evaluation):
The paper lacks discussion on the applicable scenarios of two normalization losses.

Author response:
**A3.** We appreciate your concern regarding the need for discussion on potential applicable scenarios. While our proposed method primarily focuses on benchmarking soft prompt-tuning VLMs and several downstream VLM-based tasks, including few-shot image classification, domain generalization, and base-to-new generalization, it should be noted that the benefits of the two proposed normalization losses may be not limited to these tasks alone. We speculate that they can also be applied to other parameter-efficient tuning (**PEFT**) methods, such as prompt-tuning, prefix-tuning, P-tuning, adapter-tuning, and LoRA, as well as their various downstream tasks, including vision-related tasks like image classification, object detection and semantic segmentation, as well as language-related tasks like text classification and generation. 

To verify this, we conducted preliminary experiments on a few PEFT methods and their applicable scenarios, including **visual prompt-tuning (VPT)** [2] for image classification and **prefix-tuning** [3] for paraphrase classification. Due to time constraints during the rebuttal period, we focused exclusively on testing our proposed PEN loss, which can be seamlessly and effortlessly integrated into existing PEFT methods. On the other hand, the proposed PAN loss may involve designing specific corruption experiments and adopting distinct performance comparison metrics depending on different methods and tasks. To be more specific, we should adopt various performance metrics when identifying the positions that induce the Low-Norm Effect, such as classification accuracy for classification tasks, mAP (mean Average Precision) for object detection tasks, IoU (Intersection over Union) for segmentation tasks, and BLEU (BilinguaL Evaluation Understudy) for text generation. We hope for more research to explore its applicability and adaptability in various PEFT tasks in the future. Hence, here we only report our preliminary results based on the PEN loss using various values of $\omega$, as shown in following tables.  $\omega$ is a scaling coefficient that controls the normalization strength. 
| Datasets | $\omega$=0 | $\omega$=0.001 | $\omega$=0.01 | $\omega$=0.1 | $\omega$=1 | $\omega$=10 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: |
| Sun397 (Natural) | 49.86 | 49.76 | **50.04** | 48.89 | 46.67 | 45.07 |
| EuroSAT (Specialized) | 91.89 | 91.30 | 92.00 | **92.81** | 91.63 | 91.31 |
| DMLab (Structured) | 33.54 | 33.39 | 33.75 | 34.85 | **35.75** | 35.50 |

| Datasets | Performance Metrics | $\omega$=0 | $\omega$=0.0001 | $\omega$=0.001 | $\omega$=0.01 | $\omega$=0.1 | $\omega$=1 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| MRPC | F1 Score | 89.37 | **91.06** | 90.90 | 88.70 | 80.62 | 80.81 |
| MRPC | Accuracy | 85.10 | **88.00** | **88.00** | 85.10 | 69.62 | 70.90 |

The VPT using the VTAB-1k [4] benchmark, including 19 datasets categorized into three groups: Natural, Specialized, and Structured. To validate our proposed loss (the PEN loss), we select one dataset from each group. Additionally, we also evaluate the effectiveness of the PEN loss on the MRPC dataset [5], which is a popular benchmark of natural language understanding. The experiment details are based on the GitHub repository at <https://github.com/KMnP/vpt> for VPT with the ViT-B/16 backbone and <https://github.com/huggingface/peft> for prefix-tuning with the RoBERTa-Large backbone.

From presented results, the proposed PEN loss can **enhance the performance** in all conducted experiments. However, it is acknowledged that determining the ideal weight for the proposed loss requires further investigation, which could be a worthy direction for our future work.

In summary, those preliminary responses our research prospect discussed in the Conclusion Section (i.e. Section 6) and we hope that our finds and approaches will provide new insights to PEFT methods and inspire future research in these fields.

Reviewer comment (Experiments):
The paper proposes two normalization methods, while only testing the effects of PEN and PAN on the experimental results respectively. Why cannot both types of losses be used simultaneously? If there is a contradiction between the two losses, it is recommended that the authors discuss the differences. If the two losses are similar, can the two losses be unified? If the two losses gain from different perspective, should relevant experiments be provided?

Author response:
**A4.** We sincerely appreciate your suggestion to explore the simultaneous combination of the PEN loss and the PAN loss. In response to this concern, we conducted experiments to investigate the combined effect of these two losses on model performance. To be specific, we evaluated various values of $\beta$, including 0, 0.1, 0.3, 0.5, 0.7, 0.9, and 1, where 0 and 1 represent the use of the PAN and PEN loss, respectively. The detailed results are shown in the following table. 
|  | $\beta$=0 | $\beta$=0.1 | $\beta$=0.3 | $\beta$=0.5 | $\beta$=0.7 | $\beta$=0.9 | $\beta$=1 |
| :---- | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| 1-shot | 59.33 | 59.45 | **59.54** | 59.29 | 59.26 | 59.37 | 59.42 |
| 16-shots | 74.20 | **74.39** | 73.37 | 74.02 | 73.93 | 73.79 | 73.47 |

The results presented in the table are obtained by average results of 11 datasets used in the paper and the best performance is shown in bold. The results display an evident pattern that combining two proposed normalization losses **improves model performance** compared to using only a single normalization loss, whether it is 1-shot or 16-shots setting. Furthermore, it can be observed that $\beta$=0.3 and $\beta$=0.1 achieve the best results, respectively. This suggests that the PAN loss plays a dominant role, while the PEN loss provides assistance, which can lead to improved performance.

We apologize for not discussing the ablation studies of $\beta$ in the paper. We understand the importance of including this information, so we have added it into the Ablation Study Section (i.e. Section 4.7) of the updated paper.

Reviewer comment (Evaluation):
Can author discuss application circumstance of two normalization methods? In practical applications, what kind of normalization loss should we choose for what situation? Suggest the authors to discuss.

Author response:
**A5.** Thank you for highlighting this concern. This paper proposed two types of normalization losses to harness the Low-Norm Effect during soft prompt-tuning vision-language models. Based on the few-shot recognition results presented in the first subfigure of Figure 2 and Table A5, it can be observed that the PEN loss generally outperforms the PAN loss for a small number of training examples (i.e., 1, 2, and 4 shots). On the other hand, the PAN loss exhibits superior performance compared to the PEN loss when using a larger number of training samples (i.e., 8 and 16 shots). This suggests that the PEN loss can be employed for settings with small shots while the PAN loss is more suitable for large shots. 

Additionally, based on the domain generalization results presented in Table 1, we can observe that the PEN loss performs better than the PAN loss in target domains, while the latter shows superior performance in source domains compared to the former. This implies that the PEN loss is more robust for domain shifts.

To sum up, **our recommendation** is to utilize the PEN loss for training samples with small shots or when handling domain shifts. Meanwhile, employing the PAN loss can significantly enhance the model's performance on intra-domain data with large shots.