Paper: Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR-2024)

Reviewer comment (Presentation):
I suggest the authors reorganize the experiment results. The current ones are confusing since the Mixed and FT-then-PT are introduced in the later part of the paper while the corresponding results are shown before.

Author response:
**A1:** Thank you for your valuable suggestion! We have added additional comments on "Mixed" and "FT-then-PT" to ensure that readers can easily follow the settings for future reference. These new comments offer a more logical and coherent sequence. Thank you.

Reviewer comment (Experiments):
In fact the original VPT has two variants, i.e. VPT-shallow and VPT-deep. It seems VPT-deep is adopted in this paper. I wonder if the same pattern holds for VPT-shallow too.

Author response:
**A3:** Thank you for your careful review. Indeed, VPT suggests two versions: VPT-Shallow and VPT-Deep. While most current work [ref2-5] focuses on comparing VPT-Deep, we use VPT-Deep as our default VPT version for all experiments. However, we do observe a similar trend in VPT-Shallow regarding the findings stated in our paper. Please note that the criteria may vary, such as the performance turning point of VPT and full fine-tuning, as discussed in Sec. 4.4, since the performance of VPT-Shallow is significantly lower than that of VPT-Deep. Thank you.

Reviewer comment (Experiments):
The current scope of this paper is ok. But it would be much greater if the authors could expand the study to prompt tuning in language models.

Author response:
**A5:** First, we would like to express our gratitude for your positive comments regarding the scope and research field of our paper. Second, it is important to note that in our paper (see Appendix E.4), we explicitly mention that the prompt tuning methods in vision draw significant inspiration from works in the domain of language. Consequently, there is a natural connection between these two domains, warranting further discussion.

During the course of our study, we found encouragement in related studies on prompt tuning in language. For instance, [ref6] highlights a similar trend in the language domain, where prompt tuning leads to enhanced performance when task-specific data is scarce (as observed in Sec. 4.4, where substantial performance gaps arise when downstream datasets are small in size). Furthermore, we explored the concept of prompt tuning from various perspectives in both language and vision domains. Specifically, [ref7] discusses the robustness of prompt tuning in handling label noises, while [ref8] demonstrates how prompt tuning empowers the model to focus on context-relevant information.

Our method delves deeper into the possible reasons and mechanisms behind the performance improvements achieved through visual prompt tuning. It also provides a thorough comparison with full fine-tuning and other competitive methods. We firmly believe that our work contributes significantly to the future exploration of intriguing topics related to prompt tuning in both vision and language.

Third, it is worth noting that the overall discussion surrounding prompt tuning studies in both vision and language remains somewhat limited. To address this, we have included additional discussions on the potential extensions to language models in Appendix Sec. G.3 and G.6, ensuring a comprehensive coverage of the subject matter.

Once again, we sincerely appreciate your feedback and interest in our work.

Reviewer comment (Evaluation):
The reason why VPT outperforms FT in scenarios characterized by distinct data distributions and high task disparity is still unclear, although the authors make a hypothesis (Section 5.3).

Author response:
**A2.4.1:** That is an excellent question. Our hypothesis is that the backbone parameters primarily capture features of the data distribution, while additional prompt parameters can contain information about the specific downstream task. When the tasks are similar but the data distribution is distinct, it becomes more crucial to learn the data distribution. This is why FT outperforms VPT in the first case. However, when both the task and the data are distinct, FT is more susceptible to overfitting, and as a result, VPT performs better when there is limited data (see Sec. 5.1). Nevertheless, it's important to note that this does not necessarily imply that VPT handles this case very effectively.

Reviewer comment (Experiments):
[Question] The experiment is limited to Transformer-based models. Do you think all the results are also valid when we use CNNs?

Author response:
**A1:** Thank you for the question. As discussed in Sec. 2, prompt tuning methods are currently available only for adapting AI foundation models to new tasks. While most of this research is focused on Transformer-based architectures, there is no proof that the findings in our paper hold for other architectures, such as CNNs. However, when discussing Transformer-based models, they have widespread applications in both vision and language, with a series of models (e.g., BERT [ref1], RoBERTa [ref2], ViT [ref3], Swin [ref4-5], PVT [ref6-7]). Our findings in this paper might provide potential contributions to the future development of such a large family of models. Thank you.

Reviewer comment (Experiments):
[Question (major)] Figure 4: Does this result include the randomness introduced by the choice of the training set samples? The broken yellow curve in the left panel looks intensely fluctuating in the small set region, and I guess this fluctuation comes from that kind of randomness.

Author response:
**A3:** Thank you for your question. It's important to take note of the randomness in the left panel. Firstly, when discussing the choice of training set samples, we adhere to common practice [ref9] and employ fixed random seeds during training. Consequently, the training data used in both full fine-tuning and visual prompt tuning should remain consistent. Secondly, we also observe that randomness is rarely encountered in other datasets, occurring in only 1 out of 19 datasets. Thirdly, based on the visualization of the dataset, SVHN [ref15] is a digit classification benchmark dataset containing 600,000 32×32 RGB images of printed digits cropped from pictures of house number plates. When collecting a specific number of images (i.e., dataset capacity) for training, the distribution for each class might vary, potentially leading to the observed oscillations. Thank you.

Reviewer comment (Experiments):
[Question (major)] In one-shot classification experiment (Table 3), how did the authors choose the single training sample? The performance should highly correlate with the one-shot training sample.

Author response:
**A4:** Thank you for your question. The selection of a representative image for each class is crucial, and we aim to ensure that it is not an outlier [ref16]. Instead, it should exhibit common features shared with other images from the same class. To achieve this, we employ a method called iterative testing. Specifically, we experiment with different images from the same class and assess our model's performance. If the model's performance falls within a reasonable range, we conclude that the selected images are representative. Additionally, we apply data augmentation methods to mitigate image variations. Further discussion on one-shot learning can be found in Appendix Sec. G.4. Thank you.

Reviewer comment (Evaluation):
[Comment] At the end of Section 4, "These results further support our assumption that prompt tuning is more effective than full finetuning when only limited finetuning examples are available." Doesn't this statement conflict with the Patch Camelyon result (green curves) in Figure 4?

Author response:
**A2.2:** The Patch Camelyon dataset belongs to the "low task disparity and distinct data distributions" quadrant (Figure 3); therefore, full fine-tuning outperforms VPT. We have made it explicit that this statement is limited to the three quadrants mentioned above.

Reviewer comment (Presentation):
[Comment] Figure 5 should use log scales. Also, all figures including Figure 5 should be vector images.

Author response:
**A5:** Thank you for your suggestions. We have attached the log-scale version of Figure 5 in Appendix Section A, which is Figure 9. Additionally, we have ensured that all figures are in vector image format. Once again, thank you.

Reviewer comment (Theory):
[Comment] In Section 5.2, "Another possible explanation for why prompt tuning can achieve superior performance is that it can escape from local minima or saddle points compared to full finetuning, due to the additional dimensions introduced by the prompts." This is clearly an overstatement (or a misuse of words). Local minima and saddle points are rigorously defined in a mathematical way. I guess what the authors mean here is if we can observe performance gain or not. However, strictly speaking, escaping from local minima or saddle points is not a necessary nor sufficient condition of generalization in general settings. Although I understand some papers use these words interchangeably, and my comment may be a bit nitpicking, I would like to recommend that the authors modify the statement here. By the way, it will be a dramatic research if one theoretically proves that VPT in fact makes the network escape from local minima or saddle points.

Author response:
**A2.3:** Thank you for your constructive feedback. We agree that the hypothesis could be better phrased: it is possible that the additional dimensions introduced by the prompt simply help optimize the loss further, and our experiments have shown that this hypothesis is not true. We have modified the statement to better reflect this.

Reviewer comment (Experiments):
Although the hypotheses in Sec. 4.3 and Sec. 5.2 look very interesting and inspiring (as mentioned in the Strength point 4), I’m concerned the experiments are not convincing enough to support them. In Sec. 4.3, it’s a bit risky to draw the conclusion using only 19 data points. Adding more tasks including the detention or dense prediction ones could be better.

Author response:
**A1.1:** Thank you for your concern regarding the number of datasets we employed in this work.

First, we would like to clarify that we formally utilized all 19 datasets from the VTAB-1k benchmark for all our experiments to maintain consistency. This approach is notably comprehensive in terms of dataset quantity, in contrast to previous studies in the field of vision prompt tuning, which limited their investigations to significantly fewer datasets. Specifically, [ref1-2] focused their analyses on single and eight datasets, respectively. The extensive utilization of our dataset not only distinguishes our work but also significantly enhances its completeness and generality.

Secondly, to address your concern further, we conducted a semantic segmentation experiment on ADE20K [ref3], resulting in a corresponding FID score of 260.7. As observed, when such a higher FID is obtained, full fine-tuning leads to superior performance. Our results align with the trend reported in VPT [ref4] and TOAST [ref5] concerning semantic segmentation.

Thirdly, we present additional results on the FGVC benchmark [ref4], which includes five benchmarked Fine-Grained Visual Classification datasets: CUB-200-2011 [ref6], NABirds [ref7], Oxford Flowers [ref8], Stanford Dogs [ref9], and Stanford Cars [ref10]. A higher FID score demonstrates higher accuracy in full fine-tuning, and vice versa, consistent with the hypothesis outlined in Section 4.3. We have included these additional experiments in Appendix Section G.3. Thank you for your attention.


| Dataset  | ADE20K |
| :-: | :-: |
| Full fine-tuning | 36.54 |
| VPT | 33.42 |

| Dataset  | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars |
| :-: | :-: | :-: | :-: | :-: | :-: |
| Full fine-tuning | 87.3 | 82.7 | 98.8 | 89.4 | **84.5** |
| VPT | **88.5** | **84.2** | **99.0** | **90.2** | 83.6 |
| FID | 146.845 | 169.117 | 167.690 | 143.840 | 182.487 |

Reviewer comment (Evaluation):
Although Fig. 7 in Sec. 5.2 is very intriguing, the experiment results are not able to support it, i.e., Mixed and FT-then-PT still fall behind VPT with clear margins. This also leads to a very ambiguous conclusion in the last paragraph in Sec. 5.2.

Author response:
**A1.2:** 
Apologies for the confusion. Figure 7 illustrates one of the potential hypotheses, but in our experiments, we have discovered that this hypothesis is **not accurate**. As you correctly noted, the results suggest that the additional dimensions do not significantly assist the model in escaping local minima. This observation was reflected in the title of Section 5.2, as well as in the last paragraph of Section 5.2. We have further enhanced the clarity of Figure 7's caption by explicitly stating that this hypothesis is not supported. Thank you.

Reviewer comment (Presentation):
In Tab. 1, Mixed and FT-then-PT are included without any introduction in the Sec. 4.1 alongside, which could be very confusing to the audience. Maybe remove them from Tab. 1 and make another comparison table in Sec. 5.2, where these two methods are actually introduced?

Author response:
**A3:** Thank you for bringing this to our attention! We have made the necessary updates to the content orders in our paper to enhance clarity. Specifically, we have included additional instructions on Mixed and FT-then-PT for future reference within the paper. Thank you.

Reviewer comment (Presentation):
The number of wins notation was introduced in Tab. 3 but has already appeared in Tab. 1 and 2 before, which could confuse the audience. Also, ‘[]’ was used for both this notation and also reference. Maybe change it to ‘()’ to avoid any confusion?

Author response:
**A4:** Thank you for your valuable suggestion! We have made the necessary adjustments to address the confusion regarding the notations (see Table 1 and 3) and the use of "[]" in all tables within our main paper and Appendix. Please refer to our revised paper for more details. Thank you.

Reviewer comment (Novelty):
A similar observation on data scale (Sec. 4.4) is actually also covered in the original VPT paper/experiments (Page 8, Fig. 3).

Author response:
**A5:** Thank you for your thorough review. We would like to address this concern in two aspects:
First, we would like to address the claim made in [ref4], which partially answers this question by stating, "In contrast, VPT-deep still consistently outperforms Full across training data sizes." Our experiments in Sec. 4.4 demonstrates that this claim is untenable. Furthermore, we present evidence that full fine-tuning actually yields superior results as the dataset scale continues to expand. In our study, we take a critical approach to the claims made in the previous research, rather than simply adopting them in their original form.
Secondly, in the case of VPT, the results are presented on a different dataset benchmark (i.e., FGVC). Our paper contributes an additional piece to the puzzle of addressing gaps in common image recognition benchmarks. To provide a comprehensive perspective, we have included an extended discussion in our paper that elucidates the relationship between our work and the findings of VPT. However, we acknowledge the need to incorporate a discussion regarding training data scale with VPT, and we have addressed this in Appendix Sec. G.3.
Thank you once again for your valuable feedback.

Reviewer comment (Experiments):
In Section 5.4, the authors show that the GradCAM maps of VPT is more interpretable and more focused on task-relevant regions than fully fine-tuning. It seems the examples are all from natural image classification. It would be helpful to compare the attention on four quadrants of transfer learning to see the difference when data/task disparity varies and how it is connected to the conclusion in Section 4.

Author response:
**A1:** This is an excellent point! The examples currently presented in Figure 8 and Appendix Sec. B are drawn from a single dataset in Natural, which might introduce bias in our understanding of the effect of VPT on features. However, during our experiments, we consistently observed visual evidence across different datasets. We have included attention visualization (i.e., GradCAM) for the other three quadrants of transfer learning in Appendix Sec. B. The visual evidence suggests that in the 1st-3rd quadrants, visual prompt tuning generally results in a denser and more accurate focus on desired positions compared to full fine-tuning. For datasets falling into the 4th quadrant, our results reveal noticeable improvements when subjected to full fine-tuning. Thank you.

Reviewer comment (Experiments):
The visualization of attention in Section 5.4 is based on GradCAM. Have the authors tried visualizing the attention from cls token to other tokens, as in previous work in the ViT literature (e.g., [1])? On the other hand, a previous paper [2] also discusses the relation between transfer learning and attention. They find that current transfer learning methods tend to have noisy attention and fixing the attention can boost the transfer learning performance. It would be helpful to add some discussion on the relation between this work and [2].

Author response:
**A2.1:** Thank you for your question. Currently, we adhere to common practices [ref1-3] and visualize the final layer of the network. However, it's worth noting that an alternative approach to understanding the impact of VPT on features is to visualize the attention from the cls token to other tokens [ref4]. We have also attempted to visualize the patch embeddings from the transformer, and these visualizations have consistently yielded meaningful insights. Given that the cls token is explicitly designed for classification, we find it reasonable to visualize the attention from the cls token to other tokens. We plan to delve deeper into these results in our future studies. Thank you.  

**A2.2:** 
Thank you for the reference! We consider [ref5] to be a highly significant work, particularly in the context of the discussion regarding the relationship between attention, visual evidence, and performance. In current research, it is often the case that visual evidence cannot offer an intuitive or straightforward explanation for performance improvement, but [ref5] provides a pathway to establish such a connection. We believe this topic is exceptionally intriguing and can substantially enhance our reassessment in Section 5.4. We have included a discussion on this in Appendix G.6 as part of our future work. Thank you.

Reviewer comment (Experiments):
This paper is focused on VPT. Since other transfer learning methods such as LoRA [3] may have more favorable performances on various transfer learning tasks like VTAB-1k, readers may wonder if these methods also have similar properties. It would be helpful to see if the same conclusions also hold for other transfer learning methods.

Author response:
**A3:** Thank you for bringing this to our attention. We have observed that [ref5] presents a comparison of several baselines for transfer learning, including linear, full fine-tuning, VPT, and LoRa [ref6], which yield valuable insights. In this paper, our primary focus is on visual prompt tuning, an area that remains largely unexplored and has not been systematically studied. We narrow our scope to conduct a comprehensive and systematic investigation.

When discussing LoRa, it introduces low-rank metrics and exclusively tunes these metrics while keeping other components frozen. While there are certain similarities between these two methods (namely, both can be categorized as parameter-efficient approaches [ref7-8]), their distinctions may lead to ambiguous results. We contend that our paper takes a substantial step toward a deeper understanding of prompt tuning. In the future, we naturally plan to expand our study to include LoRa and other parameter-efficient techniques.

During the rebuttal phase, we managed to provide some results on LoRa (specifically, we varied the dataset scale on CIFAR-100 from 400 to 10,000 training samples for five runs and compared LoRa with the Full fine-tuning method) in the table below (following the code provided by TOAST [ref5], utilizing the ViT-B-in21k (bottom-up) checkpoint). As depicted, as the dataset scale increases, the gap between LoRa and full fine-tuning gets narrow significantly, aligning with our observations.

We have included further discussion in Appendix Sec. G.6. Thank you for your attention.


| # training sample  | 400 | 800 | 10000 |
| :-: | :-: | :-: | :-: |
| Full fine-tuning | 44.47 | 70.2 | 87.9  |
| LoRa | 69.6 | 83.6 | 90.7  |
| Performance gap | 25.1 | 13.4 | 2.8 |