Paper: Prompt Tuning Is All We Need? (ICLR-2024)

Reviewer comment (Presentation):
Title Clarity. The current title appears too broad and potentially misleading readers. Initially, one might assume the paper provides an analysis of the pros and cons of prompt tuning across various tasks. However, the main content focuses on the present new method of tuning the CLIP image encoder, which is not reflected in the title.

Author response:
Thank you for your feedback on the Title Clarity. We primarily intended to convey our research findings in the context of domain generalization tasks. We appreciate your suggestion, and we will make sure to revise our title accordingly.

Reviewer comment (Experiments):
Missing baselines. The authors only compare with CoOp/CoCoOp in Table 1 and Table 2, which are not enough. CoOp/CoCoOp are published two years ago and not the SOTA method. A series of methods mentioned in the related work section are suggested to compare with, such as ProGrad (Zhu et al., 2022), Test-time prompt tuning (Shu et al., 2022) and MaPLe (Khattak et al., 2023). They also provide the results on the base-to-new evaluation and cross-dataset transfer settings. Some recent prompt tuning approaches that are not listed in the related work section are also suggested to compare with, such as LASP Language Aware Soft Prompting for Vision-Language Models, CVPR 2023.

Author response:
Thank you for your feedback regarding the missing baselines. Our Im-Tuning method outperforms MaPLe and ProGrad in domain generalization tasks. Specifically, we conducted experiments on three DG datasets, namely PACS, Office-Home, and DomainNet. Our method achieved an average accuracy of 87.2%, while MaPLe achieved an average accuracy of 86.7%, and ProGrad achieved an average accuracy of only 85.4%. Additionally, our method is applicable to networks with ResNet as the backbone, further improving domain generalization performance, which is not achievable by MaPLe.

Reviewer comment (Experiments):
One main work [1] to compare in experiments is missing, as it focuses on multi-modal tuning. To convince the conclusion that tuning prompt and image encoder is better than multi-modal prompt tuning, more empirical results are required.

Author response:
Thank you for reading and appreciating our work! We address your questions and concerns below.  
Q1:One main work to compare in experiments is missing, as it focuses on multi-modal tuning. To convince the conclusion that tuning prompt and image enocder is better than multi-modal prompt tuning, more empirical results are required.

A1：Thank you for providing the baseline. We conducted additional experiments on three domain generalization datasets, and our results outperform MaPLe. On PACS, Office-Home, and DomainNet, MaPLe achieves an average accuracy of 86.7%, while our Im-Tuning achieves 87.2%. Furthermore, our experiments provide insights into methods like MaPLe that focus on multi-modal tuning. These methods are effective because they improve the representation of image features, making them more separable. We attempted to remove the image branch in MaPLe and observed a significant decrease in performance, which validates our viewpoint.

Reviewer comment (Reproducibility):
Could the authors provide more detailed experimental settings and implementation details for reproducibility?

Author response:
Thank you for raising this question. We apologize for any confusion caused.We use the SGD optimizer to train the model with a learning rate of 2e-3 and betas values (0.9, 0.999), which is decayed by the cosine annealing rule. we actually employ a similar cross-entropy loss function as used in CoOp to train the entire model.

Reviewer comment (Experiments):
It would be interesting to see more analysis and comparisons with other multimodality models.

Author response:
This is an excellent question. We have observed a similar phenomenon of indistinguishable image features within the same category in our further experiments with the ALIGN model. We attribute this to potential induction biases that arise during the training process of large-scale multimodal models based on contrastive learning. For example, many images of dogs may have different textual descriptions associated with them during training, and the textual variations can cause the model to push their image features further apart, resulting in the phenomenon of inseparable image features in the classification task. Our method aims to mitigate this issue to some extent and achieve higher accuracy. We will further analyze and compare our approach with other multimodality models as suggested.