Paper: Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks a comprehensive comparison with existing DML methods, especially those using transformer based backbone.

Author response:
**Addressing the Need for Comprehensive Comparison with Existing DML Methods**

In alignment with our response to **Reviewer 1 (hvFX)**, we wish to further clarify the baseline methodologies used in our study. The "full fine-tuning" baseline, as discussed in both our main paper and Appendix, is based on the proxy-anchor approach [1], which is recognized as one of the state-of-the-art methods in DML. Additionally, we have conducted comparisons with other leading approaches, such as Hyp-ViT [2] and RS@k [3], which are also based on ViT architectures and considered state-of-the-art.

It is important to note that all these state-of-the-art methods [1][2][3], while varying in their optimization loss, share a commonality in their computational costs, as they are all fundamentally full fine-tuning methods. To provide a more detailed comparison, we have included an additional analysis in Table 6 in the Appendix. This table compares our methods with those mentioned and other existing DML methods, offering a comprehensive comparative view.

For an in-depth understanding of these comparisons and the rationale behind our approach, we kindly refer you to our detailed response to **Reviewer 1 (hvFX)**, where we elaborate on these points further.

Reviewer comment (Experiments):
It would be beneficial to compare the proposed Visual Prompts (VPT) framework not only against full fine-tuning but also against other state-of-the-art parameter-efficient methods, as this would provide a more complete assessment of its effectiveness and novelty.

Author response:
**Addressing the comparison with other parameter-efficienct fine-tuning methods**

We would like to draw your attention to our comprehensive analysis comparing our VPT method with several leading parameter-efficient techniques in the DML benchmark. This detailed comparison is outlined in Section 5.1.2 of our paper. The results of this comparative study are showcased in Table 1 of the main manuscript, where we assess and compare the performance of each method in terms of DML metrics (specifically R@1 and MAP@R), the number of adjustable parameters, and memory usage.

To ensure a fair and unbiased comparison, as elaborated in Section 5.1.3, we conducted an extensive hyperparameter search for each method. This rigorous approach ensures that our comparative analysis is not only comprehensive but also equitable, taking into account the varied characteristics and capabilities of each parameter-efficient method under consideration.

We trust that these clarifications address your concerns regarding the comparative analysis of our methods with other DML approaches. Your constructive feedback is invaluable in our pursuit of continual improvement of our manuscript.

Reviewer comment (Evaluation):
It seems the results are good on large pre-training datasets but fair on small pre-training. Could you explain more about this phenomenon?

Author response:
**Performance on Large vs. Small Pre-training Datasets**

As elaborated in the Experimental section, we posit that the performance of VPT and other parameter efficiency methods is influenced by the distributional variances between pre-training and fine-tuning datasets. Smaller pre-training datasets like ImageNet-1K often display a more significant variance in distribution compared to target datasets. This observation is consistent with prior findings in VPT research [1], lending support to our analysis.

Reviewer comment (Presentation):
It is suggested to move some results in the Appendix, like results on other datasets and pre-trained models, to the main paper.

Author response:
**Reorganization of Results and Appendix Content**

In response to your suggestion, we have included the results from the iNaturalist dataset in the main paper (now in Table 3). Due to space and format constraints, we are unable to relocate more results from the Appendix to the main text. We believe that the current structure of the manuscript strikes a balance between detail and conciseness, effectively presenting our findings.

Reviewer comment (Writing):
Some typo or grammar errors: "which includes a random horizontal flip, cropping at random..." "we found that it linear probing..."

Author response:
**Attention to Minor Details**

All minor points raised in your review have been meticulously addressed and corrected. We believe these revisions significantly enhance the clarity and accuracy of our manuscript.

Reviewer comment (Experiments):
The paper claims the proposed method to be parameter-efficient for deep metric learning (DML) tasks, but there is not comparison in term of efficiency as compared to existing DML methods.

Author response:
**Comparison in Terms of Efficiency with Existing DML Methods**

Aligning with the metrics in existing parameter-efficient works [1][2], we assess efficiency by comparing the number of tunable parameters in the model, which reflects computational costs per iteration. Most existing DML methods [4][5] rely on full fine-tuning optimization with varying loss functions or sampling strategies, making their efficiency comparable to our full fine-tuning baseline.

Furthermore, we have compared actual running latencies in Section C and Table 8 of our Appendix (formerly Supplementary Materials). This table shows the average running time per iteration in identical environments across 100 trials. Our results show that our proposed VPT and other parameter-efficient methods significantly reduce latency compared to full fine-tuning approaches. It's important to note that running latency is subject to hardware and software configurations. Hence, parameter-efficient works [1][2] often use tunable parameters as a universal metric for evaluating fine-tuning efficiency.

Reviewer comment (Experiments):
A baseline using default/vanilla proxy-based deep metric learning (without visual prompt learning) is missing to convince the effectiveness of introducing visual prompt tuning.

Author response:
**Consideration of More Existing DML Methods and a Vanilla Proxy-Based Baseline**

In response to your concern, we have made the following clarifications and additions:
1) Our full fine-tuning approach, detailed in the main paper and Appendix, is based on the vanilla proxy-anchor (PA) [3] methodology, enhanced with a ViT backbone and larger ImageNet-21k pre-training datasets. We have included comparisons with state-of-the-art methods like Hyp-ViT [4] and RS@k [5], which also utilize a ViT backbone. These comparisons are specific to ViT backbones trained on ImageNet-21k, as earlier works [3][6][7][8] with CNN backbones and ImageNet-1k training show significantly lower DML benchmark performance.
2) Most current state-of-the-art methods, including Hyp-ViT [4] and RS@k [5], use full fine-tuning of the ViT with different losses or sampling strategies, similar to our PA baseline. These methods often require larger batch sizes (as indicated in Table 2), leading to higher latencies compared to our baseline. Therefore, their overall efficiency is close to or below our full fine-tuning PA baseline, while our VPT methods demonstrate higher DML performance, as shown in Table 2.
3) Although it's not entirely fair to compare with earlier works using different backbones and pre-training models, we have included performance and tunable parameters for some earlier works [6][7][8] and the vanilla proxy-anchor [3] as references. This comparison is also added to our Appendix as Table 6.

***
|Method 	              | Arch | Parameters 	| R@1 | R@2| R@4| MAP@R | 
| :---------------------- | :------- | :----------------- | :------ | :------- | :----- | :------ |
|ProxyNCA++[6]     | R50 | 26.5M 		| 69.0 | 79.8 | 87.3 | - |
|NIR [7]	              | R50 | 26.5M             | 69.1  | 79.6 | -    | -  |
|XBM[8]                  | BN	| 12M                | 65.8 | 75.9 | 84.0 |   - 	|
|PA (vanilla)[3]       | R50 | 26.5M              | 69.7 | 80.0  | 87.0 | 26.5 |
|Hyp[4] | ViT-S/16 | 30.2M			| 85.6 | 91.4 | **94.8** | - |
|Full fine-tuning (PA) | ViT-S/16 | 30.2M  |  85.1  | 91.1 | 94.0 | 51.1 |
|VPTSP (ours) | ViT-S/16 | **1.56M**          | **86.6** | **91.7** | 94.7 | **52.0** |
***

Reviewer comment (Evaluation):
What is the computational cost of the proposed method as compared to (1) vanilla proxy-based DML method, (2) other fine-tuning methods such as BitFit and Adapter? Please discuss the comparison on computation cost for model training and inference.

Author response:
**Q1: What is the Computational Cost of the Proposed Method**

As mentioned, we evaluate our method's computational cost in two aspects:
1) The total trainable parameters that are updated in each optimization step (shown in Table 1).

2) The actual running latency per optimization step (shown in Table 8 of the Appendix).

Table 1 in the main paper demonstrates that our method requires only about 5% of the trainable parameters needed for full fine-tuning methods. For running latency, our method significantly undercuts full fine-tuning methods, as listed in Table 8 of the Appendix.

Reviewer comment (Experiments):
How does the proposed method compare to the state-of-the-art deep metric learning methods?

Author response:
**Q2 and Q3: Comparison to State-of-the-Art DML Methods and Vanilla Proxy-Based DML**

To better represent our "full fine-tuning method" as a stronger baseline compared to the vanilla proxy-anchor, we have added explanatory text in our revision. We have also updated Table 1 in the main paper and Table 8 in the Appendix for greater clarity. Additionally, a new table (Table 6 in Appendix) comparing our proposed method with existing DML works using ViT backbones [4][5] and earlier DML methods with CNN backbones [3][6][7][8] has been added.

We hope these revisions and additional explanations comprehensively address your concerns, thereby enhancing the clarity and impact of our work. We are deeply appreciative of the opportunity to refine our manuscript based on your thoughtful feedback and eagerly await any further suggestions you may have.

Reviewer comment (Experiments):
How does the proposed method compared to the vanilla proxy-based deep metric learning in terms of model performance and compute cost?

Author response:
**Q2 and Q3: Comparison to State-of-the-Art DML Methods and Vanilla Proxy-Based DML**

To better represent our "full fine-tuning method" as a stronger baseline compared to the vanilla proxy-anchor, we have added explanatory text in our revision. We have also updated Table 1 in the main paper and Table 8 in the Appendix for greater clarity. Additionally, a new table (Table 6 in Appendix) comparing our proposed method with existing DML works using ViT backbones [4][5] and earlier DML methods with CNN backbones [3][6][7][8] has been added.

We hope these revisions and additional explanations comprehensively address your concerns, thereby enhancing the clarity and impact of our work. We are deeply appreciative of the opportunity to refine our manuscript based on your thoughtful feedback and eagerly await any further suggestions you may have.