Paper: Curriculum metric learning for robust image retrieval (ICLR-2024)

Reviewer comment (Novelty):
Lack of novelty: The paper's main finding, that using adversarial trained models to initialize retrieval models improves their robustness against adversarial ranking attacks, is not surprising. Adversarial training has been widely studied and shown to enhance robustness in various downstream applications. Therefore, this finding cannot be considered as novel. If there are any novel aspects, they might lie in the quantitative experiments conducted on the improvement in retrieval tasks.

Author response:
We appreciate your feedback, and we revised the Contributions / Conclusions section to better highlight our contributions:
  * We wanted to point out that our paper is the first, to the best of our knowledge, to explore and quantify the adversarial ranking robustness of adversarial trained models. This contribution is significant as it opens a new avenue in understanding how adversarial training affects learned features, and adds ranking robustness as an additional benefit to robust representations, in addition to for example better transferability [R1], or explainability [R2].
  * We introduce a two-step approach inspired by curriculum learning, which as demonstrated experimental results significantly outperforms existing methods in defending against ranking attacks. This improvement is not merely incremental and represents a substantial improvement over baseline methods. Our approach provides a more effective (across all tested defenses and datasets) defense mechanism, which is crucial for the practical application of these models in real-world scenarios.

Reviewer comment (Novelty):
Ambiguity in the concept of curriculum learning: The paper claims to propose a CURRICULUM METRIC LEARNING method, but it is observed that there is a lack of direct connection between the pre-training task and the subsequent adversarial training task in retrieval. Additionally, the authors have not provided any contributions related to the curriculum. As a result, it is challenging to perceive the proposed method as a CURRICULUM METRIC LEARNING approach.

Author response:
Regarding your concerns about the curriculum metric learning method proposed in our paper, we recognize the need for a clearer explanation. Our approach utilizes curriculum learning by proposing to break down the learning of robust retrieval representations into solving two progressively difficult sub-tasks: (1) learning robust features using adversarial training, which is well understood and explored, (2) followed by robust retrieval training. We hypothesize adversarial training is an easier task, as we do not try to directly manipulate the feature space as one does in robust metric learning. Therefore, we hypothesize it will be easier to first learn robust features via (AT), followed by better aligning features to me semantically meaningful while remaining robust metric learning.

Reviewer comment (Evaluation):
Insufficient analysis of experimental conclusions: Although the experimental results demonstrate performance improvement, there is a lack of in-depth analysis. The authors could further explore the reasons behind the improvement and explain why specific adversarial training initialization methods are more effective in enhancing the robustness of retrieval models. Such analysis would strengthen the credibility of the experimental conclusions.

Author response:
We acknowledge your point on the need for a more in-depth analysis of our experimental results. In the revised version of our paper, we include a discussion on why adversarial pre-training methods are so effective in improving robust metric learning, and discussion on the proper choice of the strength of initialization robustness, and the effect of sampling strategy on learning.

Reviewer comment (Theory):
Can the authors provide a more comprehensive analysis or theoretical justification for why adversarial training initialization is particularly crucial for enhancing robustness in retrieval tasks? This would help to better understand the underlying mechanisms and provide insights into the importance of this approach in the context of retrieval models.

Author response:
As outlined in response to W3, we provide a discussion on the theoretical intuition on why AT is crucial for enhancing ranking robustness.

Reviewer comment (Experiments):
Considering the characteristics of robust feature learning and the unique property of retrieval models, are there any specific modifications or adaptations that can be made to further optimize the utilization of adversarial training for improved robustness in retrieval? It would be valuable to explore how to harness the advantages of robust feature learning and tailor it specifically for retrieval tasks, rather than applying it in a generic manner.

Author response:
This is a very good question. At this juncture, we must candidly admit that we do not have a comprehensive answer to your question.

Reviewer comment (Presentation):
This paper makes the readers hard to follow due to the following reasons: a) most of the figures and tables are not clear due to a lack of description of notations. For example in Figure 2, what are EST, ACT, and HM? What does the term ERS represent? In addition, in Table 1, what are CA+, CA-, QA+, QA-, etc? b) this paper contains several typos and grammar issues. These above parts make the readers confused and easily get lost.

Author response:
* We have revised Figure and Table captions to enhance the notations and descriptions thus improving paper clarity. We appreciate you pointing out the editorial issues – we revised the paper and aimed to correct the grammatical mistakes, thus making the paper easier to read.

Reviewer comment (Theory):
This paper needs more theoretical analysis of the proposed learning method to better support the argument of this work, not simply empirical results. Moreover, a comparison between the proposed method and some existing curriculum learning related methods could be made to better demonstrate the efficacy of this work.

Author response:
* As Reviewer Khkm had similar concerns, we add an additional subsection to provide intuition of why sub-task curriculum provides such a significant improvement in learning robust and meaningful representations for retrieval. We do believe a more detailed theoretical analysis is warranted and is left for future work. We however strongly believe these empirical results are interesting in their own right and the lack of theoretical analysis does in no way detract from the value of this work.

Reviewer comment (Novelty):
As to the related work part, more curriculum learning related work in the robust feature learning area might be added to better locate the position of this work.

Author response:
* In order to better locate the position of our work, we add a small addition to the related work section to illustrate typical uses of curriculum learning and how our work relates to it.

Reviewer comment (Theory):
The theoretical analysis of the proposed method is lacking.

Author response:
* Weakness 1 / Question 1: As described previously, we add an additional subsection to provide intuition of why the sub-task curriculum provides such a significant improvement in learning robust and meaningful representations for retrieval. We do believe a more detailed theoretical analysis is warranted and is left for future work. We however strongly believe these empirical results are interesting in their own right and the lack of theoretical analysis does in no way detract from the value of this work.

Reviewer comment (Novelty):
The key ideas of the paper are not well presented and also make the novelty not clear enough.

Author response:
* Weakness 2: We revised the Abstract, Contributions, Background and Conclusions sections to better highlight the novelty of our work.

Reviewer comment (Theory):
The main concern is the lack of theoretical analysis of the proposed method in the paper. The authors have given several hypotheses and abundant empirical experimental results. However, some necessary theoretical analysis is missing.

Author response:
* Weakness 1 / Question 1: As described previously, we add an additional subsection to provide intuition of why the sub-task curriculum provides such a significant improvement in learning robust and meaningful representations for retrieval. We do believe a more detailed theoretical analysis is warranted and is left for future work. We however strongly believe these empirical results are interesting in their own right and the lack of theoretical analysis does in no way detract from the value of this work.

Reviewer comment (Writing):
What is the meaning of beta in Formula 2? Authors should give clear descriptions of these formulas.

Author response:
* Question 2: Parameter “beta” in what was previously Equation 2 (Now 3 in the Appendix) is the margin parameter in the triplet loss.

Reviewer comment (Theory):
In Figure 4, why does the random triplet sampling strategy achieve better performance than EST and ACT? More theoretical analysis should be given.

Author response:
* Question 3: We hypothesize the Random sampling strategy outperforms the Soft-hard sampling strategy due to the conflict of how Soft-hard sampling generates triplets and the goal of the adversary in EST / ACT. Soft-hard sampling generates triplets that “likely” have hard positives and hard negatives [R1], that are likely hard to collapse (ACT), making the random sampling more effective in this case. Please see Appendix A1 for further discussion.

Reviewer comment (Reproducibility):
Please give more details about the mentioned curriculum learning strategy in the paper.

Author response:
* Question4: Please see the Background section for the expanded description of the curriculum learning strategy.