Paper: Threshold-Consistent Margin Loss for Open-World Deep Metric Learning (ICLR-2024)

Reviewer comment (Experiments):
The biggest weakness in the paper is the lack of experiments related to face verification, where threshold importance is evident. While image retrieval mostly uses metrics like mAP or Recall@k, face verification relies heavily on thresholds and uses metrics like TAR@FAR. The introduced method appears more suited for face verification than image retrieval.

Author response:
**A1** We sincerely appreciate the reviewer's suggestion and input. However, we would like to point out that the focus of our work is to introduce the notion of threshold consistency for image retrieval in the general image domain and bring awareness of the threshold inconsistency phenomenon in deep metric learning. We want to emphasize the crucial role that the threshold plays in real-world image retrieval systems, where a threshold-based retrieval criterion is often preferred over a top-k approach due to its ability to identify negative queries that do not have matches in the gallery. However, the issue of threshold inconsistency in DML can often complicate the process of selecting thresholds when deploying large-scale commercial image retrieval systems in practice. To address this practical challenge, our paper presents two key contributions: the OPIS metric and the TCM regularization loss, both designed to mitigate the threshold inconsistency problem in deep metric learning.

Regarding the application of our work to face verification, we acknowledge the relevance of threshold consistency in this context. However, due to the potential significant legal risks with using technologies that may generate or process biometric information, we are not able to use datasets where we cannot ensure that our use the datasets complies with applicable law, such as the [Illinois Biometric Information Privacy Act](https://www.ilga.gov/legislation/ilcs/ilcs3.asp?ActID=3004&ChapterID=57) (BIPA). For example, prior to using a dataset for such purposes, we must determine where the data subjects reside and confirm that the data subjects received the required notice and provided the required consent for us to use their data for our proposed purpose. Because we cannot ensure that the proposed use of the face datasets for face verification experiments complies with applicable laws, we are unable to use these datasets as you request. In addition, noteworthy organizations like National Institute of Standards and Technology (NIST) have discontinued the distribution of all three IJB datasets (IJB-A, IJB-B, and IJB-C) required for standard face recognition evaluation in the literature (refer to [link](https://www.nist.gov/programs-projects/face-challenges)). We kindly request suggestions from the reviewer on alternative experiments that we can undertake to address your concern without compromising privacy or violating applicable laws. Your insights and recommendations would be greatly appreciated.

Reviewer comment (Experiments):
The paper mentions related works like Liu et al. (2022) and OneFace, but experiments comparing the proposed method to these in the realm of face recognition are missing. Such comparisons are necessary to understand the proposed method's improvements in threshold consistency.

Author response:
We would like to express our gratitude once again for your valuable reviews. We are committed to fully addressing all of your concerns. To avoid potential legal risks associated with using real human face datasets, we have opted to utilize a synthetic face dataset, DigiFace [5], to demonstrate the effectiveness of our TCM method in comparison with OneFace in the realm of face verification. Same as the open-world setting described in the main paper, we randomly split 90% of person identities for training, reserving the remaining 10% of identities for testing. The training and testing identities (classes) are completely disjoint to each other. The results in the table below gives a comparison ArcFace [6], OneFace [1] and our proposed TCM method. Notably, as OneFace does not open-source their code, we followed their paper and implemented based on the description in the paper. As shown in the table, our method outperforms both ArcFace and OneFace in terms of threshold consistency (measured by OPIS, the lower the better), while achieving the highest accuracy (measured by TAR@FAR=1e-4, the higher the better) among the three. We hope this experiment adequately addresses your concerns about experiments in face verification. Please do not hesitate to reach out if you have any additional questions or concerns.

Table 3. The performance of ArcFace, OneFace and our proposed method on DigiFace in TAR@FAR=1e-4 and OPIS.

|        	        |    Backbone-embedding dimension	    | TAR@FAR=1e-4 (%)	  | OPIS x 1e-2	  |
|:---------------:|:---------------:|:-----------------:|:-------------:|
|    ArcFace	     | ResNet100-128	  |       98.20	       |     1.42	     |
|    OneFace	     | ResNet100-128	  |      99.03	       |     1.41	     |
| ArcFace + TCM	  | ResNet100-128	  |      99.54	       |     1.28	     |

Reviewer comment (Experiments):
The paper needs to update state-of-the-art results on the CUB and Cars-196 datasets [1]. [1] Kim et al., HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization, CVPR 23

Author response:
**A4** Thank you for pointing out the state-of-the-art methods. We appreciate your suggestion to ensure the accurate representation of our paper. We have promptly updated the numbers in Table 5 of the main paper to reflect the latest progress. For a fair comparison, we also conducted experiments using the same backbone (DINO) as HIER [4] on the Cars dataset. As indicated in the table, TCM outperforms HIER in both OPIS and accuracy, all while reduces training time. We acknowledge and regret any inadvertent oversight in not being initially aware of the SOTA method and we appreciate its identification being brought to our attention.

Table 1. The performance of HIER and our proposed method on Cars in Recall@1 and OPIS.

|  | Backbone-embedding dimension	 |Recall@1	| OPIS x 1e-3	  | Time for Loss Computation (s / iteration)|
|:-----------------:|:---------:|:---:|:-------------:|:-------------------------------------------:|
| ProxyAnchor+HIER| DINO-384|91.7 *| 0.55|0.41|
| ProxyAnchor+TCM| DINO-384 |**91.9**|**0.41**|**0.22**|

*We use the released codebase and use the same hyper-parameter described in [4], and our reproduced result is better than the Recall@1 of 91.3 reported in [4].

Reviewer comment (Evaluation):
Figure 3 shows ProxyAnchor (ResNet50) with a low threshold consistency. It would be beneficial to compare the improvement in R@K and OPIS when using the proposed method. Ideally, the proposed method should show a significant OPIS improvement compared to others.

Author response:
**A5** We apply TCM on top of ProxyAnchor as required. As illustrated in the table and hypothesized by the reviewer, our proposed method demonstrates a significant performance improvement over the base loss in both Recall@1 and OPIS on the iNaturalist dataset.

Table 2. The performance of applying TCM on top of ProxyAnchor on iNaturalist-2018 in Recall@1 and OPIS.

|  |Backbone-embedding dimension | Recall@1 |OPIS x 1e-3|Relative improvement in OPIS after incorprating TCM as a regularizer|
|:-------------------:|:---------------:|:-------------:|:--------------:|:----------------------------------------------------------------------:|
| ProxyAnchor + TCM  | ResNet50-512 | 65.7 (↑6.6) | 0.22 (↓0.76) | 77.60%  |

Reviewer comment (Experiments):
The use of the term "large-scale" in this paper may be misleading as the experiment datasets do not contain a sufficiently large number of samples to be accurately characterized as "large-scale." Typically, datasets with more than 10 million or 1 billion samples could be considered as large-scale.

Author response:
**A1** Thanks for the valuable comments. In our experiments, we evaluated our models on standard image retrieval benchmarks, aligning with practices in mainstream deep metric learning studies [1, 2, 3, 4]. We follow previous works [5, 6, 7] which used the word "large-scale" to describe the iNaturalist-2018 dataset, which is much bigger than traditional metric learning datasets such as CUB, Cars, and etc. However, we acknowledge that the scale of iNaturalist-2018 is limited compared to many recently-released datasets such as LAION-400M and LAION-5B, although these datasets are typically not used for metric learning. To enhance clarity and better reflect the scale of our experiments, we have incorporated the suggested revisions into the manuscript by taking out the term "large-scale experiments".

Reviewer comment (Novelty):
The threshold inconsistency problem, as described in this paper, is also referred to as the generalization problem and has been previously discussed in the deep metric learning (DML) literature [r1, r2]. In reference [r1], the authors proposed the adoption of a metric variance constraint (MVC) to enhance generalization ability, which is essentially a variance-based metric. Reference [r2] provided an in-depth discussion of the generalization problem in DML. It would be beneficial for this paper to incorporate discussions and comparisons with these existing works in the context of addressing the threshold inconsistency problem.

Author response:
**A2** We genuinely appreciate the reviewer for sharing the two referenced works with us. Through our examination, we have indeed identified noteworthy parallels between the generalization problem discussed in [1, 2] and our open-world setting. Particularly, one of conclusions in [2], stating that excessive feature compression actually hurts DML generalization, aligns with our observations in the ablation study for margin, as shown in Fig 6 of the paper, where overly restrictive margins lead to decreased accuracy. We have promptly incorporated references to these two works into our revised paper to further enrich the context and related research in the field. However, it's worth noting that these two works primarily focus on enhancing generalization to improve accuracy, particularly in scenarios characterized by train-test distribution or concept shifts. In contrast, our work specifically targets the crucial aspect of threshold consistency. Our proposed evaluation criteria, OPIS and epsilon-OPIS, are tailored to measure threshold consistency within the calibration range across different test distributions and classes. They are designed as orthogonal metrics to accuracy. 

Additionally, comparing TCM to the Margin Variance Control (MVC) proposed in CBML[1], we find commonalities between the two techniques. However, we would like to point out the three distinctions between TCM and MVC that makes TCM more suited for encouraging threshold consistency:

1. TCM employs two cosine margins to simultaneously regulate positive and negative pairs, whereas MVC is exclusively applied to negative pairs. In Appendix A.2.3, we demonstrate that relying solely on the negative term lags considerably behind the full TCM regularization in terms of Recall@1 (72.5 for negative term only vs. 84.4 for both positive and negative terms).
2. TCM applies penalties to hard sample pairs, while MVC minimizes variance for all negative pairs. In Appendix A.2.3, we consider a TCM variant without hard mining (referred to as TCM’ in Appendix A.2.3) by removing the indicator function in Eq. (5), where the regularization is applied to all positive and negative pairs. However, as shown in Appendix A.2.3, the omission of the indicator function yields a significant degradation in Recall@1 (74.1 for RS@K + TCM without indicator function vs. 84.8 for RS@K + TCM with indicator function, the higher the better), while producing a worse OPIS result (0.31 x 1e-3 for RS@K + TCM without indicator function vs 0.17 x 1e-3 for RS@K + TCM with indicator function, the lower the better). This result underscores the critical importance of hard sample mining.
3. TCM utilizes a global margin, whereas MVC calculates sample-specific margin, namely, target value for the similarity hyperplane as defined in [1].

Reviewer comment (Experiments):
If your method can use a single model to maintain the same threshold across multiple test sets, would it be meaningful, such as in this work[1].

Author response:
**Q2 If your method can use a single model to maintain the same threshold across multiple test sets?**

**A2** Yes, in our method, a single model can be used to test across multiple datasets and optimize for threshold consistency. We thank the reviewer for providing pointers on the dataset and setting in [1]. However, due to the presence of human faces in [Inshop dataset](https://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/InShopRetrieval.html) and [Met dataset](https://cmp.felk.cvut.cz/met/), to ensure compliance with data privacy protection laws and minimize the potential infringement risks regarding personal data and privacy, we could not directly use the UnED dataset [1]. Instead, we opt for an alternative approach by merging the four datasets in our paper, namely iNaturalist-2018, Stanford Online Product, CUB-200-2011, and Cars-196, following the dataset construction methodology described in [1]. For testing, we follow [1] and equalize the number of instances per class across different domains to prevent the evaluation being dominated by majority classes with a large number of images. The following table presents the results in terms of Recall@1 and OPIS. These results highlight the efficacy of TCM regularization in cross-domain settings.

Table 1. The performance of our proposed method on aforementioned dataset in Recall@1 and OPIS. The numbers in brackets represent the absolute improvement of models trained after incorporating TCM as a regularizer.

|        	         |   Backbone-embedding dimension |  Recall@1 ↑	  | OPIS x 1e-3 ↓	  | Relative improvement in OPIS after incorporating TCM as a regularizer|
|:----------------:|:-------------:|:-------------:|:---------------:|:----------------------------------------------------------------------:|
| SmoothAP + TCM	  | ResNet50-512	 | 75.2 (↑4.1)	  | 1.55 (↓0.64) 	  |                                29.20%	                                 |
| SmoothAP + TCM	  | ViT-B/16-512	 | 76.7 (↑5.4)	  | 2.06 (↓0.73) 	  |                                26.20%	                                 |

References for Response to reviewer 4pGE: 

[1] Ypsilantis, Nikolaos-Antonios, et al. "Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.