Paper: Mean Field Theory in Deep Metric Learning (ICLR-2024)

Reviewer comment (Theory):
Deep metric learning involves a range of loss functions, and it is unclear whether the mean field theory can be applied to other loss functions commonly used in this context.

Author response:
Thank you for your insightful comments and the opportunity to discuss the extensions of our work. We are grateful for the chance to elaborate on how Mean-Field Theory (MFT) can further enhance deep metric learning.

To address your question about extending MFT to other common loss functions in deep metric learning, we have added theoretical insights into applying MFT to losses with anchors in Section A.4 in the appendix. 

Unlike the proxy-based method, which replaces positive and negative samples with proxies, the mean field approach results in two distinct terms. In one term, positive and negative samples are replaced by mean fields, similar to the proxy-based approach. However, in the other term, anchors are replaced by mean fields while the positive and negative samples remain. This term is analogous to the ProxyAnchor loss, which was introduced heuristically.

We hope this response adequately addresses your query and provides further clarity on our work. We are thankful for your engagement with our research and welcome any additional comments or questions you may have.

Reviewer comment (Theory):
It would be valuable for the authors to specify under what conditions and contexts the mean field theory is applicable and offer practical guidance for its implementation in other scenarios.

Author response:
We have carefully considered your comments and have made the following significant updates to address your concerns:

**Application to NCA Loss**

In response to your query on applying Mean-Field Theory (MFT) to other common loss functions in deep metric learning, we have demonstrated the application of MFT to the NCA loss and derived Mean Field NCA (MFNCA) loss (Eq. (21) of our revised manuscript). We think this extension shows the versatility of MFT in the context of deep metric learning and provides a practical guidance for its implementation in various scenarios.

**Benchmarking MFNCA loss** 

To empirically validate the effectiveness of the MFNCA loss, we conducted an MLRC benchmark on the CUB dataset. The results, as detailed in Table 15 of Section B.7, show a consistent improvement in accuracy metrics. This experiment suggests that the additional term derived from the expansions with respect to positive and negative samples contributes to a more accurate approximation of the original loss functions, indicating the utility of MFT in losses with anchors.

We believe these updates comprehensively address your concerns and further strengthen our manuscript. We remain open to any additional suggestions or queries you might have and look forward to your feedback.

Reviewer comment (Theory):
How can the mean field theory be extended to encompass other commonly used loss functions in deep metric learning?

Author response:
Thank you for your insightful comments and the opportunity to discuss the extensions of our work. We are grateful for the chance to elaborate on how Mean-Field Theory (MFT) can further enhance deep metric learning.

To address your question about extending MFT to other common loss functions in deep metric learning, we have added theoretical insights into applying MFT to losses with anchors in Section A.4 in the appendix. 

Unlike the proxy-based method, which replaces positive and negative samples with proxies, the mean field approach results in two distinct terms. In one term, positive and negative samples are replaced by mean fields, similar to the proxy-based approach. However, in the other term, anchors are replaced by mean fields while the positive and negative samples remain. This term is analogous to the ProxyAnchor loss, which was introduced heuristically.

We hope this response adequately addresses your query and provides further clarity on our work. We are thankful for your engagement with our research and welcome any additional comments or questions you may have.

Reviewer comment (Evaluation):
Part of the motivation for the approximation (not including its statistical physics analogy) was the reduction in runtime complexity pair-based loss function in metric learning. However, there is no runtime values reported in the paper.

Author response:
**Runtime comparison**

Thank you for highlighting the importance of runtime comparison. We have now included a new figure in our revised manuscript (Fig. 4 in Appendix), which compares the runtimes of the mean field losses against their regular counterparts. As indicated in the figure, the mean field variants not only offer a reduction in computational complexity due to the mean field approximation but also demonstrate faster runtime performance. This improvement aligns with our theoretical expectations and supports the practical applicability of our approach.

Reviewer comment (Theory):
One of the major interesting components of the paper is that mean field approximation performs better than its non-mean field counterparts. Is there a good hypothesis for why this may be the case? Have you found a good characterization of when one out performs the other? Additional insight here would be great since the paper claims in the empirical section that the mean field losses provide better "training complexity but also results in better embeddings".

Author response:
**Performance of mean field approximations**

We appreciate your interest in the superior performance of mean field approximations. We hypothesize that the mean field approximation reduces the noise introduced in pairwise comparisons by considering the class means, thus simplifying the optimization process. This could lead to more robust and efficient training, as reflected in our empirical results. As you suggested, we added a sentence 
`This is perhaps because the mean field losses can reduce the noise introduced in pairwise comparisons by the mean fields.` in Sec. 4.3.

R2: **Additional terms in MFT and their impact**

As detailed in Section A.4 of the latest manuscript version, Applying MFT to anchor loss introduces terms that are not derived through the proxy-based method. These terms emerge from the expansion with respect to positive/negative samples and are typically neglected in proxy-based approaches.
By incorporating these terms, MFT yields a closer approximation to the original loss function, particularly near the optimal solution. This finer approximation is likely a significant factor contributing to the enhanced performance observed in our experiments, as it provides a more accurate representation of the loss landscape.

To empirically validate this point, we conducted a benchmark test using the MLRC protocol for our MFNCA loss (the mean field counterpart of ProxyNCA, derived in Eq. (21)) on the CUB dataset.
The results, which we discuss in Section B.7 of our manuscript, demonstrate consistent improvements in accuracy metrics MFNCA compared to ProxyNCA. This empirical evidence further supports our hypothesis that the superior approximation capabilities of MFT lead to improved performance.

Reviewer comment (Theory):
Furthermore, what do that authors mean by "... the above discussion implies the mean field theory is independent of the concept of anchors ..."?

Author response:
**Mean field theory and concept of anchors**

Your question brings to light a crucial aspect of our work. Our approach differs fundamentally from proxy-based methods, which assume a loss depends on the triad of anchor, positive, and negative elements. In contrast, our mean field theory (MFT) approach can be applied to any loss function through Taylor expansion and does not assume a special form of the loss function. This distinct methodology allows for broader applicability and simplicity.
To clarify this point, we modified the sentence as follows:
`Furthermore, in contrast to the proxy-based method, which can be applied only to a pair-based loss with an anchor, the mean field theory is applicable to wider types of pair-based loss functions as it is based on the Taylor expansions.`

Reviewer comment (Theory):
The authors hide the actual derivation in the appendix, so they do not detail enough their technical approach. On the face of it, the modified loss function could have been suggested just through statistical intuition, not derived from an energy approximation (Hubbard-Stratonovich, saddle-point approx, etc.), so it is a pity the authors don't sketch their methodology in the main text.

Author response:
We understand your concern regarding the detailed technical derivation. However, due to page limitations, it is challenging to include the full derivation in the main text without sacrificing other critical content. We have, therefore, made an effort to enhance the clarity and accessibility of the appendix, ensuring that readers can easily follow the derivation process.

Reviewer comment (Experiments):
It is hinted that the method can be more efficient (e.g. due to the lack of pair sampling or anchor points choice), but it is not reported if the training time is superior to the other methods or if the optimisation over `M`-s causes a substantial overhead (which may be justifiable with the improved results).

Author response:
We are glad you brought up the issue of computational efficiency. The equation (Eq. 3) related to optimal M does not require explicit consideration during training at T=0, and thus, our losses have no potential overhead you concerned. To clarify this point, in our revised manuscript, we have presented a pseudo code and a learning curve in runtime in the SOP dataset in Figure 4 in Appendix. This demonstrates that our methods not only converge faster in terms of epochs but also show superior performance in runtime efficiency compared to existing methods.

Reviewer comment (Experiments):
Is it always favourable to optimize the means `M` and the parameters `theta` together? Maybe alternating between optimisation steps would be superior in numerical terms.

Author response:
We agree that the optimization of M and $\theta$ is a crucial aspect of our approach.
For T=0, simultaneous optimization is feasible and effective since the standard stochastic gradient descent is applicable.
However, for T>0, the optimization becomes more complex, and an EM-like algorithm becomes necessary.
In Section A.5, we have sketched the relationship between EM (Expectation-Maximization) and MFT (Mean-Field Theory) at T>0.
based on a similarity between the partition function in our method and the log-likelihood in the EM algorithm.

Reviewer comment (Evaluation):
Are the new methods faster or slower than other methods?

Author response:
We are glad you brought up the issue of computational efficiency. The equation (Eq. 3) related to optimal M does not require explicit consideration during training at T=0, and thus, our losses have no potential overhead you concerned. To clarify this point, in our revised manuscript, we have presented a pseudo code and a learning curve in runtime in the SOP dataset in Figure 4 in Appendix. This demonstrates that our methods not only converge faster in terms of epochs but also show superior performance in runtime efficiency compared to existing methods.

Reviewer comment (Novelty):
The major concern is that the paper lacks a sufficient level of novelty and performance improvement.

Author response:
Thank you for your thoughtful and constructive feedback on our manuscript. We appreciate the opportunity to clarify and expand upon the points you raised.

**Comparison with central loss**

We acknowledge your observation regarding the similarity of our approach to central loss. However, there is a fundamental difference in methodology and computation. In central loss, there is a necessity to continually update class centers during training. In contrast, our approach rooted in mean field theory demonstrates that the optimal mean field equates to the class center. This is achieved without the extra computational cost associated with central loss. The theoretical underpinning of our method offers a more efficient alternative, emphasizing the novelty of our approach. This point is clarified in the revised manuscripts (see the last paragraph in Sec. 3.2).



**Statistical significance through t-tests**
Upon your suggestion, we conducted t-tests to validate the statistical significance of our results. The t-tests were performed comparing our methods (MFCont and MFCWMS) against ArcFace and ProxyAnchor, as shown in Tables 6 and 7 in the revised manuscript.
The tables highlight that our MFCWMS loss demonstrates statistically significant improvements over ArcFace in all situations except 512D in the Cars dataset and ProxyAnchor except 128D and 512D in the Cars dataset. Additionally, our simpler loss function, MFCont. loss, although slightly less competitive than MFCWMS loss, still holds up well against these benchmarks in CUB and SOP.

We believe these additional analyses and clarifications address your concerns regarding both the novelty and the performance improvement of our proposed methods. Our approach not only stands out in terms of theoretical efficiency over central loss but also demonstrates practical efficacy through statistically significant performance improvements.

We hope that these clarifications and additional analyses sufficiently address your concerns, and we are open to any further suggestions or queries you might have.

Reviewer comment (Novelty):
They mainly explore mean field theory into metric learning, and such metric is close to central loss [R1]. [R1]. Wen, Yandong, et al. "A discriminative feature learning approach for deep face recognition." Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. Springer International Publishing, 2016.

Author response:
Thank you for your valuable feedback on our manuscript. We here clarify our previous response and notify you of the updates in our manuscript.

**Distinction from central loss**

We appreciate your insights regarding central loss. As detailed in our previous response, our Mean Field Theory (MFT) approach differs significantly from central loss in two key aspects:

Difference in Computational Method: Central loss requires dynamic updating of class centers for regularization. In contrast, MFT achieves optimal mean fields that automatically align with class centers, eliminating the need for continual updates. This fundamental computational difference sets MFT apart from central loss.

Difference in Purpose of Method: While central loss aims to enhance the performance of classification-based losses, MFT's focus is on deriving classification-based losses from pair-based ones. Our application of MFT has led to the discovery of more efficient loss functions like MFCWMS loss.

**Novelty of our method**

Our method's applicability to a wider range of pair-based loss functions, including those without anchors, is a substantial novelty. This flexibility allows for the discovery of more efficient loss functions.

Moreover, in the revised manuscript, we have provided a detailed comparison between the proxy-based method and MFT. Theoretically, we confirmed that a loss derived from MFT more accurately approximates the original loss function, especially near the optimal solution (see Section A.4). Empirically, the MLRC benchmark results for MFNCA loss on the CUB dataset demonstrate consistent improvements in accuracy metrics (see Table 15 in Section B.7). These findings underscore the enhanced performance achieved through a more accurate approximation of the original loss functions.

We believe these revisions and additions effectively address the concerns regarding novelty and comparison with both central loss and traditional proxy-based methods.

We are grateful for your feedback, which has been instrumental in enhancing our manuscript. We remain open to any further suggestions or comments.

Reviewer comment (Evaluation):
The performance is not significant. In Table I, compared with ArcFace and ProxyAnch, it is hard to justify the significant improvement. It is essential to do t-test.

Author response:
**Statistical significance through t-tests**
Upon your suggestion, we conducted t-tests to validate the statistical significance of our results. The t-tests were performed comparing our methods (MFCont and MFCWMS) against ArcFace and ProxyAnchor, as shown in Tables 6 and 7 in the revised manuscript.
The tables highlight that our MFCWMS loss demonstrates statistically significant improvements over ArcFace in all situations except 512D in the Cars dataset and ProxyAnchor except 128D and 512D in the Cars dataset. Additionally, our simpler loss function, MFCont. loss, although slightly less competitive than MFCWMS loss, still holds up well against these benchmarks in CUB and SOP.

Reviewer comment (Novelty):
The clarification of model novelty.

Author response:
Thank you for your valuable feedback on our manuscript. We here clarify our previous response and notify you of the updates in our manuscript.

**Distinction from central loss**

We appreciate your insights regarding central loss. As detailed in our previous response, our Mean Field Theory (MFT) approach differs significantly from central loss in two key aspects:

Difference in Computational Method: Central loss requires dynamic updating of class centers for regularization. In contrast, MFT achieves optimal mean fields that automatically align with class centers, eliminating the need for continual updates. This fundamental computational difference sets MFT apart from central loss.

Difference in Purpose of Method: While central loss aims to enhance the performance of classification-based losses, MFT's focus is on deriving classification-based losses from pair-based ones. Our application of MFT has led to the discovery of more efficient loss functions like MFCWMS loss.

**Novelty of our method**

Our method's applicability to a wider range of pair-based loss functions, including those without anchors, is a substantial novelty. This flexibility allows for the discovery of more efficient loss functions.

Moreover, in the revised manuscript, we have provided a detailed comparison between the proxy-based method and MFT. Theoretically, we confirmed that a loss derived from MFT more accurately approximates the original loss function, especially near the optimal solution (see Section A.4). Empirically, the MLRC benchmark results for MFNCA loss on the CUB dataset demonstrate consistent improvements in accuracy metrics (see Table 15 in Section B.7). These findings underscore the enhanced performance achieved through a more accurate approximation of the original loss functions.

We believe these revisions and additions effectively address the concerns regarding novelty and comparison with both central loss and traditional proxy-based methods.

We are grateful for your feedback, which has been instrumental in enhancing our manuscript. We remain open to any further suggestions or comments.

Reviewer comment (Evaluation):
The performance improvement.

Author response:
**Statistical significance through t-tests**
Upon your suggestion, we conducted t-tests to validate the statistical significance of our results. The t-tests were performed comparing our methods (MFCont and MFCWMS) against ArcFace and ProxyAnchor, as shown in Tables 6 and 7 in the revised manuscript.
The tables highlight that our MFCWMS loss demonstrates statistically significant improvements over ArcFace in all situations except 512D in the Cars dataset and ProxyAnchor except 128D and 512D in the Cars dataset. Additionally, our simpler loss function, MFCont. loss, although slightly less competitive than MFCWMS loss, still holds up well against these benchmarks in CUB and SOP.