Paper: Mathematical Justification of Hard Negative Mining via Isometric Approximation Theorem (ICLR-2024)

Reviewer comment (Presentation):
The illustrative examples are not clear. Figure 2 is clear on the examples, but Figure 3 is not clear.

Author response:
Thank you for the feedback. We have updated the figure to remove extraneous lines. 
The difference in the distance metric should now be more evident.
Further, we improved the clarity of the object marked by $d_\textrm{iso}(f_\theta,\mathcal F_{TS})$ by removing the clutter of lines surrounding the text in the figure.
We hope this addresses the illustration in Fig. 3.

Reviewer comment (Presentation):
The line in Figure 3 overlaps too much; it would be good to redraw it.

Author response:
Thank you for the feedback. We have updated the figure to remove extraneous lines. 
The difference in the distance metric should now be more evident.
Further, we improved the clarity of the object marked by $d_\textrm{iso}(f_\theta,\mathcal F_{TS})$ by removing the clutter of lines surrounding the text in the figure.
We hope this addresses the illustration in Fig. 3.

Reviewer comment (Writing):
It is not clear what diso(fθ,FTS) represents in Figure 3.

Author response:
Thank you for the feedback. We have updated the figure to remove extraneous lines. 
The difference in the distance metric should now be more evident.
Further, we improved the clarity of the object marked by $d_\textrm{iso}(f_\theta,\mathcal F_{TS})$ by removing the clutter of lines surrounding the text in the figure.
We hope this addresses the illustration in Fig. 3.

Reviewer comment (Presentation):
Figure 4 can be extended to multiple views to illustrate the progress of the collapse as shown in [1], and if the diso(fθ,FTS) value can be added in the visualization, it will be valued.

Author response:
Thank you for the suggestion and for the reference. We have modified our Fig. 4 to match the style in [1]. Additionally, we compute $d_\textrm{haus}(f_\theta,\mathcal F_{TS})$ as $d_\textrm{haus}(f_\theta, f^*)$ using the approximated $f^*$ function and have marked the function value in Fig. 4 to provide additional context.

Reviewer comment (Writing):
Minor issue in the typos: section 1 "CUBwah et al. (2011))".

Author response:
Thank you for pointing this out. We have fixed the typo in the revised version.

Reviewer comment (Theory):
Triplet loss is rarely used in Self-supervised Learning with a random initialization network; maybe one of the reasons is also collapse. Can your theory explain the problem? Is the DML problem similar to the SSL problem at the embedding level?

Author response:
The reviewer points to an interesting connection between self-supervised learning (SSL) and deep metric learning (DML) at the embedding level. While we acknowledge a similarity in how SSL and DML use contrastive loss, there exists a difference in the structures of their respective datasets. 
We believe exploring and presenting the extension of our theory to SSL will likely need new terminology and careful consideration in the proofs, and thus we keep this for future work.

Reviewer comment (Experiments):
The paper primarily focuses on hard negative mining as a solution to network collapse but does not thoroughly compare its performance with alternative methods or strategies. A comprehensive comparison with other approaches would provide a more comprehensive understanding of the effectiveness of hard negative mining and its relative advantages or disadvantages compared to other techniques. Without such comparisons, it is difficult to assess the competitiveness of the proposed method.

Author response:
Thank you for the question. As we state in the fifth paragraph of section 1, this work aims to explain why network collapse happens by using the theory of isometric approximation to draw an equivalence between the triplet loss with hard negative mining and Hausdorff-like distance metric. We would like to clarify that our paper does not propose a new method, and the performance of the triplet loss with hard negative mining has already been recorded in other papers. 

Prior works such as [1] point to a network collapse in the context of hard negative mining while the authors in [2] show the absence of network collapse with hard negative mining for a similar dataset. To address these seemingly contradictory results, our work offers a systematic explanation for network collapse, its causes, and how to avoid it in the context of triplet loss with hard negative mining. 

[1] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 815–823, 2015.

[2] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person reidentification.
arXiv preprint arXiv:1703.07737, 2017.

Reviewer comment (Evaluation):
While the paper presents experimental results on multiple datasets and network architectures, it does not provide precise quantitative measurements of the performance improvements achieved by preventing network collapse through hard negative mining. Without specific metrics, such as accuracy or loss values, it is challenging to assess the magnitude of the improvement or compare it with alternative approaches. Including quantitative evaluations would enhance the rigor and credibility of the proposed method.

Author response:
Continuing our response from the previous section, the focus of this work is to provide a mathematical justification for network collapse and not present a new method to perform hard negative mining. Consequently, our experiments are directed to corroborate the likelihood of network collapse when the batch size is large or when the embedding dimension is small and not to provide precise quantitative measurements for hard negative mining.

Reviewer comment (Evaluation):
The absence of mining time analysis and discussion on the time complexity limits the applicability of the proposed method to real-time or time-sensitive applications. Real-time systems often require fast and efficient processing, and the computational cost of hard negative mining can be a critical factor in determining the feasibility of the approach. Considering the practical implications and performance trade-offs in real-time scenarios would enhance the relevance and applicability of the proposed method.

Author response:
We feel that the mining time analysis is not relevant to this work. While it is true that discussion on time complexity is an essential factor when proposing new triplet sampling methods, our work primarily investigates the phenomenon of network collapse on an existing method, i.e., hard negative mining.

Reviewer comment (Experiments):
The paper primarily focuses on the theoretical framework and experimental results of hard negative mining in preventing network collapse. However, the experiments are conducted on a limited set of network architectures, and it does not include widely used backbone architectures such as ResNet-50 or Inception-BN. These mainstream architectures are commonly employed in deep metric learning and computer vision tasks. The lack of evaluation on such architectures limits the generalizability and applicability of the proposed method to real-world scenarios.

Author response:
While our initial submission reports findings from three different network architectures (ResNet-18, GoogLeNet, and a 2-layer convnet), we have now conducted experiments with ResNet-50 as the backbone, as per the reviewer's suggestion. The results from this new experiment corroborate our claims that network collapse is more likely when the batch size is large or when the embedding dimension is small. We have now included the details of these results in Appendix E and highlighted the changes in yellow. We thank the reviewer for the suggestion, since including these additional results has surely strengthened our manuscript.

Reviewer comment (Experiments):
It would be beneficial to see a more thorough analysis of various triplet loss variants, particularly with regard to different triplet sampling methods. This in-depth examination is essential for a comprehensive understanding of the effectiveness of triplet loss in different settings.

Author response:
While our work focuses on hard negative mining on the triplet loss, it is worth noting that the theory of isometric approximation as used in this manuscript is independent of the triplet loss. Consequently, it can be applied to any system utilizing the Euclidean metric, for instance the pairwise contrastive loss ($\mathcal{L} = |x - x^+| - [\alpha - |x-x^-|]_+$) or the margin loss ($\mathcal{L}=\left[|x-x^+| + \alpha - \beta\right]\_+ + [-|x-x^-| + \alpha + \beta]\_+$).

On the other hand, the fundamental reason for the choice of hard negative mining is that it resembles a $\max_{x,x^+,x^-} \mathcal L(x, x^+, x^-)$ type of operation, which via the isometric approximation theorem, can be directly connected with our defined Hausdorff-like metric $d_\textrm{haus}(f_1, f_2) = \max_x d(f_1(x), X_{f_2}^i)$ or $\max_x d(f_2(x), X_{f_1}^i)$, because they both use max operations.
To rigorously examine other triplet sampling methods, we would likely need a careful exploration of functions that naturally connect with those other sampling methods, and thus we keep this for future work.

Reviewer comment (Experiments):
The experimental evaluations in this paper are notably limited, making it challenging to substantiate the claims put forth. For instance, the use of a very large batch size with semi-hard triplets in FaceNet raises questions that require further clarification or justification.

Author response:
Thank you for the feedback. Quoting from the FaceNet [1] paper, ``Selecting the hardest negatives can in practice lead to bad local minima early on in training, specifically it can result in a collapsed model (i.e. f(x) = 0).'' FaceNet [1] work uses this fact as a motivating example to propose semi-hard triplets to work with very large batch sizes. 
Our work supports and justifies their observations that very large batch sizes in combination with hard negative mining adversely affects the network. 

In this work, we provide rigorous mathematical analysis to show how large batch sizes and small embedding dimensions adversely affect hard negative mining. Additionally, we present experimental verification with the person re-identification dataset (Market-1501) reconcile the findings of FaceNet [1], where a batch size in the order of thousands led to network collapse, and [2] where a batch size of N = 72 showed no network collapse. We
further support our predictions via experiments spanning three additional datasets (SOP, CARS, and CUB200) and three different network
architectures (ResNet-18, GoogLeNet, and a 2-layer
convnet). We are happy to conduct additional experiments on any specific datasets or network architectures to expand the breadth of our verification, as per the reviewer's additional requests.

Furthermore, our recent experiments with ResNet-50 as the backbone resonate with the findings from the previous three network architectures. We hope the experimental evaluation is now adequate to substantiate the claims put forth in the manuscript. 
We are happy to conduct supplementary experiments on specific datasets or network architectures to expand the breadth of our verification per the reviewer's additional requests. 

[1] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 815–823, 2015.

[2] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person reidentification.
arXiv preprint arXiv:1703.07737, 2017.