Paper:
The Triplet Loss approach to Distance Metric Learning is defined by the strategy to select triplets and the loss function through which those triplets are optimized. During optimization, two especially important cases are easy positive and hard negative mining which consider, the closest example of the same and different classes. We characterize how triplets behave based during optimization as a function of these similarities, and highlight that these important cases have technical problems where standard gradient descent behaves poorly, pulling the negative example closer and/or pushing the positive example farther away. We derive an updated loss function that fixes these problems and shows improvements to the state of the art for CUB, CAR, SOP, In-Shop Clothes datasets.

Review:
Summary: The authors propose a new loss function as well as an adjoining visualization for improved performance of hard negative / easy positive mining for deep triplet metric learning. The authors note that under the NCA loss, if one selects an easy positive / hard negative and computes the gradient with respect to this example, this can lead to the negative example also being pulled closer to the anchor which is undesired. Similar phenomena can also be observed for easy positive / semi-hard negative mining as well. Motivated by this, the authors begin by designing a visualization to make this issue with NCA loss more apparent. Then they design what they refer to as an “entanglement factor” to quantify this issue more precisely. Using the desired dynamics of the gradients for the easy positive / hard negative mining and integrate to form what they refer to as the “second order loss.” Using this loss, they compare against the standard NCA loss on several datasets, showing modest performance gains. They also compare against a variety of other deep triplet embedding frameworks and show competitive results.
 
Comments, questions, and concerns:
- Overall the paper is well written and clear.
- The authors do a good job selecting reasonable baselines upon which to compare their method.
- It is worth noting, though not stated in the paper, that this trade-off between pushing away hard negatives and pulling in easy positives has actually previously been considered in the linear metric learning setting, though the terminology was different then. See the “Large Margin Nearest Neighbors” algorithm by Weinberger and Saul, for example which trains on “target neighbors” and “imposters.”
- Is projecting back to the hypersphere actually an issue? If the modelling assumption is truly that the learned representation should lie on the hypersphere, then this amounts to projected gradient descent and is a standard tool. In fact, projecting back to the sphere is correct under the model. If magnitude is truly important, then enforcing that the data be on the sphere is incorrect.
- The claim of a “systematic characterization for triplet selection strategies…” seems overly broad. It seems more correct to state this with respect to NCA loss specifically.
- NCA loss is just logistic loss penalizing the difference of similarities. The second order loss is the logistic loss penalizing (1/2 times) the squared difference of similarities, hence the cross term. Might be good to state this explicitly.
- The figures throughout are difficult to read due to the small font size.
- The y-axis in figure 4 is very different for each plot, which makes the effect seem much larger than it is.
- There are no error bars on plots or a discussion about how much variance a practitioner should expect running these experiments. If every single training run ever takes the same amount of time, who cares, but otherwise, we have one sample from each distribution and it’s hard to infer from that.
- Additionally, in Table 1, since there are no confidence regions, it is difficult to know if when the algorithm does perform well if these differences are significant. Especially important since on several datasets, the model does not perform as well as others.