Paper:
Working hard to know your neighbor's margins: Local descriptor learning loss
We introduce a loss for metric learning, which is inspired by the Lowe's matching criterion for SIFT. We show that the proposed loss, that maximizes the distance between the closest positive and closest negative example in the batch, is better than complex regularization methods; it works well for both shallow and deep convolution network architectures. Applying the novel loss to the L2Net CNN architecture results in a compact descriptor named HardNet. It has the same dimensionality as SIFT (128) and shows state-of-art performance in wide baseline stereo, patch verification and instance retrieval benchmarks.
best of our knowledge, no work in local descriptor learning fully mimics such strategy as the learning objective.
Simonyan and Zisserman [20] proposed a simple filter plus pooling scheme learned with convex optimization to replace the hand-crafted filters and poolings in SIFT. Han et al. [14] proposed a twostage siamese architecture -for embedding and for two-patch similarity. The latter network improved matching performance, but prevented the use of fast approximate nearest neighbor algorithms like kd-tree [21]. Zagoruyko and Komodakis [15] have independently presented similar siamese-based method which explored different convolutional architectures. Simo-Serra et al [22] harnessed hardnegative mining with a relative shallow architecture that exploited pair-based similarity.
The three following papers have most closedly followed the classical SIFT matching scheme. Balntas et al [23] used a triplet margin loss and a triplet distance loss, with random sampling of the patch triplets. They show the superiority of the triplet-based architecture over a pair based. Although, unlike SIFT matching or our work, they sampled negatives randomly. Choy et al [7] calculate the distance matrix for mining positive as well as negative examples, followed by pairwise contrastive loss.
Tian et al [24] use n matching pairs in batch for generating n 2 − n negative samples and require that the distance to the ground truth matchings is minimum in each row and column. No other constraint on the distance or distance ratio is enforced. Instead, they propose a penalty for the correlation of the descriptor dimensions and adopt deep supervision [25] by using intermediate feature maps for matching. Given the state-of-art performance, we have adopted the L2Net [24] architecture as base for our descriptor. We show that it is possible to learn even more powerful descriptor with significantly simpler learning objective without need of the two auxiliary loss terms.

Review:
The paper presents a variant of patch descriptor learning using neural networks and a triplet loss. While many similar approaches exist, the particular variant proposed here appears to have better results in a large number of benchmarks.
Still, I am really unsure about the technical contribution as the paper is not very clear on this point. The approach appears to be very similar to others such as Balntas 16 that also used triplet losses and deep nets to learn patch descriptors. It seems that the main difference is to consider in the loss hard negative examples. However, Balntas 16 *also* uses hard negatives in the triplets (not merely random samples as argued here on line 46). So what is the difference that changes the empirical results so much?
The paper generally needs far more polish. Experiments should carefully describe the difference between different approaches (e.g. descriptor dimensionally, neural network architecture, training set, and anything else that could make a difference). Then, the main reason for the observed empirical boost should be unequivocally identified through empirical assessment.  For example, if the claim is that the key is to pool hard negatives, a carefully ablation study comparing this to alternative tripled-formation strategies should be included. Part of such a study may be included in the HardTFeat vs TFeat experiment of Fig. 7, but I would liket to check with the authors that they ran both methods using the same code base and only changing the feature sampling strategy. If more than one thing changes at a time, is difficult to reach a conclusion.
If the authors could clearly identify an interesting reason explaining their empirical boost (e.g. they may conclude that the key is a new and better way of doing hard negative mining compared to what has been done so far), then the paper could be interesting enough for acceptance. If, however, the boost is due to other differences such as more tuning or a tweaked architecture, then there would be much less of a reason to accept this into NIPS.
Rebuttal: the authors provided an informative rebuttal.
One of my key question was about the difference with Blantas 16. The authors argue that Blantas 16 does *not* do hard negative mining. On re-reading Blantas 16, this is partially true: their point is that a simplified form of hard negative mining (called in-triplet hard negatives) is just as effective and in fact superior to hard negative mining, tested in e.g. "Discriminative learning of deep convolutional feature point descriptors".
Given the new experiments in the rebuttal, it seems that the main idea here is a new variant of negative mining, within each batch rather than in the whole datasets as done in "Discriminative learning of deep convolutional feature point descriptors", which seems to be similar to the HNM method described in the rebuttal.
Hence to me the main message of the paper is that, while HB and HNM are very simple and very similar approaches, HB is in practice far better than HNM.  The authors should modify the paper to include a careful experimental analysis of this point, extending and consolidating the new experiments in the rebuttal.
With this and other promised improvements, the paper would be good enough for acceptance in my opinion. However, the modifications required from the submitted version are fairly large.