Paper:
Local Similarity-Aware Deep Feature Embedding
Existing deep embedding methods in vision tasks are capable of learning a compact Euclidean space from images, where Euclidean distances correspond to a similarity metric. To make learning more effective and efficient, hard sample mining is usually employed, with samples identified through computing the Euclidean feature distance. However, the global Euclidean distance cannot faithfully characterize the true feature similarity in a complex visual feature space, where the intraclass distance in a high-density region may be larger than the interclass distance in low-density regions. In this paper, we introduce a Position-Dependent Deep Metric (PDDM) unit, which is capable of learning a similarity metric adaptive to local feature structure. The metric can be used to select genuinely hard samples in a local neighborhood to guide the deep embedding learning in an online and robust manner. The new layer is appealing in that it is pluggable to any convolutional networks and is trained end-to-end. Our local similarity-aware feature embedding not only demonstrates faster convergence and boosted performance on two complex image retrieval datasets, its large margin nature also leads to superior generalization results under the large and open set scenarios of transfer learning and zero-shot learning on ImageNet 2010 and ImageNet-10K datasets.

Review:
The authors propose to capture local feature similarity metric using PDDM, to tackle the issues brought by global distance metrics. They also introduce the large-margin double-header hinge loss to jointly optimize the similarity metric learning. They show good performance on image retrieval and transfer learning tasks to demonstrate the efficacy of their method.
The paper is well written with clear logic and well-designed figures, and shows promising results in both image retrieval and transfer learning tasks. The authors demonstrate clearly the effectiveness of combining distance metric with local similarity achieved by PDDM mechanism. The contribution of PDDM mechanism shows a novel local similarity metric which combines feature difference and absolute position information from [35], and presents surprising fast training convergence and better performance. The contribution of two double-header hinge losses in Eq (4) which combine local similarity metric and distance metric on the hard quadruplet has similar idea with [27], but uses less samples and has low computational cost and faster convergence.  Questions for authors: 1) For better understand and self-inclusive of the paper, there should be more formalized description of the comparative methods (i.e. contrastive embedding, triplet embedding and lifted structured embedding) to emphasize the improvement and advantage of the proposed method besides the comparison in Figure 3. 2) The authors should give more analysis or demonstration to explain the key reason of the effectiveness brought by the concatenation of feature difference vector and feature mean vector in the PDDM unit, which could be beneficial and instructive to readers. 3) The authors mention that the whole network, which contains 4 identical CNNs and PDDM units, is trained end-to-end. However, the hard quadruplet mining during a forward pass also needs 2 CNNs and PDDM units. The authors should add more information on the particular forward pass for easy understanding of the end-to-end architecture. 4) I want to confirm that you choose ONLY ONE hard quadruplet ($\hat{i}$, $\hat{j}$, $\hat{k}$, $\hat{l}$) for training per mini-batch (64 samples). If so, you have very few samples to fine-tune the CNN part and train the fully-connected layers of PDDM unit per epoch. I wonder if the samples are enough for training the network parameters on CUB-200-2011 dataset (5,864 training images). Please include more details about this.  There are a few typos which need to be amended: 1) Line 47: learn an local-adaptive -> learn a local-adaptive 2) Line 103: into an feature space -> into a feature space 3) Eq (2). $u_1$ and $v_1$ should be $u'$ and $v'$