Paper:
We present a radial basis function solver for convolutional neural networks that can be directly applied to both distance metric learning and classification problems. Our method treats all training features from a deep neural network as radial basis function centres and computes loss by summing the influence of a feature's nearby centres in the embedding space. Having a radial basis function centred on each training feature is made scalable by treating it as an approximate nearest neighbour search problem. End-to-end learning of the network and solver is carried out, mapping high dimensional features into clusters of the same class. This results in a well formed embedding space, where semantically related instances are likely to be located near one another, regardless of whether or not the network was trained on those classes. The same loss function is used for both the metric learning and classification problems. We show that our radial basis function solver outperforms state-of-the-art embedding approaches on the Stanford Cars196 and CUB-200-2011 datasets. Additionally, we show that when used as a classifier, our method outperforms a conventional softmax classifier on the CUB-200-2011, Stanford Cars196, Oxford 102 Flowers and Leafsnap fine-grained classification datasets.

Review:
The authors propose a loss that is based on a RBF loss for metric learning and incorporates additional per exemplar weights in the index for classification. Significant improvements over softmax are shown on several datasets.
IMHO, this could be a worthwhile paper, but the framing of the paper into existing literature is lacking and thus it appears as if the authors are re-inventing the wheel (NCA loss) under a different name (RBF solver).
The specific problems are:
- The authors completely miss the connection to NCA loss (https://papers.nips.cc/paper/2566-neighbourhood-components-analysis.pdf) and thus appear to be re-inventing the wheel.
  - The proposed metric learning scenario is exactly as proposed in the NCA loss works, while the classification approach adds an interesting twist by learning per exemplar weights. I haven't encountered this before and it could make an interesting proposal. Of course the benefit of this should be evaluated in ablation studies( Tab 3 shows one experiment with marginal improvements).
- The authors' use of 'solver' seems uncommon and confusing. What is proposed is a loss in addition to building a weighted index in the case of classification.
- In the metric learning comparison with softmax (end of page 9) the authors mentions that a Gaussian standard deviation for softmax is learned. It appears as if the authors use the softmax logits as embedding whereas the more common approach is to use the bottleneck layer. This is also indicated by the discussion at the end of page 10 where the authors mention that softmax is restricted to axis aligned embeddings. All softmax metric learning experiments should be carried out on appropriately sized bottleneck layers.
- Some of the motivations of what the various methods learn seem flawed, e.g. triplet loss CAN learn multiple modes per class and there is nothing in the Softmax loss that encourages the classes to fill a large region of the space.
- Why don't the authors compare on ImageNet?
Some positive points:
- The authors mention in Sec 3.3 that updating the RBF centres is not required. This is a crucial point that should be made a centerpiece of this work, as there are many metric learning works that struggle with this. Additional experiments that can investigate this point would greatly contribute to a well rounded paper.
- The numbers reported in Tab 1 show very significant improvements
If the paper was re-framed and builds on top of the already existing NCA loss, there could be valuable contributions in this paper. The experimental comparisons are lacking in some respect, as the comparison with Softmax as a metric learning method seems uncommon, i.e. using the logits instead of the bottleneck layer. I encourage the authors to extend the paper and flesh out some of the experiments and then submit it again.