  Pair-wise loss functions have been extensively studied and shown to
continuously improve the performance of deep metric learning (DML). However,
they are primarily designed with intuition based on simple toy examples, and
experimentally identifying the truly effective design is difficult in
complicated, real-world cases. In this paper, we provide a new methodology for
systematically studying weighting strategies of various pair-wise loss
functions, and rethink pair weighting with an embedding memory. We delve into
the weighting mechanisms by decomposing the pair-wise functions, and study
positive and negative weights separately using direct weight assignment. This
allows us to study various weighting functions deeply and systematically via
weight curves, and identify a number of meaningful, comprehensive and
insightful facts, which come up with our key observation on memory-based DML:
it is critical to mine hard negatives and discard easy negatives which are less
informative and redundant, but weighting on positive pairs is not helpful. This
results in an efficient but surprisingly simple rule to design the weighting
scheme, making it significantly different from existing mini-batch based
methods which design various sophisticated loss functions to weight pairs
carefully. Finally, we conduct extensive experiments on three large-scale
visual retrieval benchmarks, and demonstrate the superiority of memory-based
DML over recent mini-batch based approaches, by using a simple contrastive loss
with momentum-updated memory.
