  Deep Metric Learning (DML), a widely-used technique, involves learning a
distance metric between pairs of samples. DML uses deep neural architectures to
learn semantic embeddings of the input, where the distance between similar
examples is small while dissimilar ones are far apart. Although the underlying
neural networks produce good accuracy on naturally occurring samples, they are
vulnerable to adversarially-perturbed samples that reduce performance. We take
a first step towards training robust DML models and tackle the primary
challenge of the metric losses being dependent on the samples in a mini-batch,
unlike standard losses that only depend on the specific input-output pair. We
analyze this dependence effect and contribute a robust optimization
formulation. Using experiments on three commonly-used DML datasets, we
demonstrate 5-76 fold increases in adversarial accuracy, and outperform an
existing DML model that sought out to be robust.
