  This paper presents a novel method for embedding transfer, a task of
transferring knowledge of a learned embedding model to another. Our method
exploits pairwise similarities between samples in the source embedding space as
the knowledge, and transfers them through a loss used for learning target
embedding models. To this end, we design a new loss called relaxed contrastive
loss, which employs the pairwise similarities as relaxed labels for
inter-sample relations. Our loss provides a rich supervisory signal beyond
class equivalence, enables more important pairs to contribute more to training,
and imposes no restriction on manifolds of target embedding spaces. Experiments
on metric learning benchmarks demonstrate that our method largely improves
performance, or reduces sizes and output dimensions of target models
effectively. We further show that it can be also used to enhance quality of
self-supervised representation and performance of classification models. In all
the experiments, our method clearly outperforms existing embedding transfer
techniques.
