  Many speech processing methods based on deep learning require an automatic
and differentiable audio metric for the loss function. The DPAM approach of
Manocha et al. learns a full-reference metric trained directly on human
judgments, and thus correlates well with human perception. However, it requires
a large number of human annotations and does not generalize well outside the
range of perturbations on which it was trained. This paper introduces CDPAM, a
metric that builds on and advances DPAM. The primary improvement is to combine
contrastive learning and multi-dimensional representations to build robust
models from limited data. In addition, we collect human judgments on triplet
comparisons to improve generalization to a broader range of audio
perturbations. CDPAM correlates well with human responses across nine varied
datasets. We also show that adding this metric to existing speech synthesis and
enhancement methods yields significant improvement, as measured by objective
and subjective tests.
