  Anomaly detection aims at identifying deviant instances from the normal data
distribution. Many advances have been made in the field, including the
innovative use of unsupervised contrastive learning. However, existing methods
generally assume clean training data and are limited when the data contain
unknown anomalies. This paper presents Elsa, a novel semi-supervised anomaly
detection approach that unifies the concept of energy-based models with
unsupervised contrastive learning. Elsa instills robustness against any data
contamination by a carefully designed fine-tuning step based on the new energy
function that forces the normal data to be divided into classes of prototypes.
Experiments on multiple contamination scenarios show the proposed model
achieves SOTA performance. Extensive analyses also verify the contribution of
each component in the proposed model. Beyond the experiments, we also offer a
theoretical interpretation of why contrastive learning alone cannot detect
anomalies under data contamination.
