  Modern natural language processing (NLP) methods employ self-supervised
pretraining objectives such as masked language modeling to boost the
performance of various application tasks. These pretraining methods are
frequently extended with recurrence, adversarial or linguistic property
masking, and more recently with contrastive learning objectives. Contrastive
self-supervised training objectives enabled recent successes in image
representation pretraining by learning to contrast input-input pairs of
augmented images as either similar or dissimilar. However, in NLP, automated
creation of text input augmentations is still very challenging because a single
token can invert the meaning of a sentence. For this reason, some contrastive
NLP pretraining methods contrast over input-label pairs, rather than over
input-input pairs, using methods from Metric Learning and Energy Based Models.
In this survey, we summarize recent self-supervised and supervised contrastive
NLP pretraining methods and describe where they are used to improve language
modeling, few or zero-shot learning, pretraining data-efficiency and specific
NLP end-tasks. We introduce key contrastive learning concepts with lessons
learned from prior research and structure works by applications and cross-field
relations. Finally, we point to open challenges and future directions for
contrastive NLP to encourage bringing contrastive NLP pretraining closer to
recent successes in image representation pretraining.
