A ug CSE : Contrastive Sentence Embedding with Diverse Augmentations (2022)

Abstract: Data augmentation techniques have been proven useful in many applications in NLP fields. Most augmentations are task-specific, and cannot be used as a general-purpose tool. In our work, we present AugCSE, a unified framework to utilize diverse sets of data augmentations to achieve a better, general-purpose, sentence embedding model. Building upon the latest sentence embedding models, our approach uses a simple antagonistic discriminator that differentiates the augmentation types. With the finetuning objective borrowed from domain adaptation, we show that diverse augmentations, which often lead to conflicting contrastive signals, can be tamed to produce a better and more robust sentence representation. Our methods achieve state-of-the-art results on downstream transfer tasks and perform competitively on semantic textual similarity tasks, using only unsupervised data.

## introduction data augmentation in nlp can be useful in many situations, from low resource data setting, domain adaptation @xcite , debiasing @xcite , to improving generalization, robustness @xcite . in the vision domain, @xcite shows that a diverse set of augmentation can be used to learn a robust generalpurpose representation with contrastive learning. similar work in sentence embedding space @xcite has shown that a simple single augmentation such as dropouts from transformers @xcite can be used for contrastive objective. however, no previous work has thoroughly explored the impacts of a diverse set of augmentations with contrastive learning in the sentence embedding space. it is not straightforward to find the best augmentations that work for contrastive learning in different datasets or tasks @xcite . single augmentation can instill invariance in models for a specific aspects of linguistic variability, while naively combining a diverse set of augmentations can lead to contradicting gradients, preventing models from generalizing well ( ## analysis and discussion in our experiments, we selected subsets of top performing augmentations by looking at their individual finetuned performances. such selection procedure may not be feasible due to resource constraints. in the following sections and in app. a.13, we discuss a few metrics that could be used to provide some signal in selecting the best augmentation (or dataset) for contrastive learning. we also discuss the broader impact of our work, advantages, and yet unresolved problems in the field. ## conclusion we present augcse, a general framework that combines diverse sets of augmentations to improve general sentence embeddings. in addition to the contrastive loss, we introduce an antagonistic discriminator that loosely constrain the model to be-7 sentence-transformers/all-mpnet-base-v2 come invariant to distributional shifts created from augmentations. in addition to outperforming previous methods, our framework is much more controllable, which has an added advantage of being able to mitigate undesirable properties from pretrained lms, which inherit bias and toxicity from training data on the internet. additionally, augcse can work with cheaper augmentations to run, resulting in a more resource-friendly approach to training generic sentence embedding models.