D iff CSE : Difference-based Contrastive Learning for Sentence Embeddings (2022)

Abstract: We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning, which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other “harmful” types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.

## introduction learning "universal" sentence representations that capture rich semantic information and are at the same time performant across a wide range of downstream nlp tasks without task-specific finetuning is an important open issue in the field @xcite @xcite @xcite @xcite . recent work has shown that finetuning pretrained language models with contrastive learning makes it possible to learn good sentence embeddings without any labeled data @xcite @xcite . contrastive learning uses multiple augmentations on a single datum to construct positive pairs whose representations are trained to be more similar to one another than negative pairs. while different data augmentations (random cropping, color jitter, rotations, etc.) have been found to be crucial for pretraining vision models @xcite , such augmentations have generally been unsuccessful when applied to contrastive learning of sentence embeddings. indeed, @xcite find that constructing positive pairs via a simple dropout-based augmentation works much better than more complex augmentations such as word deletions or replacements based on synonyms or masked language models. this is perhaps unsurprising in hindsight; while the training objective in contrastive learning encourages representations to be invariant to augmentation transformations, direct augmentations on the input (e.g., deletion, replacement) often change the meaning of the sentence. that is, ideal sentence embeddings should not be invariant to such transformations. we propose to learn sentence representations that are aware of, but not necessarily invariant to, such direct surface-level augmentations. this is an instance of equivariant contrastive learning @xcite , which improves vision representation learning by using a contrastive loss on insensitive image transformations (e.g., grayscale) and a prediction loss on sensitive image transformations (e.g., rotations). we operationalize equivariant contrastive learning on sentences by using dropout-based augmentation as the insensitive transformation (as in simcse @xcite ) and mlm-based word replacement as the sensitive transformation. this results in an additional crossentropy loss based on the difference between the original and the transformed sentence. we conduct experiments on 7 semantic textual similarity tasks (sts) and 7 transfer tasks from sen-teval @xcite and find that this difference-based learning greatly improves over standard contrastive learning. our diffcse approach can achieve around 2.3% absolute improve-"you @xcite know what you're gonna ment on sts datasets over simcse, the previous state-of-the-art model. we also conduct a set of ablation studies to justify our designed architecture. qualitative study and analysis are also included to look into the embedding space of diffcse. ## difference-based contrastive learning our approach is straightforward and can be seen as combining the standard contrastive learning objective from simcse (figure 1 , left) with a difference prediction objective which conditions on the sentence embedding (figure 1 @xcite . given an unlabeled input sentence x, simcse creates a positive example x + for it by applying different dropout masks. by using the bert base encoder f , we can obtain the sentence embedding @xmath0. the training objective for simcse is: on the right-hand side of figure 1 is a conditional version of the difference prediction objective used in electra @xcite , which contains a generator and a discriminator. given a sentence of length t , @xmath1, x (2) , ..., x (t ) ], we first apply a random mask @xmath2, m (2) , ..., m (t ) ], m (t) ∈ [0, 1] on x to obtain x ′ = m • x. we use another pretrained mlm as the generator g to perform masked language modeling to recover randomly masked tokens in x ′ to obtain the edited sentence x ′′ = g(x ′ ). then, we use a discriminator d to perform the replaced token detection (rtd) task. for each token in the sentence, the model needs to predict whether it has been replaced or not. the cross-entropy loss for a single sentence x is: finally we optimize these two losses together with a weighting coefficient λ: the difference between our model and electra is that our discriminator d is conditional, so it can use the information of x compressed in a fixeddimension vector @xmath3. the gradient of d can be backward-propagated into f through h. by doing so, f will be encouraged to make h informative enough to cover the full meaning of x, so that d can distinguish the tiny difference between x and x ′′ . this approach essentially makes the conditional discriminator perform a "diff operation", hence the name diffcse. when we train our diffcse model, we fix the generator g, and only the sentence encoder f and the discriminator d are optimized. after training, we discard d and only use f (which remains fixed) to extract sentence embeddings to evaluate on the downstream tasks. ## ablation studies in the following sections, we perform an extensive series of ablation studies that support our model design. we use bert base model to evaluate on the development set of sts-b and transfer tasks. ## analysis 6.1 qualitative study a very common application for sentence embeddings is the retrieval task. here we show some retrieval examples to qualitatively explain why dif-fcse can perform better than simcse. in this study, we use the 2758 sentences from sts-b testing set as the corpus, and then use sentence query to retrieve the nearest neighbors in the sentence embedding space by computing cosine similarities. we show the retrieved top-3 examples in the first query sentence is "you can do it, too.". the simcse model retrieves a very similar sentence but has a slightly different meaning ("you can use it, too.") as the rank-1 answer. in contrast, diffcse can distinguish the tiny difference, so it retrieves the ground truth answer as the rank-1 answer. the second query sentence is "this is not a problem". simcse retrieves a sentence with opposite meaning but very similar wording, while diffcse can retrieve the correct answer with less similar wording. we also provide a third example where both simcse and diffcse fail to retrieve the correct answer for a query sentence using double negation. ## conclusion in this paper, we present diffcse, a new unsupervised sentence embedding framework that is aware of, but not invariant to, mlm-based word replacement. empirical results on semantic textual similarity tasks and transfer tasks both show the effectiveness of diffcse compared to current stateof-the-art sentence embedding methods. we also conduct extensive ablation studies to demonstrate the different modeling choices in diffcse. qualitative study and the retrieval results also show that diffcse can produce a better embedding space for sentence retrieval. one limitation of our work is that we do not explore the supervised setting that uses human-labeled nli datasets to further boost the performance. we leave this topic for future work. we believe that our work can provide researchers in the nlp community a new way to utilize augmentations for natural language and thus produce better sentence embeddings.