S im CSE : Simple Contrastive Learning of Sentence Embeddings (2021)

Abstract: This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using “entailment” pairs as positives and “contradiction” pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman’s correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show—both theoretically and empirically—that contrastive learning objective regularizes pre-trained embeddings’ anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

## introduction learning universal sentence embeddings is a fundamental problem in natural language processing and has been studied extensively in the literature @xcite @xcite @xcite . in this work, we advance state-of-the-art sentence embedding methods and demonstrate that a contrastive objective can be extremely effective when coupled with pre-trained language models such as bert @xcite or roberta @xcite . we present simcse, a simple contrastive sentence embedding framework, which can produce superior sentence embeddings, from either unlabeled or labeled data. our unsupervised simcse simply predicts the input sentence itself with only dropout @xcite used as noise (figure 1(a) ). in other words, we pass the same sentence to the pre-trained encoder twice: by applying the standard dropout twice, we can obtain two different embeddings as "positive pairs". then we take other sentences in the same mini-batch as "negatives", and the model predicts the positive one among negatives. although it may appear strikingly simple, this approach outperforms training objectives such as predicting next sentences @xcite and discrete data augmentation (e.g., word deletion and replacement) by a large margin, and even matches previous supervised methods. through careful analysis, we find that dropout acts as minimal "data augmentation" of hidden representations, while removing it leads to a representation collapse. our supervised simcse builds upon the recent success of using natural language inference (nli) datasets for sentence embeddings @xcite ) and incorporates annotated sentence pairs in contrastive learning (figure 1(b) ). unlike previous work that casts it as a 3-way classification task (entailment, neutral and contradiction), we leverage the fact that entailment pairs can be naturally used as positive instances. we also find that adding corresponding contradiction pairs as hard negatives further improves performance. this simple use of nli datasets achieves a substantial improvement compared to prior methods using the same datasets. we also compare to other labeled sentence-pair the pets are sitting on a couch. ## background: contrastive learning contrastive learning aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors @xcite . it assumes a set of paired examples @xmath0, x + i )} m @xmath1, where x i and x + i are semantically related. we follow the contrastive framework in @xcite and take a cross-entropy objective with in-batch negatives @xcite : let h i and h + i denote the representations of x i and x + i , the training objective for (x i , x + i ) with a mini-batch of n pairs is: in this work, we encode input sentences using a pre-trained language model such as bert @xcite or roberta @xcite : @xmath2, and then fine-tune all the parameters using the contrastive learning objective (eq. 1). ## unsupervised simcse the idea of unsupervised simcse is extremely simple: we take a collection of sentences {x i } m @xmath3 the key ingredient to get this to work with identical positive pairs is through the use of independently sampled dropout masks for x i and x + i . in standard training of transformers @xcite , there are dropout masks placed on fully-connected layers as well as attention probabilities (default @xmath4.1). we denote h z @xmath5, z) where z is a random mask for dropout. we simply feed the same input to the encoder twice and get the two columns denote whether we use one encoder or two independent encoders. next 3 sentences: randomly sample one from the next 3 sentences. delete one word: delete one word randomly (see for a mini-batch of n sentences. note that z is just the standard dropout mask in transformers and we do not add any additional dropout. dropout noise as data augmentation. we view it as a minimal form of data augmentation: the positive pair takes exactly the same sentence, and their embeddings only differ in dropout masks. we compare this approach to other training objectives on the sts-b development set @xcite 2 . we also compare this self-prediction training objective to the next-sentence objective used in logeswaran and lee (2018), taking either one encoder or two independent encoders. as shown in ## supervised simcse we have demonstrated that adding dropout noise is able to keep a good alignment for positive pairs (x, x + ) ∼ p pos . in this section, we study whether we can leverage supervised datasets to provide better training signals for improving alignment of our approach. prior work @xcite has demonstrated that supervised natural language inference (nli) datasets @xcite are effective for learning sentence embeddings, by predicting whether the relationship between two sentences is entailment, neutral or contradiction. in our contrastive learning framework, we instead directly take (x i , x + i ) pairs from supervised datasets and use them to optimize eq. 1. choices of labeled data. we first explore which supervised datasets are especially suitable for constructing positive pairs (x i , x + i ). we experiment with a number of datasets with sentence-pair examples, including 1) qqp foot_1 : quora question pairs; 2) flickr30k @xcite : each image is annotated with 5 human-written captions and we consider any two captions of the same image as a positive pair; 3) paranmt @xcite : a large-scale back-translation paraphrase dataset foot_2 ; and finally 4) nli datasets: snli @xcite and mnli @xcite . we train the contrastive learning model (eq. 1) with different datasets and compare the results in contradiction as hard negatives. finally, we further take the advantage of the nli datasets by using its contradiction pairs as hard negatives 6 . in nli datasets, given one premise, annotators are required to manually write one sentence that is absolutely true (entailment), one that might be true (neutral), and one that is definitely false (contradiction). therefore, for each premise and its entailment hypothesis, there is an accompanying contradiction hypothesis foot_4 (see figure 1 for an example). formally, we extend (x i , x + i ) to (x i , x + i , x - i ), where x i is the premise, x + ## connection to anisotropy recent work identifies an anisotropy problem in language representations @xcite , i.e., the learned embeddings occupy a narrow cone in the vector space, which severely limits their expressiveness. demonstrate that language models trained with tied input/output embeddings lead to anisotropic word embeddings, and this is further observed by ethayarajh (2019) in pre-trained contextual representations. @xcite show that singular values of the word embedding matrix in a language model decay drastically: except for a few dominating singular values, all others are close to zero. a simple way to alleviate the problem is postprocessing, either to eliminate the dominant principal components @xcite , or to map embeddings to an isotropic distribution @xcite . another common solution is to add regularization during training @xcite . in this work, we show that-both theoretically and empirically-the contrastive objective can also alleviate the anisotropy problem. the anisotropy problem is naturally connected to uniformity (wang and isola, 2020), both highlighting that embeddings should be evenly distributed in the space. intuitively, optimizing the contrastive learning objective can improve uniformity (or ease the anisotropy problem), as the objective pushes negative instances apart. here, we take a singular spectrum perspective-which is a common practice in analyzing word embeddings @xcite @xcite , and show that the contrastive objective can "flatten" the singular value distribution of sentence embeddings and make the representations more isotropic. following @xcite , the asymptotics of the contrastive learning objective (eq. 1) can be expressed by the following equation when the number of negative instances approaches infinity (assuming f (x) is normalized): let w be the sentence embedding matrix corresponding to {x i } m @xmath6, i.e., the i-th row of w is h i . optimizing the second term in eq. 6 essentially minimizes an upper bound of the summation of all elements in ww , i.e., sum(ww ) = m @xmath7. since we normalize h i , all elements on the diagonal of ww are 1 and then tr(ww ) (the sum of all eigenvalues) is a constant. according to @xcite , if all elements in ww are positive, which is the case in most times according to figure g.1, then sum(ww ) is an upper bound for the largest eigenvalue of ww . when minimizing the second term in eq. 6, we reduce the top eigenvalue of ww and inherently "flatten" the singular spectrum of the embedding space. therefore, contrastive learning is expected to alleviate the representation degeneration problem and improve uniformity of sentence embeddings. compared to post-processing methods in @xcite ; @xcite , which only aim to encourage isotropic representations, contrastive learning also optimizes for aligning positive pairs by the first term in eq. 6, which is the key to the success of simcse. a quantitative analysis is given in §7. ## experiment 6.1 evaluation setup we conduct our experiments on 7 semantic textual similarity (sts) tasks. note that all our sts experiments are fully unsupervised and no sts training sets are used. even for supervised simcse, we simply mean that we take extra labeled datasets for training, following previous work @xcite . we also evaluate 7 transfer learning tasks and provide detailed results in appendix e. we share a similar sentiment with @xcite that the main goal of sentence embeddings is to cluster semantically similar sentences and hence take sts as the main result. semantic textual similarity tasks. we evaluate on 7 sts tasks: sts 2012-2016 @xcite @xcite @xcite , sts benchmark @xcite and sick-relatedness @xcite . when comparing to previous work, we identify invalid comparison patterns in published papers in the evaluation settings, including (a) whether to use an additional regressor, (b) spearman's vs pearson's correlation, and (c) how the results are aggregated (table b .1). we discuss the detailed differences in appendix b and choose to follow the setting of @xcite in our evaluation (no additional regressor, spearman's correlation, and "all" aggregation). we also report our replicated study of previous work as well as our results evaluated in a different setting in table b.2 and table b.3. we call for unifying the setting in evaluating sentence embeddings for future research. training details. we start from pre-trained checkpoints of bert @xcite or roberta @xcite (cased) and take the [cls] representation as the sentence embedding foot_6 (see §6.3 for comparison between different pooling methods). we train unsupervised simcse on 10 6 randomly sampled sentences from english wikipedia, and train supervised simcse on the combination of mnli and snli datasets (314k). more training details can be found in appendix a. ## analysis in this section, we conduct further analyses to understand the inner workings of simcse. uniformity and alignment. figure 3 shows uniformity and alignment of different sentence embedding models along with their averaged sts results. in general, models which have both better alignment and uniformity achieve better performance, confirming the findings in @xcite . we also observe that (1) though pre-trained embeddings have good alignment, their uniformity is poor (i.e., the embeddings are highly anisotropic); ## related work early work in sentence embeddings builds upon the distributional hypothesis by predicting surrounding sentences of a given one @xcite ## conclusion in this work, we propose simcse, a simple contrastive learning framework, which greatly improves state-of-the-art sentence embeddings on semantic textual similarity tasks. we present an unsupervised approach which predicts input sentence itself with dropout noise and a supervised approach utilizing nli datasets. we further justify the inner workings of our approach by analyzing alignment and uniformity of simcse along with other baseline models. we believe that our contrastive objective, especially the unsupervised one, may have a broader application in nlp. it provides a new perspective on data augmentation with text input, and can be extended to other continuous representations and integrated in language model pre-training.