Smoothed Contrastive Learning for Unsupervised Sentence Embedding (2022)

Abstract: Unsupervised contrastive sentence embedding models, e.g., unsupervised SimCSE, use the InfoNCE loss function in training. Theoretically, we expect to use larger batches to get more adequate comparisons among samples and avoid overfitting. However, increasing batch size leads to performance degradation when it exceeds a threshold, which is probably due to the introduction of false-negative pairs through statistical observation. To alleviate this problem, we introduce a simple smoothing strategy upon the InfoNCE loss function, termed Gaussian Smoothed InfoNCE (GS-InfoNCE). In other words, we add random Gaussian noise as an extension to the negative pairs without increasing the batch size. Through experiments on the semantic text similarity tasks, though simple, the proposed smoothing strategy brings improvements to unsupervised SimCSE.

## introduction good sentence representation benefits many natural language processing tasks, and sentence representation learning has been widely studied @xcite . contrastive learning has recently been proposed and extensively explored to learn high-quality sentence representations based on the pre-trained language models @xcite . contrastive learning aims to learn effective representation by pulling close semantically similar sentences while pushing apart dissimilar ones @xcite . among those unsupervised sentence embedding learning methods with contrastive learning, the latest state-of-the-art method, as far as we know, is unsupervised simcse (unsup-simcse) @xcite . unsup-simcse implicitly hypothesizes "dropout" as minimal data augmentation and ## background: contrastive learning contrastive learning is a discriminative representation learning framework extensively used for unsupervised representation learning. the core idea is to compare a sentence with a semantically similar one (i.e., positive example) and many semantically dissimilar ones (i.e., negative examples). in this way, the semantically similar sentences are closer in the representation space, while the semantically dissimilar ones are farther apart. infonce @xcite propose to take a cross-entropy objective with in-batch negatives, namely the infonce objective function. it is a commonly used loss function for contrast learning by pulling similar sentences closer and pushing dissimilar ones apart in the representation space. specifically, given a set of sentence pairs: ## gaussian smoothed infonce we introduce a gaussian noise term to the infonce loss function, termed gaussian smoothed infonce (gs-infonce). given a gaussian distribution as follows: g ∼ n µ, σ foot_2 , whose mean is µ, and the variance is σ 2 , we randomly sample m gaussian noise vectors from it with the same dimensions as the sentence vector. these vectors constitute high confident negative pairs with each sample in the batch to fill and smooth the representation space. note that these gaussian noise vectors will not participate in the positive pair constitution. in that way, the loss function of gs-infonce is denoted as follows: the python implementation of gs-infonce is quite simple, with only three lines of codes based on the original infonce implementation in unsup-simcse. ## experiments we focus on unsup-simcse and replace the original infonce objective loss function with gs-infonce. following @xcite , the main goal of sentence embeddings is to cluster semantically similar sentences. for a fair comparison, we conduct our experiments on seven semantic textual similarity (sts) tasks introduced below and take sts results to compare sentence embedding methods. semantic textual similarity tasks semantic textual similarity measures the semantic similarity of any two sentences. sts 2012-2016 @xcite @xcite @xcite and sts benchmark @xcite are widely used semantic textual simiarlty benchmark datasets. following unsup-simcse, we use spearman correlation 2 to measure the correlation between the ranks of predicted scores and the ground-truth. ## related work to use label smoothing as a regularization method that makes the clusters between categories more compact and avoids adversarial examples with over high confidence. text smoothing @xcite also seems to be able to bring further improvements in tasks such as text classification and machine translation by smoothing the one-hot representation of the input text into the probability distribution representation of the dictionary. our gs-infonce can also be regarded as a smoothing strategy that makes the distribution of negative samples smoother by introducing multiple random gaussian noise vectors as an extension of the negative examples. compared with label smoothing and text smoothing, gs-infonce directly uses the standard gaussian distribution for sampling, largely saving computational costs. ## conclusion and future work this paper proposes gs-infonce for unsupervised simcse methods by introducing a simple smoothing strategy upon the infonce loss function to bring sufficient comparison for samples without increasing the batch size. in the future, we will explore how to improve the generalization capability of gs-infonce and verify its effectiveness on more contrastive learning methods.