ES im CSE : Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding (2022)

Abstract: Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as a minimal data augmentation method, and passes the same input sentence to a pre-trained Transformer encoder (with dropout turned on) twice to obtain the two corresponding embeddings to build a positive pair. As the length information of a sentence will generally be encoded into the sentence embeddings due to the usage of position embedding in Transformer, each positive pair in unsup-SimCSE actually contains the same length information. And thus unsup-SimCSE trained with these positive pairs is probably biased, which would tend to consider that sentences of the same or similar length are more similar in semantics. Through statistical observations, we find that unsup-SimCSE does have such a problem. To alleviate it, we apply a simple repetition operation to modify the input sentence, and then pass the input sentence and its modified counterpart to the pre-trained Transformer encoder, respectively, to get the positive pair. Additionally, we draw inspiration from the community of computer vision and introduce a momentum contrast, enlarging the number of negative pairs without additional calculations. The proposed two modifications are applied on positive and negative pairs separately, and build a new sentence embedding method, termed Enhanced Unsup-SimCSE (ESimCSE). We evaluate the proposed ESimCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that ESimCSE outperforms the state-of-the-art unsup-SimCSE by an average Spearman correlation of 2.02% on BERT-base.

## introduction recently, researchers have proposed using contrastive learning to learn better unsupervised sentence embeddings @xcite @xcite @xcite . contrastive learning aims to learn effective sentence embeddings based on the assumption that length diff avg. similarity @xmath0.34 ≤ 3 18.18 (+1.84) using dropout as a minimal data augmentation method is simple and effective, but there is a weak point. simcse models are built on transformer blocks, which will encode a sentence's length information through positional embeddings. in a positive pair, two embeddings are derived from the same sentence to contain the same length information. in contrast, in a negative pair, two embeddings in a negative pair are derived from two different sentences and generally contain different length information. therefore, positive and negative pairs are different in their length information, acting as a feature to distinguish them. the semantic simi-method text similarity original sentence i like this apple because it looks so fresh and it should be delicious. as shown in comparison details on each dataset can refer to to alleviate this problem, we propose a simple but effective enhancement method to simcse. for each positive pair, we expect to change the length of a sentence without changing its semantic meaning. existing methods to change the length of a sentence generally use random insertion and random deletion. however, inserting randomly selected words into a sentence may introduce extra noise, which will probably distort the meaning of the sentence; deleting keywords from a sentence will also change its semantics substantially. such operations are detrimental to simcse learning, which is also discussed in a contemporaneous work @xcite . therefore, we propose a safer method, termed "word repetition", which randomly duplicates some words in a sentence. for example, as shown in apart from the optimization above for positive pairs construction, we further explore how to optimize the construction of negative pairs. since contrastive learning is carried out between positive pairs and negative pairs, theoretically, more negative pairs can lead to a better comparison between the pairs @xcite . and thus, a potential optimization direction is to leverage more negative pairs, encouraging the model towards more refined learning. however, according to @xcite , larger batch size is not always a better choice. for example, for the simcse-bert base model, the optimal batch size is 64, and other settings of the batch size will lower the performance. therefore, we tend to figure out how to expand the negative pairs more effectively. in the community of computer vision, to alleviate the gpu memory limitation when expanding the batch size, a feasible way is to introduce the momentum contrast @xcite , which is also applied to natural language understanding @xcite . momentum contrast allows us to reuse the encoded embeddings from the immediate preceding mini-batches to expand the negative pairs by maintaining a queue. it always enqueues the sentence embeddings of the current mini-batches and meanwhile dequeues the "oldest" ones. as the enqueued sentence embeddings come from the preceding mini-batches, we keep a momentum updated encoder by taking the moving average of its parameters and use the momentum encoder to generate enqueued sentence embeddings. note that, we turn off dropout when using the momentum encoder, which can narrow the gap between training and prediction. the above two optimizations are proposed separately for building positive and negative pairs. we finally combine both with simcse, termed enhanced simcse (esimcse). we illustrate the schematic diagram of esimcse in figure 1 . the proposed esimcse is evaluated on the semantic text similarity (sts) task with 7 sts-b test sets. experimental results show that esimcse can improve the similarity measuring performance in different model settings over the previous state-ofthe-art simcse. specifically, esimcse gains an average increase of spearman's correlation over simcse by +2.02% on bert base . our contributions can be summarized as follows: we propose a simple but effective "word repetition" method to alleviate the problem. ## background: simcse with h i and h + i for each sentence in a mini-batch with batch size n , the contrastive learning objective w.r.t x i is formulated as follows, ## proposed enhanced simcse in this section, we first introduce the word repetition method to construct better positive pairs. then we introduce the momentum contrast method to expand negative pairs. ## ablation study this section investigates how different settings affect esimcse's performance. all results are compared on bert base scale models and are evaluated on the development set of sts-b unless otherwise specified. ## related work unsupervised sentence representation learning has been widely studied. @xcite propose to learn sentence representation according to the internal structure of each sentence. @xcite predict the surrounding sentences of a given sentence based on the distribution hypothesis. @xcite propose sent2vec, a simple unsupervised model allowing to compose sentence embeddings using word vectors along with n-gram embeddings. recently, contrastive learning has been explored in unsupervised sentence representation learning and has become a promising trend @xcite @xcite @xcite @xcite . those contrastive learning based methods for sentence embeddings are generally based on the assumption that a good semantic representation should be able to bring similar sentences closer while pushing away dissimilar ones. therefore, those methods use various data augmentation methods to randomly generate two different views for each sentence and design an effective loss function to make them closer in the semantic representation space. among these contrastive methods, the most related ones to our work are unsup-consert @xcite and unsup-simsce @xcite . consert explores various effective data augmentation strategies(e.g., adversarial attack, token shuffling, cutoff, dropout) to generate different views for contrastive learning and analyze their effects on unsupervised sentence representation transfer. unsup-simsce, the current state-of-the-art unsupervised method uses only standard dropout as minimal data augmentation, and feed an identical sentence to a pretrained model twice with independently sampled dropout masks to generate two distinct sentence embeddings as a positive pair. unsup-simsce is very simple but works surprisingly well, performing on par with previously supervised counterparts. however, we find that simcse constructs each positive pair with two sentences of the same length, which can mislead the learning of sentence embeddings. so we propose a simple but effective method temed "word repetition" to alleviate it. we also propose to use the momentum contrast method to increase the number of negative pairs involved in the loss calculation, which encourages the model towards more refined learning. ## conclusion and future work in this paper, we propose optimizations to construct positive and negative pairs for simcse and combine them with simcse, which is termed esim-cse. through extensive experiments, the proposed esimcse achieves considerable improvements on standard semantic text similarity tasks over sim-cse. in the future, we will focus on designing a more refined objective function to improve the discrimination between different negative pairs. also we will make attempt to optimize the performance on both semantic textual similarity tasks and transfer tasks.