Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding (2022)

Abstract: Contrastive learning is emerging as a powerful technique for extracting knowledge from unlabeled data. This technique requires a balanced mixture of two ingredients: positive (similar) and negative (dissimilar) samples. This is typically achieved by maintaining a queue of negative samples during training. Prior works in the area typically uses a fixed-length negative sample queue, but how the negative sample size affects the model performance remains unclear. The opaque impact of the number of negative samples on performance when employing contrastive learning aroused our in-depth exploration. This paper presents a momentum contrastive learning model with negative sample queue for sentence embedding, namely MoCoSE. We add the prediction layer to the online branch to make the model asymmetric and together with EMA update mechanism of the target branch to prevent the model from collapsing. We define a maximum traceable distance metric, through which we learn to what extent the text contrastive learning benefits from the historical information of negative samples. Our experiments find that the best results are obtained when the maximum traceable distance is at a certain range, demonstrating that there is an optimal range of historical information for a negative sample queue. We evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS) task and obtain an average Spearman’s correlation of 77.27%. Source code is available here.

## introduction in recent years, unsupervised learning has been brought to the fore in deep learning due to its ability to leverage large-scale unlabeled data. various unsupervised contrastive models is emerging, continuously narrowing down the gap between supervised and unsupervised learning. contrastive learning suffers from the problem of model collapse, where the model converges to a constant value and the samples all mapped to a single point in the feature space. negative samples are an effective way to solve this problem. in computer vision, simclr from chen @xcite and moco from he @xcite is known for using negative samples and get the leading performance in the contrastive learning. simclr uses different data augmentation (e.g., rotation, masking, etc.) on the same image to construct positive samples, and negative samples are from the rest of images in the same batch. moco goes a step further by randomly select the data in entire unlabeled training set to stack up a first-infirst-out negative sample queue. recently in natural language processing, contrastive learning has been widely used in the task of learning sentence embedding. one of current state-of-the-art unsupervised method is simcse @xcite . its core idea is to make similar sentences in the embedding space closer while keeping dissimilar away from each other. simcse uses dropout mask as augmentation to construct positive text sample pairs, and negative samples are picked from the rest of sentences in the same batch. the mask adopted from the standard transformer makes good use of the minimal form of data augmentation brought by the dropout. dropout results in a minimal difference without changing the semantics, reducing the negative noise introduced by augmentation. however, the negative samples in simcse are selected from the same training batch with a limited batch size. our further experiments show that simcse does not obtain improvement as the batch size increases, which arouses our interest in using the negative sample queue. to better digging in the performance of contrastive learning on textual tasks, we build a contrastive model consisting of a two-branch structure and a negative sample queue, namely mocose (momentum contrastive sentence embedding with negative sample queue). we also introduce the idea of asymmetric structure from byol @xcite by adding a prediction layer to the upper branch (i.e., the online branch). the lower branch (i.e., the target branch) is updated with exponential moving average (ema) method during training. we set a negative sample queue and update it using the output of target branch. unlike directly using negative queue as in moco, for research purpose, we set an initialization process with a much smaller negative queue, and then filling the entire queue through training process, and update normally. we test both character-level (e.g., typo, back translation, paraphrase) and vector-level (e.g., dropout, shuffle, etc.) data augmentations and found that for text contrastive learning, the best results are obtained by using fgsm and dropout as augmentations. using the proposed mocose model, we design a series of experiments to explore the contrastive learning for sentence embedding. we found that using different parts of samples from the negative queue leads to different performance. in order to test how much text contrastive learning benefit from historical information of the model, we proposed a maximum traceable distance metric. the metric calculates how many update steps before the negative samples in the queue are pushed in, and thus measures the historical information contained in the negative sample queue. we find that the best results can be achieved when the maximum traceable distance is within a certain range, reflected in the performance of uniformity and alignment of the learned text embedding. which means there is an optimal interval for the length of negative sample queue in text contrastive learning model. our main contributions are as follows: contrastive learning in nlp since bert @xcite redefined stateof-the-art in nlp, leveraging the bert model to obtain better sentence representation has become a common task in nlp. a straightforward way to get sentence embedding is by the [cls] token due to the next sentence prediction task of bert. but the [cls] embedding is non-smooth anisotropic in semantic space, which is not conducive to sts tasks, this is known as the representation degradation problem @xcite . bert-flow @xcite and bert-whitening @xcite solve the degradation problem by post-processing the output of bert. simcse found that utilizing contrasting mechanism can also alleviate this problem. data augmentation is crucial for contrastive learning. in clear @xcite , word and phrase deletion, phrase order switching, synonym substitution is served as augmentation. cert @xcite ) mainly using back-and-forth translation, and cline @xcite proposed synonym substitution as positive samples and antonym substitution as negative samples, and then minimize the triplet loss between positive, negative cases as well as the original text. consert @xcite uses adversarial attack, token shuffling, cutoff, and dropout as data augmentation. clae @xcite ) also introduces fast gradient sign method, an adversarial attack method, as text data augmentation. several of these augmentations are also introduced in our work. the purpose of data augmentation is to create enough distinguishable positive and negative samples to allow contrastive loss to learn the nature of same data after different changes. works like @xcite points out that longer negative sample queues do not always give the best performance. this also interests us how the negative queue length affects the text contrastive learning. finally, the model calculates contrasting loss between query, key and negative queue to update the online branch. in the process, key vector serves as positive sample with respect to the query vector, while the sample from queue serves as negative sample to the query. the target branch truncates the gradient and updated with the ema mechanism. the queue is a first-in-first-out collection of negative samples with size k which means it sequentially stores the key vectors generated from the last few training steps. the pytorch style pseudo-code for training mo-cose with the negative sample queue is shown in algorithm 1 in appendix a.3. ## empirical study to further explore the performance of the mocolike contrasting model on learning sentence embedding, we set up the following ablation trials. ## conclusion in this study, we propose mocose, it applies the moco-style contrastive learning model to the empirical study of sentence embedding. we conducted experiments to study every detail of the model to provide some experiences for text contrastive learning. we further delve into the application of the negative sample queue to text contrastive learning and propose a maximum traceable distance metric to explain the relation between the queue size and model performance. zhong zhang, chongming gao, cong xu, rui miao, qinli yang, and junming shao. 2020b. revisiting representation degeneration problem in language modeling. in findings of the association for computational linguistics: emnlp 2020, online event, 16-20 november 2020, volume emnlp 2020 of findings of acl, pages 518-527. association for computational linguistics.