Debiased Contrastive Learning of Unsupervised Sentence Representations (2022)

Abstract: Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives (false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework DCLR (Debiased Contrastive Learning of unsupervised sentence Representations) to alleviate the influence of these improper negatives.In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space.Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: blue https://github.com/RUCAIBox/DCLR .

## introduction as a fundamental task in the natural language processing (nlp) field, unsupervised sentence representation learning @xcite aims to derive high-quality sentence representations that can benefit various downstream tasks, especially for low-resourced domains or computationally expensive tasks, e.g., zero-shot text semantic matching @xcite , large-scale semantic similarity comparison @xcite , and document retrieval @xcite . recently, pre-trained language models (plms) @xcite have become a widely-used se- † † corresponding author figure 1 : the distribution of cosine similarity between an input sentence and 255 in-batch negatives from the commonly-used wikipedia corpus. it is evaluated by the simcse model @xcite . almost half of the negatives have high similarities with the input. mantic representation approach, achieving remarkable performance on various nlp tasks. however, several studies have found that the native sentence representations derived by plms are not uniformly distributed with respect to directions, but instead occupy a narrow cone in the vector space @xcite , which largely limits their expressiveness. to address this issue, contrastive learning @xcite has been adopted to refine plm-derived sentence representations. it pulls semantically-close neighbors together to improve the alignment, while pushes apart non-neighbors for the uniformity of the whole representation space. in the learning process, both positive and negative examples are involved in contrast with the original sentence. for positive examples, previous works apply data augmentation strategies @xcite on the original sentence to generate highly similar variations. while, negative examples are commonly sampled from the batch or training data (e.g., in-batch negatives @xcite ) at random, due to the lack of ground-truth annotations for negatives. although such a negative sampling way is simple and convenient, it may cause sampling bias and affect the sentence representation learning. first, the sampled negatives are likely to be false negatives that are indeed semantically close to the original sentence. as shown in figure 1 , given an input sentence, about half of in-batch negatives have a cosine similarity above 0.7 with the original sentence based on the simcse model @xcite . it is likely to hurt the semantics of the sentence representations by simply pushing apart these sampled negatives. second, due to the anisotropy problem @xcite , the representations of sampled negatives are from the narrow representation cone spanned by plms, which cannot fully reflect the overall semantics of the representation space. hence, it is sub-optimal to only rely on these representations for learning the uniformity objective of sentence representations. to address the above issues, we aim to develop a better contrastive learning approach with debiased negative sampling strategies.the core idea is to improve the random negative sampling strategy for alleviating the sampling bias problem. first, in our framework, we design an instance weighting method to punish the sampled false negatives during training. we incorporate a complementary model to evaluate the similarity between each negative and the original sentence, then assign lower weights for negatives with higher similarity scores. in this way, we can detect semanticallyclose false negatives and further reduce their influence. second, we randomly initialize new negatives based on random gaussian noises to simulate sampling within the whole semantic space, and devise a gradient-based algorithm to optimize the noise-based negatives towards the most nonuniform points. by learning to contrast with the nonuniform noise-based negatives, we can extend the occupied space of sentence representations and improve the uniformity of the representation space. to this end, we propose dclr, a general framework towards debiased contrastive learning of unsupervised sentence representations. in our approach, we first initialize the noise-based negatives from a gaussian distribution, and leverage a gradient-based algorithm to update the new negatives by considering the uniformity of the representation space. then, we adopt the complementary model to produce the weights for these noise-based negatives and randomly sampled negatives, where the false negatives will be punished. finally, we augment the positive examples via dropout @xcite and combine them with the above weighted negatives for contrastive learning. we demonstrate that our dclr outperforms a number of competitive baselines on seven semantic textual similarity (sts) tasks using bert @xcite and roberta @xcite . our contributions are summarized as follows: ## related work in this section, we review the related work from the following three aspects. ## preliminary this work aims to make use of unlabeled corpus for learning effective sentence representations that can be directly utilized for downstream tasks, e.g., semantic textual similarity task @xcite . given a set of input sentences @xmath0, x 2 , . . . , x n }, our goal is to learn a representation h @xmath1. for simplicity, we denote this process with a parameterized function h in this work, we mainly focus on using bertbased plms @xcite to generate sentence representations. following existing works @xcite , we fine-tune plms on the unlabeled corpus via our proposed unsupervised learning approach. after that, for each sentence x i , we encode it by the fine-tuned plms and take the representation of the [cls] token from the last layer as its sentence representation h i . ## approach our proposed framework dclr focuses on reducing the influence of sampling bias in the con-trastive learning of sentence representations. in this framework, we devise a noise-based negatives generation strategy to reduce the bias caused by the anisotropy plm-derived representations, and an instance weighting method to reduce the bias caused by false negatives. concretely, we initialize the noise-based negatives based on a gaussian distribution and iteratively update these negatives towards non-uniformity maximization. then, we utilize a complementary model to produce weights for all negatives (i.e., randomly sampled and the noisebased ones). finally, we combine the weighted negatives and augmented positive examples for contrastive learning. the overview of our dclr is presented in figure 2 . ## experiment -analysis and extension in this section, we continue to study the effectiveness of our proposed dclr. ## conclusion in this paper, we proposed dclr, a debiased contrastive learning framework for unsupervised sentence representation learning. our core idea is to alleviate the sampling bias caused by the random negative sampling strategy. to achieve it, in our framework, we incorporated an instance weighting method to punish false negatives during training and generated noise-based negatives to alleviate the influence of anisotropy plm-derived representation. experimental results on seven sts tasks have shown that our approach outperforms several competitive baselines. in the future, we will explore other approaches to reducing the bias in contrastive learning of sentence representations (e.g., debiased pre-training). besides, we will also consider to apply our method for multilingual or multimodal representation learning.