Clustering-Aware Negative Sampling for Unsupervised Sentence Representation (2023)

Abstract: Contrastive learning has been widely studied in sentence representation learning. However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples. This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives. To address these issues, we propose ClusterNS (Clustering-aware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework. Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning. Our code has been made publicly available at github.com/djz233/ClusterNS.

## introduction learning sentence representation is one of the fundamental tasks in natural language processing and has been widely studied @xcite @xcite . @xcite show that sentence embeddings produced by bert @xcite are even worse than glove embeddings @xcite , attracting more research on sentence representation with pretrained language models (plms) @xcite @xcite . @xcite and @xcite further find out that plm embeddings suffer from anisotropy, motivating more researchers to study this issue @xcite . besides, gao et al. a man is playing guitar. ## analysis to obtain more insights about how clustering helps the training process, we visualize the variation of diverse sentence pairs similarity during training after clustering initialization in the non-prompt clusterns-roberta base model, and analyze the results in detail. ## clustering evaluation we also evaluate the quality of sentence embedding through clustering. we first use the dbpedia dataset @xcite , an ontology classification dataset extracted from wikipedia and consists of 14 classes in total. we implement k-means clustering (@xmath0.6926 0.7078 0.7355 dbpedia, and take the adjusted mutual information (ami) score as the evaluation metric following @xcite . the results in ## alignment and uniformity to investigate how clusterns improves the sentence embedding, we conduct further analyses on two widely used metrics in contrastive learning proposed by wang and isola (2020), alignment and uniformity. alignment measures the expected distance between the embeddings of positive pairs: and uniformity measures the expected distance between the embeddings of all sentence pairs: ## conclusion in this paper, we propose clusterns, a novel approach that focuses on improving the negative sampling for contrastive learning in unsupervised sentence representation learning. we integrate clustering into the training process and use the clustering results to generate additional hard negatives and identify false negatives for each sample. we also use a bidirectional margin loss to constrain the false negatives. our experiments on sts tasks show improvements over baseline models and demonstrate the effectiveness of clusterns.