Contrastive Learning of Sentence Embeddings from Scratch (2023)

Abstract: Contrastive learning has been the dominant approach to train state-of-the-art sentence embeddings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLI) data or via large-scale unlabeled sentences in an unsupervised manner. However, even in the case of unlabeled data, their acquisition presents challenges in certain domains due to various reasons. due to copyright restrictions, data distribution issues, and messy formats, among other factors. To address these issues, we present SynCSE, a contrastive learning framework that trains sentence embeddings with synthetic data. Specifically, we explore utilizing large language models to synthesize the required data samples for contrastive learning, including (1) producing positive and negative annotations given unlabeled sentences SynCSE-partial, and (2) generating sentences along with their corresponding annotations from scratch SynCSE-scratch. Notably, SynCSE-scratch constitutes the first contrastive learning method to learn sentence embeddings from scratch without manually collecting any data sample. Experimental results on sentence similarity and reranking tasks indicate that both SynCSE-partial and SynCSE-scratch greatly outperform unsupervised baselines, and SynCSE-partial even achieves comparable performance to the supervised models in most settings.

## introduction the objective of sentence representation learning is to derive sentence embeddings that can benefit a wide range of downstream tasks, including reranking @xcite , natural language understanding @xcite , and retrieval @xcite @xcite . methods built on contrastive learning, such as simcse @xcite and promcse @xcite , have dominated the field due to their competitive performance @xcite devise ten distinct and diverse sentences that may appear in the pieces of content shared on social media platforms, covering a range of subjects (education, food, technology, history, architecture, war, etc.). these sentences should present a mix of complexity levels, from elementary structures akin to "birds fly in the sky." to more sophisticated ones. aim for a low degree of lexical overlap ... figure 1 : an overview of the data synthesis process of syncse-scratch. we specify a desired domain and genre, and our framework will generate diverse unlabeled data for that domain along with their positive and negative annotations. @xcite @xcite @xcite . contrastive learning trains sentence representations through distinguishing positive samples from negative ones. in this framework, the quality of these positive and negative annotations plays a critical role. supervised approaches typically gather these annotations from labeled natural language inference (nli) datasets @xcite -however, such sources are generally unavailable for most settings, and manually creating them is cost-prohibitive. as a result, unsupervised methods that solely rely on unlabeled sentences attract significantly more attention re-cently @xcite @xcite -they mostly develop methods to automatically obtain positive and negative samples to facilitate contrastive learning. a representative example is simcse @xcite , which leverages perturbed hidden states as the positive samples and in-batch sentences as negatives to perform contrastive learning. to differentiate between in-batch negatives and the annotated negatives, the latter are often termed "hard negatives", which have proven to be significantly advantageous in enhancing sentence embeddings @xcite . despite considerable advances in recent years, the performance of these unsupervised methods still falls short when compared to their supervised counterparts. moreover, the unavailability of largescale unlabeled data for the targeted domain often poses additional limitations to these approaches. to overcome these challenges, we introduce syncse, an unsupervised contrastive framework that trains sentence embeddings with synthesized data. concretely, we propose to prompt large language models (llms) such as @xcite to synthesize the samples needed for contrastive learning. this is inspired by recent successes of prompting large language models (llms) to perform various tasks @xcite @xcite , especially the superior performance of llms over crowd-workers on text annotation @xcite . we investigate two variants of syncse in this work that correspond to two practical scenarios: (1) syncse-partial, where large-scale unlabeled sentences are available and llms are prompted to produce positive and hard negative annotations, and (2) syncse-scratch, where large-scale unlabeled sentences are not available, prompting llms to generate sentences and their corresponding annotations from scratch. the latter represents a particularly challenging yet practical scenario where we aim to learn sentence embeddings without any data samples. we conduct comprehensive experiments on the standard semantic textual similarity (sts) benchmark, along with four reranking tasks and four domain adaptation tasks. our results demonstrate that both syncse-partial and syncse-scratch substantially outperform the unsupervised baselines in all cases -for example, syncse-partial and syncsescratch exceed the unsupervised simcse baseline by 5.37 and 4.18 absolute points respectively on sts. particularly, syncse-partial often equals its supervised counterpart on sts, marking the first instance of an unsupervised method matching supervised results on this benchmark. we release our synthesized datasets to facilitate further research to learn better sentence embeddings. ## related work prior approaches for sentence embedding fall into two main categories: (1) supervised learning with labeled sentences, and (2) unsupervised sentence embedding with unlabeled sentences. among these, works based on contrastive learning have proven to be the most effective. for unsupervised methods, simcse uses dropout masks to construct positive pairs for learning, while negative examples use in-batch negative examples. some works employ data augmentation techniques on input sentences, such as word repetition @xcite , case flipping @xcite , or a combination of multiple data augmentation strategies to offset the bias caused by mono-augmentation @xcite . promptbert @xcite uses prompts instead of the [cls] token to extract embeddings. however, these unsupervised methods significantly lag behind their supervised counterparts. supervised approaches usually derive positive and hard negative samples from labeled nli datasets @xcite @xcite , but these datasets are limited in quantity and domain. additionally, annotating a new nli dataset is costly, especially in fields that require trained annotators. @xcite trained a t5 (chung et al., 2022) model capable of producing positive and hard negative samples, while @xcite implemented a continuously updated model to modify prompts for generation. however, the performance of these algorithms is still constrained by the performance of generators, which need labeled nli data for training. differing from these methods, which necessitate training an additional model, @xcite proposed a rule-based algorithm capable of generating hard negative annotations. however, its diversity is limited to the prescribed rules. @xcite used chatgpt for dataset annotation. however, their exploration was limited to tasks with explicit answer labels such as "relevant" or "irrele-vant". they did not attempt to annotate datasets that required diverse responses. @xcite also propose to generate both annotations and unlabeled sentences, while they do not focus on the contrastive learning framework. ## discussion in this work, we propose syncse, a novel contrastive learning framework for learning sentence embeddings with synthetic data. we prompt llms to synthesize unlabeled sentences and their positive and hard negative examples. furthermore, by utilizing example and prompt pools, we can specify the genre and topic of generated sentences, thereby enhancing the quality of the synthetic dataset. experiments on both sentence similarity and reranking tasks demonstrate the effectiveness of syncse. the performance of syncse in this study strongly suggests the potential of synthetic datasets generated by the increasingly advanced llms of today. we envision that, through the effective use of prompting strategies with llms, synthetic datasets produced by these models could potentially serve as promising alternatives to real-world data across a wide range of tasks. ellen m voorhees and dawn m tice. 2000. building a question answering test collection. in proceedings of the 23rd annual international acm sigir conference on research and development in information retrieval, pages 200-207. bin wang, c.-c. jay kuo, and haizhou li. 2022a. just rank: rethinking evaluation with word and sentence similarities. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 6060-6077, dublin, ireland. association for computational linguistics. hao wang, yangguang li, zhen huang, yong dou, lingpeng kong, and jing shao. 2022b. sncse: contrastive learning for unsupervised sentence embedding with soft negative samples. arxiv preprint arxiv:2201.05979. tianduo wang and wei lu. 2022. differentiable data augmentation for contrastive sentence representation learning. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 7640-7653, abu dhabi, united arab emirates. association for computational linguistics.