Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning (2024)

Abstract: Recently, large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. Existing methods have explored utilizing LLMs as data annotators to generate synthesized data for training contrastive learning based sentence embedding models such as SimCSE. However, since contrastive learning models are sensitive to the quality of sentence pairs, the effectiveness of these methods is largely influenced by the content generated from LLMs, highlighting the need for more refined generation in the context of sentence representation learning. Building upon this premise, we propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus for training base sentence embedding models into three stages (i.e., sentence generation, sentence pair construction, in-batch training) and refines the generated content at these three distinct stages, ensuring only high-quality sentence pairs are utilized to train a base contrastive learning model. Our extensive experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results. Comprehensive analyses further underscore the potential of our framework in various application scenarios and achieving better sentence representation learning with LLMs.

## introduction as a fundamental task, sentence representation learning @xcite gurevych, 2019; @xcite aims to learn universal sentence embeddings that can benefit various downstream tasks, such as semantic similarity comparison @xcite @xcite @xcite @xcite , and information retrieval @xcite @xcite . recent advancements, particularly in contrastive learning-based methods such as sim-cse @xcite and its variants @xcite @xcite @xcite , have demonstrated to be the most efficient and effective ones. in contrastive learning, the quality of sentence pairs has a large impact on the overall performance @xcite . in particular, supervised contrastive learning methods trained on natural language inference (nli) datasets @xcite can outperform their unsupervised versions by a large margin @xcite . however, obtaining large amounts of high-quality sentence pairs can be costly in both time and resources, particularly considering that various application domains can only be better handled with domain-specific training data. the recent emergence of large language models (llms), such as the flan series @xcite , llama @xcite and chat-gpt @xcite , has brought a paradigm shift in natural language processing due to their impressive performance. consequently, there has been a growing interest in harnessing the power of llm for sentence representation learning. @xcite proposed to directly measure the semantic similarities of sentence pairs with llms for a base model to imitate. however, this imitation also places great demands on the llm's understanding of semantics and constrains its application in a wider range of llms. instead of utilizing llms for semantic similarity scoring, a recent line of work has explored leveraging llms to generate sentence pairs in the nli style with provided premises @xcite and demonstrates sota performance. despite their exciting results, these methods heavily depend on the quality of generated content from llms, leading a huge performance gap between different llms. moreover, concerns about the accuracy and quality of the generated content from llms remain unsolved and have drawn significant attention from the community @xcite , which are more pronounced with relatively less advanced llms. for example, in figure 1 , contradicting the first sentence requires a clear understanding towards its meaning and only chatgpt successfully produce its contradiction. when faced with the second provided sentence, both chatgpt and flan-t5 encounter difficulties due to the limited information provided. consequently, flan-t5 only repeats the premise while chatgpt entirely gives up the generation. in a task that is sensitive to the quality of sentences like sentence representation learning, there highlights a need for a more robust framework that can automatically refine the outputs of llms for better contrastive sentence representation learning. motivated by observations above, in this work, we propose multi-level contrastive sentence representation learning (multicsr), a novel three-stage framework that contrastively refines the generations of llms at distinct stages: sentence generation, sentence pair construction and in-batch training, while fine-tuning a contrastive learning model like simcse. concretely, while generating a sentence given a specific instruction like "write its entailment:", we also deploy the noisy variants of this instruction to identify obvious error that llms will make during generating the outputs following this instruction. in a contrastive sentence generation procedure, next-token prediction logits will be systematically deviated by comparing the logits derived from the original instructions with those from the noisy instructions. this process detects and avoids the obvious error tendencies of llms, and refines their generations to align more closely with the intended instruction, rather than providing an even opposite generation, such as the first generation of flan-t5 in figure 1 . despite the effectiveness of contrastive generation strategy, directly training a base sentence embedding model on the generated corpus gives poor results in our experiments, both because of the inevitably noisy generations, and the overlooked relations between sentences (i.e., sentence pairs), since a contrastive learning loss works essentially by modeling the relations between sentences while pulling the positive pairs closer and pushing the negative pairs apart. to take the relations between sentences into consideration, in sentence pair construction stage, we show that llms can be utilized to self-curate the set of their newly generated sentences by measuring the semantic similarities of sentence pairs, ensuring that only sentence pairs in highest quality are included into the final training stage. to further prevent the false-negative issue @xcite raised during the in-batch training, where randomly selected negative samples are indeed semantically similar to the original sentence, we utilize a pre-trained sentence representation model to provide the similarity mask and contrastively filter false negatives during training. in summary, our contributions include: (1) we propose a new and promising direction to improve sentence representation learning by refining the generated content of llms. (2) we for the first time decompose the process of prompting llms to generate a corpus for training base sentence embedding models into three stages (i.e., sentence generation, sentence pair construction, in-batch training), and integrate the idea of contrast into each stage for refinement. (3) we conduct extensive experiments on standard semantic textual similarity (sts) tasks @xcite @xcite @xcite @xcite and several transfer tasks @xcite with two representative llms (i.e., flan-t5 and chatgpt). we further perform a comprehensive analysis of the behavior of multicsr, demonstrating its effectiveness from various perspectives. we hope that, our proposed method provides additional insights into achieving high-quality sentence representation learning corpus by refining llms' generations. ## methodology in this section, we present multicsr, a framework designed to enhance the quality of generated content of llms. by contrastively refining their generations at distinct stages of training a contrastive learning method, multicsr ensures that only highquality sentence pairs are utilized in the final training stage, achieving a better sentence representation learning with llms. the whole workflow can be visualized as figure 2 . ## discussion: different strategies for self-curation and in-batch training in section 3, we have introduced in detail how we perform contrastive sentence pair construction with self-curation (denoted as stage 2) and contrastive in-batch training (as stage 3), and both of them are completed by measuring the similarities. it seems that we can use llms' and also pre-trained models' similarities for both stages. however, we have our intuition behind these designs. our design was firstly motivated by efficiency considerations. before introducing the selection for stage 2, we will explain why we choose to utilize pre-trained models in stage 3. there are multiple implementations for this stage. for example, we can also adopt a method similar to stage 2, having the llm compute similarity scores and provide masks. this allows the entire framework to rely only on the llms for data quality control. however, due to the slow inference speed of the llm, we must calculate and store the similarity scores for each potential pair in advance with llms, to not impact the training speed of the base model. in addition, since in-batch training data are randomly sampled, all sentence pairs should be considered. given a training corpus with m triplets and a batch size of n , the number of similarity scores that llms must generate in advance is 2m (m -1), but only 2m (n -1) of them will be used in stage 3. since m is usually 3 orders of magnitude larger than n , a significant amount of computational resources would be wasted. therefore, we choose to use a pre-trained model to generate masks dynamically in the stage 3 which will have little impact on the training speed. we also provide some cost analysis for stage 3 in appendix h. furthermore, if we try to utilize the similar strategy of stage 3 in stage 2, we would use a pre-trained sentence embedding model corresponding to the base model to generate similarity scores. this means that when we want to train or evaluate on a different backbone, we would need to re-run the same self-curation process on the data generated in stage 1 again. this also leads to a waste of computational resources. besides, using an llm to evaluate a sentence triplet could be completed simultaneously with stage 1. when we choose to use llms in stage 2 and a pre-trained model in stage 3, stage 1&2 and stage 3 will be responsible for the dataset construction process and during-training process, respectively. the high-quality training corpus produced after stage 1&2 can be used in training or evaluation of various base models in similar settings in stage 3. we further include some related case studies in appendix i. moreover, using a pre-trained sentence embedding model in stage 2 will contradict our framework's objective in some scenarios. as previously mentioned, we would choose a pre-trained model corresponding to different base models to compute the similarity scores. when the checkpoints of a certain pre-trained model have not been released, we can even train such models using the highquality training corpus, with the obtained model used to provide masks in stage 3. however, this behavior assumes that we already have a high-quality training corpus, which would be a significant challenge with only the corpus generated from stage 1, as demonstrated in the results of our ablation study's multicsr w/o stage 2&3. and stage 2 is precisely designed as a necessary stage to improve the quality of training corpus. hence, attempting to introduce a pre-trained model in stage 2 would require either a corresponding released checkpoint or high-quality training data. in more extreme scenarios, such as the low-resource scenarios, there is no pre-trained sentence embedding model available, and the sentences generated with only contrastive generation strategy is not sufficient to form a highquality training corpus. to demonstrate this, we further apply our mul-ticsr to low-resource languages. we conduct experiments on a rather small language: tagalog (tl). for training corpus, we leverage ted2020 from @xcite with 1167 translated sentence-pairs, and use only the sentences in tl. the evaluation is performed by finding the most similar sentence inside a corpus using cosine similarity, with 1000 test pairs from laser @xcite . the results are shown in method accuracy% simcse bert 72.6 + stage 1 77.9 + stage 1 & 2 84.2 apart from these concerns, when only taking the effectiveness into consideration, the similarity scores provided directly by the llms are not as accurate as those from a pre-trained sentence embedding model, as shown in section 2.1 and ## conclusion in this paper, we introduce multicsr, a novel framework to contrastively refine the generation of llms at distinct stages of training a contrastive learning method, ensuring only high-quality and suitable sentence pairs are utilized during the model training. experiments on standard sts tasks and several downstream tasks demonstrate the effectiveness of multicsr. extensive analysis shows the potential of our work and we hope to inspire future work in achieving better sentence representation learning with llms.