SKICSE : Sentence Knowable Information Prompted by LLM s Improves Contrastive Sentence Embeddings (2024)

Abstract: Contrastive learning, which utilizes positive pairs and in-batch negatives to optimize the loss objective, has been proven to be an effective method for learning sentence embeddings. However, we argue that the previous methods of constructing positive pairs only through dropout perturbation or entailment relation are limited. Since there is more sentence knowable information (SKI) to be mined, such as sentence external knowledge, semantic analysis, and grammatical description. In this work, we first hand-craft a simple and effective prompt template that is able to obtain the knowable information of input sentences from LLMs (e.g., LLaMA). Then we combine the original sentence and its knowable information to form a positive pair for contrastive learning. We evaluate our method on standard semantic textual similarity (STS) tasks. Experimental results show that our unsupervised and supervised models using \text{BERT}_\text{base} achieve an average of 78.65% and 82.45% Spearman’s correlation respectively, a 2.40% and 0.88% improvement compared to SimCSE. Our model outperforms the previous state-of-the-art model PromptBERT in both unsupervised and supervised settings and specifically yields a new state-of-the-art performance in supervised setting.

## introduction learning sentence embeddings is a fundamental task of natural language processing (nlp), which embeds sentences of natural language text into high-dimensional dense vectors containing rich semantic information. high-quality sentence representations find applications across various practical domains, including question answering systems, translation systems, recommendation systems, and retrieval systems. in recent years, transformer-based @xcite pre-trained language models such as bert @xcite have achieved remarkable results in nlp. however, @xcite found that the performance of bert without fine-tuning is even inferior to glove @xcite on sts tasks @xcite @xcite @xcite @xcite , and proposed to train sbert through siamese network structures and supervised data such as nli @xcite , sts-b, and mrpc @xcite . @xcite analyzed the dilemma of native bert from the perspective of anisotropic sentence embedding distribution, and proposed the corresponding improved method bert-flow. @xcite proposed simcse, a simple contrastive sentence embedding framework, which improves the sentence vector space in terms of alignment and uniformity @xcite , and has made great progress on sts tasks. witnessing the notable success of simcse on sts tasks, many variations @xcite @xcite @xcite of simcse have been introduced by researchers. although many of them have novel ideas and methods, few of them can adapt to both unsupervised and supervised scenarios. another prevalent issue among them is the way to construct positive pairs, which often relies solely on minimal data augmentation foot_0 (mda). we think that more knowable information of sentences can be mined to construct positive pairs to enhance the knowledge, semantics and grammar of sentence representations. recently, llms such as chatgpt @xcite and llama @xcite have attracted widespread attention. by leveraging the comprehension and generation capabilities of llms, coupled with our effective handcrafted prompt template, we are able to obtain knowable information about input sentences, as shown in figure 1 . we further use input sentences and generated sentences as positive pairs to compute the contrastive loss, and make a trade-off with the original loss through the weighting coefficient. our main contributions can be summarized as the following two points: ## conclusion in this paper, we propose a novel concept called sentence knowable information (ski). it is an excellent complement to positive pairs constructed by minimal data augmentation and entailment relation. owing to the powerful generation capabilities of llms and our effectively handcrafted prompt template, we mine ski whose main content is external knowledge, semantic analysis, and grammatical description. ski is injected into the model through an additional standard contrastive learning objective to better optimize the sentence vector space. experimental results on sts tasks show that our method can achieve better performance than almost all sentence embedding strong baselines.