Scaling Sentence Embeddings with Large Language Models (2024)

Abstract: Large Language Models (LLMs) have recently gained significant interest due to their impressive results in various natural language tasks. However, their application to sentence embeddings is still under active research. In this work, we introduce PromptEOL, a simple and efficient method designed to enhance LLM performance on sentence embeddings with a one-word limitation. We further integrate PromptEOL with in-context learning and alignment to leverage LLMs in two settings: without fine-tuning and with fine-tuning. Our extensive experiments show that PromptEOL enables LLMs to generate superior sentence embeddings without fine-tuning, outperforming contrastive learning methods. Additionally, with fine-tuning, a 2.7B parameter model using PromptEOL surpasses the performance of a 4.8B parameter model from previous methods. We also analyze how scaling model parameters, from 125 million to 66 billion, impacts sentence embedding performance.

## introduction sentence embeddings is a fundamental problem in natural language processing, requiring language models to project sentences into a vector space based on their semantics. current methods based on contrastive learning, such as simcse @xcite , have successfully leveraged pretrained language models to generate high-quality embeddings. a significant amount of research has been devoted to refining the contrastive learning framework in order to further improve sentence embeddings @xcite @xcite . recently, llms, such as gpt-3 @xcite and llama @xcite , have demonstrated significant potential on various natural language processing tasks such as translation, question answering, and text classification. current research has also explored the application of llms for data augmentation in sentence embeddings. by generating better sentence pairs for contrastive learning, llms can help alleviate the scarcity of labeled data @xcite . however, directly utilizing llms to generate sentence embeddings presents two primary challenges. firstly, llms, as autoregressive models, produce text instead of vectors, which necessitates vectorizing the output. secondly, it is crucial to determine an effective approach for incorporating the capabilities of in-context learning into sentence embeddings. in this work, we aim to investigate the capabilities of current llms for sentence embeddings, facilitated by the availability of open-source llms @xcite . we address the following research questions: 1) how can llms be used to represent sentence embeddings, and does prompt engineering @xcite help? 2) can in-context learning @xcite enhance the quality of sentence embeddings? 3) does the scaling up the model parameters still work when the number of parameters exceeds billions? to address these questions, we conduct a systematic study by evaluating llama @xcite and opt @xcite on both semantic textual similarity (sts) tasks and transfer tasks. following jiang et al., we utilize a prompt such as this sentence: " [text] " means to enable llms to generate sentence embeddings, where [text] serves as the input slot. this method outperforms traditional representation methods, such as averaging output tokens to represent sentences. considering the causal architecture and pretraining tasks of llms compared to bert, we propose a method called prompteol to refine the prompt to generate better representations by instructing llms to encapsulate as much semantic information of the sentences as possible within the target token. inspired by @xcite , which uses definition sentences from a word dictionary to learn sentence embeddings, we find that performance can be further improved by adding definition sentences and corresponding words as examples to perform in-context learning. to mitigate the gap between examples and input sentences, we also use sentences from the sts-b @xcite training set as examples by instructing chatgpt to generate a single word to represent the meaning of sentences. by evaluating the demonstration examples based on the sts-b development set, llms can outperform previous contrastive learning-based sentence models, which were finetuned on unsupervised data. we scale up the parameters of llms in two settings: without and with fine-tuning. for the settings without fine-tuning, we find that transitioning from millions to billions of parameters results in improvements on sts tasks. however, continued scaling may not yield further improvements. one explanation corresponds to anisotropy in embeddings. we note that larger llms have greater anisotropy, which may limit their performance. for the settings with fine-tuning, since anisotropy can be alleviated by contrastive learning @xcite , llms with tens of billions of parameters exhibit strong performance. our main contributions are as follows: large language models llms @xcite @xcite ) recently show impressive performance on various natural language process, benefiting from their large parameter sizes compared to previous pretrained language models. llms can efficiently learn a new task with incontext learning by using training data as demonstrations @xcite . without any gradient updates, llms with in-context learning can solve challenging tasks like multitask language understanding @xcite , commonsense reasoning @xcite , and math problems @xcite . this performance can be further improved by scaling up language models @xcite . ## methodology in this section, we first discuss current sentence embeddings methods with llms, and then introduce a new prompt-based method with explicit one word limitation (prompteol) for llms in section 3.1. based on this method, we describe methods without fine-tuning in section 3.2 and with fine-tuning in section 3.3, respectively. ## conclusion in this paper, we focus on exploiting llms to improve sentence embeddings. to achieve this, we propose a new sentence embeddings method called prompteol, which adapts previous promptbased methods to autoregression models. furthermore, we leverage in-context learning to generate superior sentence embeddings by utilizing chat-gpt and the oxford dictionary to create sentence embeddings demonstrations. it demonstrates incontext learning allows llms to achieve performance comparable to current contrastive learning methods. with our prompt-based method, we also discover that further fine-tuning of llms can achieve the state-of-the-art performance using only efficient fine-tuning methods.