Meta-Task Prompting Elicits Embeddings from Large Language Models (2024)

Abstract: We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks are versatile embeddings that yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.

## introduction the advent of large language models (llms) such as gpt-3 @xcite and llama @xcite has marked a significant milestone in the field of natural language processing (nlp), introducing promising unsupervised methods for various nlp tasks by leveraging task-related instructions or prompts @xcite @xcite . these tasks also include the generation of sentence embeddings, which aims to produce sentence representations that can be applied across a wide range of scenarios. they have been applied to intrinsic tasks like semantic textual similarity (sts) @xcite , to downstream tasks including information retrieval @xcite , and to sentiment classification @xcite and beyond. by employ-figure 1 : the highest decoding probabilities are largely allocated to stop words that carry little useful information when conducting a meaning compression prompting, even if employing a constraint of "in one word" following @xcite . although the general semantic, movie, is contained, other aspects of this sentence are missing, like sentiments. initial efforts in this domain, as highlighted by works like @xcite @xcite , have focused on unsupervised techniques that extract sentence representations directly from llms. these methods typically involve using fill-in-the-blanks prompts, such as this sentence: "[text]" means in one word:" @xcite , to embed a sentence into a single token representation by using the output hidden state of the last token as the sentence's embedding. while they perform well, these approaches also reveal the inherent challenges of this task: embeddings may be overly simplistic or misaligned with the intended semantic nuances of the sentences. in a pilot experiment illustrated in figure 1 , we demonstrate that a previous prompt-based method @xcite can struggle to capture a sentence's meaning, especially when the usage of the sentence is associated with multiple aspects. when probing the probability distribution for the next token during decoding, which reflects the embedding quality of the last token foot_0 , the highest probabilities are mostly distributed to frequent stop words. although the general movie topic appears, other meaningful aspects like sentiments are missing. a straightforward solution to mitigate this issue is to provide llms with task-specific instructions. this approach involves instructing the model with prompts explicitly designed for a particular task, thereby tailoring the embeddings to better suit the specific requirements of that task. however, considering the wide range of distinct tasks that an embedding may be used for @xcite @xcite , this would be impractical. furthermore, while task-specific embeddings are effective for their corresponding tasks, they may fail to generalize well across different tasks. inspired by the principles of the usage-based theory of language acquisition @xcite , which asserts that the essence of meaning is rooted in the practical utilization of language, our approach aims to generate broad embeddings through the use of meta-task prompting, inspired by metatask prompted training @xcite and hyper-prompt @xcite techniques. by defining a suite of meta-tasks, each tailored to a distinct application context, metaeol prompts llms to consider multiple representational tokens from a variety of perspectives. this multifaceted approach enables the extraction of more diverse and nuanced contextualized token embeddings that collectively form a comprehensive sentence embedding. extensive experiments empirically show that: (i) simply averaging embeddings from different metatasks without any training leads to general embeddings that are competitive to contrastive-trained models on sts tasks and can achieve the best av-erage result on several downstream tasks. (ii) incrementally integrating more meta-tasks (ranging from one to four) yields consistent improvements across sts tasks, showcasing high generalities, and highlighting the significant impact of meta-task integration on overall performance. (iii) the final layer is not always the most effective for sts tasks and with a simple proportional layer selection strategy, we achieve the best results with a 70b model, which points to a potential scaling law. large language models for text representation. recent studies have explored the use of llms for enhancing text embeddings through data augmentation techniques @xcite . notably, sentence-t5 @xcite and gtr @xcite employ contrastive learning on models with billions of parameters. more recently, research has focused on converting an llm directly into a text encoder without any training. @xcite represents sentences through the distribution of possible text continuations, comparing the distributional similarity between sentences. this method, although effective, necessitates the generation of 20 trajectories, each up to 20 tokens in length, making it computationally intensive. @xcite in-figure 2 : the workflow of our method (metaeol). we use the prompt in appendix a.1 to prompt chatgpt-4 to generate templates. each input sentence will be decorated with multiple task-specific templates, indicating its various intended usage scenarios. the resulting multiple prompts will be fed to llms. then, multiple task-specific embeddings will be extracted. the final sentence embedding is obtained by averaging the task-specific embeddings. ## method in this section, we begin by reviewing two kinds of previous prompting methods for deriving sentence representation from masked and causal language models, respectively (section 3.1). subsequently, we describe our proposed method, i.e., meta-task prompting with explicit one-word limitation (metaeol) in detail (section 3.2). lastly, we describe meta-tasks involved in this paper (section 3.3). ## analysis in this section, we thoroughly analyze metaeol using the llama2-7b model. ## conclusion in this paper, we introduce metaeol, a new approach for deriving high-quality sentence embeddings from llms without requiring any training. by leveraging a diverse set of meta-task prompts, metaeol effectively captures multiple representations of sentences from distinct perspectives. we show simply averaging these meta-task derived embeddings leads to generalized general-purpose embeddings, which work remarkably well across sts datasets and transfer learning tasks.