P rompt R eps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024)

Abstract: Utilizing large language models (LLMs) for zero-shot document ranking is done in one of two ways: (1) prompt-based re-ranking methods, which require no further training but are only feasible for re-ranking a handful of candidate documents due to computational costs; and (2) unsupervised contrastive trained dense retrieval methods, which can retrieve relevant documents from the entire corpus but require a large amount of paired text data for contrastive training.In this paper, we propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. Our method only requires prompts to guide an LLM to generate query and document representations for effective document retrieval. Specifically, we prompt the LLMs to represent a given text using a single word, and then use the last token’s hidden states and the corresponding logits associated with the prediction of the next token to construct a hybrid document retrieval system. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations given by the LLM.Our experimental evaluation on the MSMARCO, TREC deep learning and BEIR zero-shot document retrieval datasets illustrates that this simple prompt-based LLM retrieval method can achieve a similar or higher retrieval effectiveness than state-of-the-art LLM embedding methods that are trained with large amounts of unsupervised data, especially when using a larger LLM.

## introduction large language models (llms) such as gpt4 and llama, which are pretrained on massive corpora and finetuned to follow user instructions, have strong zero-shot natural language understanding capabilities @xcite . via prompting, llms excel in various text generation tasks such as question answering, writing assistance, and conversational agent @xcite . inspired by the success of llms on natural language understanding tasks, research has explored the potential of using llms to perform unsupervised document ranking. one line of work focuses on directly prompting llms to infer document relevance to a given query @xcite @xcite @xcite @xcite . for instance, rankgpt @xcite casts document re-ranking as a permutation generation task, prompting llms to generate re-ordered document identifiers according to the document's relevance to the query. these methods leverage llms for document ranking in a complete zero-shot setting where no further training is required. however, these methods can only serve as a second-stage reranker on a handful of candidate documents. this is because each prompt requires one full llm inference: for example, in the case of a corpus with 1m documents, a pointwise approach would require the construction of 1m prompts and thus the execution of 1m (costly) llm inferences -making it unfeasible for an online search engine. another line of research leverages llms as a text embedding model for dense document retrieval @xcite @xcite . for example, e5mistral @xcite employs llms to create synthetic datasets of query-document pairs. these paired text data are then used to perform unsupervised contrastive training for a mistral llmbased dense retriever. since the queries and documents are encoded with llms separately; i.e., using a bi-encoder architecture, these methods could serve as a first-stage document retriever. however, all existing llm-based retrievers require an unsupervised contrastive training step to transform a generative llm into a text-embedding model. even with parameter-efficient training techniques such as lora @xcite , this extra training is still very expensive. for example, the contrastive training of e5-mistral using a large batch size (2048) and lora took ≈ 18 hours on 32 v100 gpus @xcite . in this work, we propose a new zero-shot llm-based document retrieval method called promptreps. we demonstrate that llms can be directly prompted to produce query and document embeddings, which can serve as effective text representations for neural retrieval systems. specifically, we prompt an llm by asking it to use a single word to represent a query or a document. then, we extract the last layer's hidden state of the last token in the prompt as the dense representation of the input text. simultaneously, we utilize the logits associated with predicting the subsequent token to form a sparse representation. as illustrated in figure 1 , through a single forward pass, we generate text representations for a document that can be indexed for dense, sparse, or hybrid search architectures. we also explore alternative representations in addition to the core idea in this paper, where we generate multiple words, and use multiple embeddings to represent an item (figures 3 and 4 ). our empirical evaluation on multiple datasets show that promptreps can achieve a similar or higher zero-shot retrieval effectiveness than previous trained llm-based embedding methods, especially when a large llm is utilized. of key importance is that our method is the first llmbased method that can effectively perform full corpus retrieval while at the same time not requiring contrastive training, demonstrating that prompt engineering for generative llms is capable of generating robust representations for retrieval. code for fully reproducing our results is available at https://github.com/ielab/ promptreps . ## promptreps previous work that leverages llms for document ranking are limited to document re-ranking tasks with prompts or rely on contrastive learning to transform a generative llm into an embedding model for document retrieval. unlike these previous works, here we aim to directly prompt llms to generate both dense embedding representations and sparse bag-of-words representations for docu-ment retrieval without any form of extra training effort. to achieve this, we devise the prompt as illustrated in figure 1 as the input text for llms, @xmath0. when using this prompt for text generation, the language model needs to find a single word in its token vocabulary that can best represent the given passage to generate. however, since there could be multiple words to represent the passage, there might be multiple tokens in the vocabulary that have a high probability of being sampled by the language model. such a distribution over the vocabulary, which is often refers to as "logits", could provide a good representation of the given passage. in addition, since the logits are computed by the last layer hidden state foot_2 of the last input token (' " '), which is a dense vector embedding, it could also serve as a dense representation of the passage. based on the above intuition, we develop a sparse + dense hybrid document retrieval system by utilizing both the next token logits and the last layer hidden states outputted by the llm with our designed prompt. specifically, during the document indexing phase, we pass all the documents (one at the time) with our prompt into the llm to get output hidden states and logits. to build a sparse retrieval pipeline with logits, we first need to sparsify the logits representation to be able to perform efficient sparse retrieval. this is because logits originally had values for all tokens in the vocabulary, essentially forming dense vectors with dimensions equal to the vocabulary size. to sparsify the logit representations for sparse retrieval, we perform the following steps: with these adjustments, the logits representations of documents are heavily sparsified, allowing for efficient sparse retrieval with an inverted index. for dense retrieval, we directly use the hidden states as the embeddings of the documents. for indexing these embeddings, we simply normalize all the embeddings and add them into an approximate nearest search (ann) vector index. in appendix a, we provide example python code of generating dense and sparse representations with promptreps. at query time, we process the queries exactly the same as the documents, with the only exception being that the term "passage" in the prompt is replaced with "query". 5 the dense representation of the query is utilized for semantic search via the ann index, while the sparse representation of the query is employed for exact term matching via the inverted index. following previous work @xcite , we compute the final document scores by applying min-max normalization to both dense and sparse document scores. these normalized scores are then linearly interpolated with equal weights to produce the final document scores. we do not explicitly tune the weight because our setting is zero-shot retrieval, and we wanted to maintain the "zero-shot" nature of our approach. nevertheless, in appendix b, we explore the impact of the different weight settings. @xcite . these datasets encompass various ir tasks, providing a heterogeneous evaluation environment. for msmarco we report mrr@10 and for trec deep learning and bier we report ndcg@10 scores, the commonly employed evaluation measure for these datasets. baselines: we compare promptreps with strong unsupervised first-stage retrievers including bm25, a classic term frequency-based sparse retrieval method, and e5-pt large @xcite , a state-of-the-art bert large-based dense embedding method trained on 1.3b carefully crafted unsupervised text pairs. llm2vec @xcite , a llama3-8b-instruct llmbased dense embedding method trained with bidirectional attention, masked next token prediction, and simcse @xcite on the wikipedia corpus. in addition, we also report state-of-theart supervised contrastive, fine-tuned bert-based sparse retriever splade++ @xcite and dense retriever dragon+ @xcite . we note that these methods are trained with lots of supervised training data and knowledge distillation from teacher models, thus it is unfair to compare with our method and other unsupervised baselines. however, we think it is useful to compare with supervised methods to understand the gap between supervised and unsupervised methods. ## zero-shot results we start by showing our overall results on the beir dataset, which we treated as test set; we then analyse choices in instantiation of promptreps, including different variations in the prompt using the msmarco and trec deep learning datasets, which we used as development datasets to inform the choices we made to run promptreps on beir. ## supervised results we have demonstrated the strong zero-shot effectiveness of promptreps. now we explore the question: can promptreps serve as a better initialization for llm-based embedding models in downstream contrastive supervised fine-tuning? to address this question, we conduct supervised fine-tuning experiments using msmarco. specifically, we follow the repllama training recipe @xcite to fine-tune the llama-3-8b-instruct base model with infonce loss and hard negative passages mined by a bm25 and dense retrieval hybrid system. the detailed training hyperparameters are listed in appendix d. for the re-pllama baseline, we adhere to the original implementation, which appends the prefixes "query: " and "passage: " to the query and document text, respectively, and adds the end-of-sentence token at the end of the text. the output embedding of this token is then used to represent the text. for promptreps, we use our proposed prompt (#6 in we train both repllama and promptreps for 1 epoch using the full msmarco training data, which contains 490k training examples. in addition to the full training, we also explore a low-resource training setting, where we sample 1k examples from the entire msmarco dataset. we then split the 1k examples into a training and a validation set with a 9:1 ratio. we monitor the validation loss after each training epoch and stop the training, selecting the best checkpoint if no lower validation loss is observed for three consecutive epochs. the results are shown in ## alternative representation and scoring in the previous sections, we only considered using the representations (dense and sparse) yielded from the last token in the prompt for document retrieval. these representations, in the context of generative llms, are responsible for predicting the first generated token. we define this setting as first-token single-representation (ftsr). we have demonstrated that this simple way of generating representations is effective for document retrieval; however, these representations might be sub-optimal. for example, llms use sub-word tokenization algorithms such as sentencepiece @xcite . this tokenization might split a word into sub-words, meaning that the first generated token might just be a sub-word. using the representation of the whole word might be a better representation than the first token representation. additionally, previous works in multivector dense retrieval such as colbert @xcite demonstrated that using multiple representations could be beneficial for document retrieval. how can we use promptreps to also generate single-word representations or multiple representations that can potentially enhance the retrieval effectiveness? in this section, we explore these alternative representations. first-word single-representation (fwsr) and multi-token single-representation (mtsr). instead of using the representations of the first generated token, these two methods let the llm finish the generation foot_10 of the whole word or multiple words, controlled by the given prompt ("use one word" or "use three words"), as illustrated in figure 3 . the end of generation is detected by the token ' " '. we then pool all the representations of the generated tokens to form a single dense and sparse representation for the input text. for the dense representation we use mean pooling and for the sparse representation we use max pooling. once representations are obtained, the scoring is the same as ftsr. multi-token multi-representation (mtmr) and multi-word multi-representation (mwmr). instead of using a single representation for retrieval, these two methods prompt the llm to generate multiple words and then index each generated representation separately. the difference between the two is that mtmr keeps all the token representations in the index, while mwmr first groups tokens into words by using space, and then creates a single representation for each word by using max pooling for sparse representations and mean pooling for dense representations. during retrieval, we follow the colbert scoring method where the relevance score of a document is computed by the sum of the maximum similarity of each query representation against each document representation (figure 4 ). hybrid retrieval results are shown in figure 5 , and full dense and sparse retrieval results in appendix e. results show that all the explored methods are able to perform document retrieval. the ftsr and mtsr generally perform the best. however, we note that mtsr requires more token generation steps and thus has higher query latency. the fwsr performs the worst, suggesting that subword representations hurt the retrieval performance for single-word generation prompts. on the other hand, multi-representation methods with colbert scoring methods do not seem beneficial. thus, we conclude that the simplest ftsr is sufficient to represent the input text for document retrieval. ## conclusion we introduced promptreps, a simple yet effective method that prompts llms to generate dense and sparse representations for zero-shot document retrieval without any further training. we show that modern llms are effective text encoders by themselves, and prompt engineering is sufficient to stimulate their text encoding ability. for future works, techniques like few-shot incontext learning @xcite , chain-ofthought prompting @xcite , and autoprompt optimization methods @xcite , which have proven to be effective in text-generation tasks, could potentially be leveraged here to enhance embedding generation. moreover, it has been shown that the instructionfollowing ability of llms could be transferred to embedding models with synthetic instruction fine-tuning data @xcite . in our work, we always keep the instruction prompt consistent across different ir tasks, which could be sub-optimal. it is interesting to investigate how to customize instructions for promptreps to generate embeddings specific to different domains, tasks, or even to multi-lingual and cross-lingual ir settings. finally, our prompting method could be seen as a simple approach to obtaining a better initialization of llm-based embedding models and all the previous contrastive pre-training with paired text data and synthetically generated data could be applied on top of our method and could potentially yield improved llm-based embedding models.