L lama2 V ec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024)

Abstract: Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs’ strong capability on semantic understanding. However, the LLMs are learned by auto-regression, whose working mechanism is completely different from representing whole text as one discriminative embedding. Thus, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called Llama2Vec , which performs unsupervised adaptation of LLM for its dense retrieval application. Llama2Vec consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the LLM is prompted to reconstruct the input sentence and predict the next sentence based on its text embeddings. Llama2Vec is simple, lightweight, but highly effective. It is used to adapt LLaMA-2-7B on the Wikipedia corpus. With a moderate steps of adaptation, it substantially improves the model’s fine-tuned performances on a variety of dense retrieval benchmarks. Notably, it results in the new state-of-the-art performances on popular benchmarks, such as passage and document retrieval on MSMARCO, and zero-shot retrieval on BEIR. The model and source code will be made publicly available to facilitate the future research. Our model is available at https://github.com/FlagOpen/FlagEmbedding.

## introduction dense retrieval is a new paradigm of ir empowered by deep neural networks. it represents query and document as embeddings within the same latent space, where the semantic relationship between query and document can be reflected by their embedding similarity. nowadays, dense retrieval has been a critical component in many real-world scenarios, like open-domain qa, fact verification, and retrieval-augmented generation @xcite @xcite @xcite . the capacity of text encoder is a critical factor of dense retrieval. in the past few years, the pretrained language models, e.g., bert @xcite , roberta @xcite , t5 @xcite , were widely applied to generate high-quality representations for query and document, which substantially contributed to the accuracy and generalizability of dense retrieval. besides, it was also found that dense retrieval's performance can further benefit from the continual growth of model size and training scale @xcite @xcite . recently, large language models (llms) have emerged as a unified foundation for general nlp tasks @xcite @xcite . given the llms' superior capability on semantic understanding, it will be promising to take advantage of such powerful models as new backbones for dense retrieval. with this consideration, there have been pioneering efforts where llms are trained to generate discriminative text embeddings to facilitate the retrieval tasks in many different scenarios @xcite @xcite . despite the preliminary progress, it remains an open problem to fully unleash the llm's underlying potential for dense retrieval. particularly, the typical llms are pre-trained by text generation tasks, especially auto-regression, where the text embeddings are learned to predict the next tokens. consequently, the llms' text embeddings will focus on representing the local (i.e. the next step) semantic of the context. however, dense retrieval calls for text embeddings to represent the global norwegian forest cat is a breed cat originating in northern europe. this natural breed is adapted to a very cold climate. norwegian forest cat is a breed cat originating in northern europe. the original sentence: <\@xmath0: <\@xmath1. this natural breed is adapted to a very cold climate. ## related works in this section, the related works are discussed from two perspectives: the background of dense retrieval, and the previous efforts on leveraging the llms for dense retrieval applications. ## conclusion in this paper, we present llama2vec to enhance the llm-based dense retrieval via unsupervised adaptation. llama2vec is composed of two pretext tasks, ebae and ebar, where the llm is prompted to reconstruct the input sentence and predict the following sentence purely with its text embeddings. on top of the unsupervised adaptation, the llm's text embedding can be transformed into a strong representation of global context. by using suitable prompts, it can flexibly support the semantic matching of different correlation and paraphrasing relationships. the effectiveness of llama2vec is verified by comprehensive experiments, where the adapted llm achieves new state-of-the-art performances in both supervised and zero-shot evaluations, indicating the substantial improvements on both accuracy and generalization capability of the retrieval model.