Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024)

Abstract: Pre-trained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in-domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving state-of-the-art performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations—such as parameter sizes, pre-training duration, and alignment processes—on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in-domain accuracy, data efficiency, zero-shot generalization, lengthy retrieval, instruction-based retrieval, and multi-task learning. We evaluate over 15 different backbone LLMs and non-LLMs. Our findings reveal that larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero-shot generalization, lengthy retrieval, instruction-based retrieval, and multi-task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.

## introduction dense retrieval, a novel paradigm in information retrieval (ir), has emerged with the advancement of deep neural networks. unlike traditional ir methods, dense retrieval encodes both queries and documents as embeddings within a shared latent space, capturing their semantic relationships through embedding similarities. dense retrieval †. equal contribution * . corresponding author models have become the predominant choice in recent neural retrieval approaches and are widely applied in various downstream tasks such as web search, question answering, and sentence similarity @xcite @xcite . in the past few years, dense retrieval models intensively adopted pre-trained language models, such as bert @xcite and t5 @xcite , as their backbone encoders. these models excel in identifying semantic similarities between queries and documents. however, they still face significant challenges in becoming versatile enough to handle a wide range of retrieval tasks @xcite . their in-domain retrieval accuracy is often constrained by the capacity of their backbone encoders, such as the number of parameters @xcite . additionally, dense retrieval models typically struggle to generalize to unseen data, necessitating fine-tuning with a large amount of labeled data to perform well in the target domain. finally, achieving versatility in dense retrieval models requires training on multiple retrieval tasks simultaneously, which demands sufficient capacity from the backbone encoder @xcite . recently large language models (llms) have been prompted or fine-tuned as dense retrieval models and achieved improved performance across a wide range of retrieval tasks, thanks to their superior capability for semantic understanding and rich world knowledge @xcite @xcite . these models vary in parameters from 2 billion to 56 billion, with pre-training sufficiency ranging from hundreds of billions to tens of trillions of tokens, and include both base models and human preference aligned chat models. despite the common understanding that larger models generally yield better performance @xcite , the specific benefits of vary-ing parameter numbers, pre-training sufficiency, and alignment processes of backbone llms for different retrieval tasks still remain unclear. in this study, we focus on the following two research questions: 1) for different retrieval tasks, what specific benefits can llms offer compared to non-llms as the backbone encoders? 2) for llms with varying configurations (i.e., different parameter numbers, pre-training sufficiency and alignment processes), what contributes more to different retrieval tasks as the backbone encoder. we conduct comprehensive empirical investigation across a wide range of retrieval tasks, assessing various critical retrieval capabilities: in-domain accuracy, data efficiency, zero-shot generalization, lengthy retrieval generalization, instruction-based retrieval, and multi-task learning. our study explore over 15 different backbone llms and non-llms, with parameter numbers ranging from 0.1 billion to 32 billion and varying pre-training sufficiency, including both base llms and chat llms. previous dense retrieval models have demonstrated inferior in-domain accuracy due to the limited capacity of their backbone encoders @xcite . we employ ms marco @xcite , one of the largest web search datasets, to train and evaluate the in-domain accuracy of dense retrieval models with different backbone encoders. our results indicate that both increasing the model size and enhancing pre-training sufficiency can consistently improve the upper limit of in-domain accuracy. notably, we discover that both base llms and human-preference-aligned chat llms show comparable potential as backbone encoders for dense retrieval tasks. by training with different proportions of ms marco, we explore data efficiency and find that scaling up model size facilitates convergence, allowing llms to converge swiftly even with limited annotated data, without the need for intricate multi-stage training processes. we examine generalization ability from three perspectives: zero-shot generalization, lengthy retrieval generalization, and instruction-based retrieval generalization. first, we evaluate zero-shot generalization using beir benchmark @xcite . our findings indicate that model size is the most crucial factor for zero-shot retrieval generalization. moreover, traditional dense retrieval models are limited by the maximum input length used during pre-training and retrieval train-ing. we investigate whether llm-based retrievers, pre-trained with longer context windows, can effectively generalize to lengthy retrieval tasks even when trained with shorter passage lengths. finally, dense retrieval models often lack flexibility in handling varying retrieval intents @xcite . we explore the capability of different models to incorporate instructions during retrieval, discovering that training with instruction benefits llms but not non-llms, and that human-preference alignment does not significantly improve performance compared to base llms. we further explore the multi-task learning capabilities of models with different backbone encoders, essential for developing versatile retrievers @xcite . we adopt five distinct retrieval tasks, where interference exists due to varying retrieval intents. our findings reveal that although all models experience performance decreases with multi-task training compared to training on each single-task, increasing model size consistently mitigates this gap. to summarize, we make the following contributions: 1) we conduct a thorough experimental study using more than 15 backbone encoders with different configurations for dense retrieval across six distinct retrieval tasks. 2) we demonstrate that llm-based retrievers consistently enhance performance across all retrieval tasks compared to non-llm-based retrievers. 3) we investigate how different configurations of backbone llms impact each retrieval task, focusing on distinct retrieval capabilities. ## related work the related works are reviewed from two aspects: dense retrieval, llm-based retriever. first of all, in the realm of neural retrievers, dense retrieval models have consistently demonstrated superior performance over traditional sparse models like bm25 across a wide array of retrieval tasks @xcite @xcite . a critical factor contributing to the success of dense retrieval models is the utilization of powerful pre-trained language models as their initialization. over the past few years, pre-trained language models such as bert @xcite and t5 @xcite have been intensively used as backbone encoders for dense retrieval. for instance, gtr @xcite highlights the in-domain accuracy and generalization capabilities of t5-based dense retrieval models, with model parameters reaching up to 4.8 billion. @xcite explores scaling laws for dense retrieval models but restricts their study to bert backbones with up to 110 million parameters and only explores the in-domain situation. currently, state-ofthe-art dense retrievers employ models with more than 7 billion parameters or more as backbones. @xcite discuss large-scale unsupervised text embedding pre-training, observing consistent performance improvements when scaling up gpt-based dense retrieval model sizes from 300 million to 175 billion parameters. additionally, recent studies such as @xcite have shown that fine-tuning directly with labeled data can achieve strong performance. our study focuses on fine-tuning directly using labeled data while comparing various backbone encoders. large language models (llms) have recently demonstrated significant potential as backbone encoders for dense retrieval, attributed to their vast number of parameters and extensive pre-training. repllama @xcite fine-tuned llama-2-7b and llama-2-13b to function both as dense retrievers and pointwise rerankers. llara @xcite introduced two pretraining tasks specifically designed to better adapt the backbone llama-2-7b model for dense retrieval, resulting in notable improvements in both supervised and zero-shot scenarios. e5-mistral and gecko @xcite enhanced the training of llmbased dense retrievers using synthetic data, employing models with 1.5 billion and 7 billion parameters to achieve notable results across various retrieval tasks. grit @xcite successfully unified text embedding and generation within a single llm, maintaining performance levels comparable to those of specialized embedding-only and generative-only models, using a model with 56 billion parameters (14 billion activation parameters). @xcite presented an unsupervised method for transforming decoderonly llms into dense retrievers, demonstrating significant promise for adapting llm backbone encoders for dense retrieval in an unsupervised manner. promptreps @xcite employed human preference-aligned chat llms to produce high-quality dense representations unsupervised. these models vary in parameters from 1.5 billion to 56 billion, with pre-training covering hundreds of billions to tens of trillions of tokens, and include both base llms and human preference-aligned chat llms. despite the exciting advancements in retrieval tasks achieved by leveraging various llms with distinct configurations and diverse training strategies, the specific benefits of variations in parameter count, pre-training extent, and alignment processes of backbone llms for retrieval tasks remain still uncertain. ## preliminary dense retrieval leverages an encoder to project both the query q and the candidate passage p into a shared dense embedding space, resulting in embeddings h q and h p . a scoring function, such as the inner product or cosine similarity, is then applied to these dense vectors to model relevance: this allows for the retrieval of relevant documents by performing approximate nearest neighbor (ann) search within the embedding space. in our study, we compare more than 15 backbone encoders, varying in model architecture (encoderonly and decoder-only), model size (0.1b to 32b), and pre-training sufficiency (up to 15t tokens). consistent with prior research, we utilize the [cls] token to obtain text representations for the bert model and employ mean-pooling for the t5 model. for instance, bert tokenizes the input text into a sequence t: [cls], t 1 , ..., t n , [eos]. this tokenized sequence is subsequently encoded by bert, generating output embeddings that are combined to form the text embedding, with the [cls] token performing this integration: when using large language model (llm) as the backbone encoder, text embeddings need to be created differently. most llms use a decoder-only architecture and causal attention mechanism, meaning that only the last token in the input sequence can access the global context. as a result, the text embedding is taken from the output embedding of the special token [eos]: given the query-passage pair (q i , p + i ), we adopt the standard infonce @xcite loss l over the in-batch negatives and hard negatives for training: ## empirical study in this section, we aim to address two key research questions: 1) for different retrieval tasks, what specific benefits can llms offer compared to non-llms as the backbone encoders? 2) for llms with varying configurations (i.e., different parameter numbers, pre-training sufficiency, and alignment processes), what contributes more to different retrieval tasks as the backbone encoder. to answer these questions, we conduct a comprehensive empirical study across six critical dimensions of dense retrieval, each encompassing several specific retrieval tasks. these dimensions are investigated using various pre-trained language models as backbone encoders, focusing on: in-domain accuracy (section 4.1), data efficiency (section 4.2), zeroshot generalization (section 4.3), lengthy retrieval generalization (section 4.4), instruction-based retrieval (section 4.5), and multi-task learning (section 4.6). ## conclusions in this paper, we conduct a comprehensive empirical investigation into the benefits and configurations of llms as backbone encoders for dense retrieval tasks. our focus is on comparing llms with non-llms and analyzing the impact of various llm configurations, such as parameter count, pre-training sufficiency, and alignment processes. our study highlights the significant advantages of utilizing llms as backbone encoders for dense retrieval tasks. we find that increasing the parameter count and ensuring sufficient pre-training of backbone encoders enhance in-domain accuracy. additionally, adopting larger models consistently yields performance gains in zero-shot retrieval generalization, lengthy retrieval generalization, and multitask learning. these insights provide a foundation for future research aimed at optimizing dense retrieval models by balancing model size and pretraining sufficiency of backbone llms to achieve superior performance across diverse retrieval scenarios.