Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking (2023)

Abstract: In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. We make our codebase publicly available at https://github.com/ielab/llm-qlm.

## introduction ranking models (or rankers) are a fundamental component in many information retrieval (ir) pipelines. pre-trained language models (plms) have recently been leveraged across biencoder @xcite @xcite @xcite , cross-encoder @xcite @xcite , and sparse @xcite @xcite ) ranker architectures, showing impressive ranking effectiveness. despite this success, the strong effectiveness of plm-based rankers does not always generalise without sufficient in-domain training data @xcite @xcite . transferring knowledge from other domains has been used to overcome this issue @xcite by training these rankers on large-scale supervised qa datasets such as ms marco @xcite . alternatively, generative large language models (llms) like gpt3 @xcite have been used to synthesize domain-specific training queries, which are then used to train these rankers @xcite . despite their effectiveness, all of these methods consume significant expenses in training a plmbased ranker. in this paper, we consider a third avenue to address this challenge: leveraging llms to function as query likelihood models (qlms) @xcite @xcite . essentially, qlms are expected to understand the semantics of documents and queries, and estimate the possibility that each document can answer a certain query. notably, recent advances in this direction have greatly enhanced the ranking effectiveness of qlm-based rankers by leveraging plms like bert @xcite and t5 @xcite . these plm-based qlms are finetuned on query generation tasks and subsequently employed to rank documents as per their likelihood @xcite @xcite . we focus on a specific plm-based qlm, the recently proposed unsupervised passage re-ranker (upr) @xcite . upr leverages advanced llms to obtained the query likelihood estimations. empirical results show that using the t0 llm @xcite as a qlm, large gains in ranking effectiveness can be obtained. a key aspect of this work is that this effectiveness is obtained without requiring additional fine-tuning data, making sachan et al. highlight the zero-shot ranking capabilities of their llm-based qlm. however, we argue that the experimental setting used by @xcite does not fully align with a genuine zero-shot scenario for the qlm ranking task. this is because t0 has already undergone fine-tuning on numer-ous question generation (qg) tasks and datasets, subsequent to its unsupervised pre-training. 1 consequently, there exists a discernible task leakage to the downstream qlm ranking task, thereby rendering their approach more akin to a transfer learning setting, rather than a true zero-shot approach. to gain a comprehensive understanding of the zero-shot ranking capabilities of llm-based qlm rankers, in this paper we take a fresh examination of this topic. our approach involves harnessing the power of state-of-the-art transformer decoderonly llms, such as llama @xcite , which have undergone pre-training solely on unstructured text through unsupervised next token prediction. importantly, the models we consider have not undergone any additional supervised instruction fine-tuning, ensuring a truly complete zero-shot setting for our investigation. we further extend our analysis by comparing the effectiveness of these llms with various popular instruction-tuned llms in the context of zeroshot ranking tasks. interestingly, our findings reveal that further instruction fine-tuning adversely affects the effectiveness of qlm ranking, particularly when the fine-tuning datasets lack specific qg tasks. this insight highlights the strong zero-shot qlm ranking ability of llms that solely rely on pre-training, thereby suggesting that further instruction fine-tuning is unnecessary for achieving strong zero-shot effectiveness. building upon these insights, we push the boundaries of zero-shot ranking even further by integrating a hybrid zero-shot firststage retrieval system, followed by re-ranking using the zero-shot llm-based qlm re-rankers and a relevance score interpolation technique @xcite . our approach achieves state-of-theart effectiveness in zero-shot ranking on a subset of the bier dataset @xcite . ## methodology zero-shot qlm re-ranker: we follow the setting introduced in previous works @xcite to evaluate the zero-shot qlm ranking capability of llms. specifically, given a sequence of query tokens q and a set d containing candidate documents retrieved by a first-stage zero-shot retriever such as bm25, the objective is to rank all candidate documents @xmath0: https://huggingface.co/datasets/ bigscience/p3 the average log likelihood of generating all query tokens, as estimated by a llm. the relevance scoring function is defined as: interpolating with first-stage retriever: following @xcite , instead of solely relaying on the query likelihood scores estimated by the llms, we also linearly interpolate the qlm score with the bm25 scores from the first-stage retriever by using the weighted score sum: here, α ∈ [0, 1] represents the weight assigned to balance the contribution of the bm25 score and the qlm score. in our experiments, we heuristically apply min-max normalization to the scores and assign more weight to the qlm scores, given its pivotal role as the second-stage re-ranker. this is achieved by setting α = 0.2 without conducting any grid search. we use the python library ranx foot_0 @xcite to implement the interpolation algorithm. few-shot qlm re-ranker: since llms are strong few-shot learners (brown et al., 2020), we also conducted experiments to explore how llmbased qlm re-rankers could be further enhanced by providing a minimal number of human-judged examples. to achieve this, we employed a prompt template known as "guided by bad questions" (gbq) @xcite . the gbq template consists of only three document, good question, and bad question triples. we use it to guide the llm-based qlm to produce more accurate query likelihood estimations. we refer readers to the original paper for details about the gbq template. ## experimental settings llms: our focus is on the response of llms in the qlm ranking task, specifically in a genuine zero-shot setting. to accomplish this, we used llama @xcite and falcon @xcite , both of which are transformer decoder-only models that are pre-trained solely on large, publicly available unstructured datasets @xcite . we specifically consider open-source llms because we can control the data used to train them, thus guaranteeing no qg dataset was used. to evaluate the influence of instruction finetuning data on qlm estimation, we compared these models with other well-known llms that were fine-tuned with instructions, including t5 @xcite , alpaca @xcite , stablelm, stablevicuna, and falcon-instruct @xcite . it is important to note that the finetuning instruction data for these models are unlikely to include qg tasks. 3 additionally, we follow @xcite to include t0 @xcite and @xcite , which underwent fine-tuning specifically for qg instructions. all llms used in this paper are openly available, see appendix b for more details. baselines and datasets: in our evaluation, we compared llm-based qlms with several existing methods, including bm25 @xcite , qlm-dirichlet @xcite , contriever @xcite , and hyde @xcite , which are zero-shot first-stage retrievers. we also compared them with fine-tuned retrievers and re-rankers trained on ms marco passage ranking data, representing a transfer learning setting. specifically, the evaluated retrievers are contriever-msmarco, @xcite , and dragon+ @xcite , while the re-rankers are t5-qlmlarge @xcite , monot5-3b @xcite , and monot5-3b-inpars-v2 @xcite . additionally, we compared our best qlm ranking pipeline with promptaga-tor @xcite , a state-of-the-art zero-shot and few-shot method. see appendix c for detailed information about the baselines. to ensure feasibility, we conducted experiments on a popular subset of the beir benchmark datasets foot_2 : trecc @xcite , dbpedia @xcite , fiqa @xcite , and robust04 @xcite . the evaluation metric used is ndcg@10, the official metric of the beir benchmark. statistical significance analysis was performed using student's two-tailed paired t-test with corrections, as per common practice in information retrieval. the results of this analysis is reported in appendix d due to space constraints. ## conclusion in this paper, we adapt recent advanced llms into qlms for ranking documents and comprehensively study their zero-shot ranking ability. our results highlight that these llms possess remarkable zeroshot ranking effectiveness. moreover, we observe that additional instruction fine-tuned llms unperformed in this task. this important insight is overlooked in previous studies. furthermore, our study shows that by integrating llm-based qlms with a hybrid zero-shot retriever, a novel state-of-the-art ranking pipeline can be obtained that excels in both