P a R a D e: Passage Ranking using Demonstrations with LLM s (2023)

Abstract: Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.

## introduction large language models (llms) exhibit strong performance on a variety of tasks without additional task-specific fine-tuning. their success is often attributed to in-context learning, where the parameters of the language model are frozen and it learns how to perform a new task by reading demonstrations in the prompt @xcite @xcite . while llms are often used to generate answers, our focus is on scoring for the task of passage reranking-passages are first retrieved by an efficient retriever, e.g. bm25, then rated and reordered by the llm. existing works like upr @xcite demonstrate promising results for zero-shot ranking using llm. we aim to improve over zeroshot ranking by including demonstrations in the prompt and explore multiple strategies for selecting demonstrations. manual selection is often sub-optimal and requires a human-in-the-loop when using the llm for a new task. instead, we seek a method that finds effective demonstrations automatically, with minimal or no human involvement. in this paper, we investigate approaches for automatic demonstration selection to improve upon upr's zero-shot ranking approach. our initial analysis highlights the complex nature of the problem, showing that ranking performance varies drastically depending on the demonstrations included in the prompt. furthermore, simply including more demonstrations does not always lead to better ranking quality. next, we investigate the use of established demonstration selection methods, i.e. similarity-based selection @xcite , on ranking tasks and show that similarity of demonstrations does not correlate well with ranking quality. thereafter, we propose difficultybased selection (dbs) as a simple and effective approach to automatically find challenging, i.e. low likelihood, demonstrations to include in the prompt. although we prompt frozen llms, we intend to emulate the training dynamics of fine-tuning, and choose hard samples because they potentially correspond to large gradient updates and are often chosen to improve learning in gradient descent @xcite . finally, given the increasing importance of question generation for ranking @xcite @xcite , we extend the uses of the proposed difficulty-based selection for better question generation. to this end, we present passage ranking with demonstrations (parade). our main contributions include: (1) analysis highlighting the complexity of demonstration selection; (2) dbs, an automatic and effective way to choose demonstrations; and (3) extensive experiments on re-ranking and question generation, including results with an extension of dbs that jointly selects multiple demonstrations. ## llm re-ranking by query likelihood background. given a query q and set of initially retrieved documents d, the goal is to rank each document d in d by relevance with respect to q. upr @xcite introduces a zeroshot method to rank documents according to the log-likelihood of q given d using a large language model, ## experiments on trec and beir to empirically measure the effectiveness of demonstration selection, we conduct analysis using the reranking task on trec 2019 and 2020, and further perform evaluation on seven datasets from beir @xcite . we use language models known to effectively incorporate demonstrations through in-context learning, including the flan-t5-xl and xxl @xcite and the more recently palm 2-s (google et al., 2023). ## related work concurrent with upr, promptrank @xcite is the most related prior work, using demonstrations to re-rank "document paths" for multihop-qa. details of how they select demonstrations is unclear, motivating us to conduct our own study. our difficulty-based demonstration selection ( §4.2) is closely related to active learning @xcite @xcite . similarly, @xcite measure uncertainty with generation instead of scoring. @xcite formulate demonstration selection as a reinforcement learning problem. @xcite use llm-scoring to find hard negatives for their trained demonstration retriever. others explore demonstration ordering @xcite and joint selection @xcite @xcite . concurrent to our work, @xcite perform multiple rounds of hill climbing to find groups of demonstrations that perform well according to a validation set. in contrast, dbs selects demonstrations directly and does not rely on validation. discriminative methods are widely used in supervised ranking @xcite @xcite . listwise prompting is an alternative to query likelihood, but requires a sliding window strategy as not all documents fit in the context @xcite . rather than query likelihood, hyde @xcite achieves zero-shot ranking through document generation, which we hypothesize would be improved through demonstrations. parade is bounded by the first stage bm25 retrieval, and it may be fruitful to explore approaches that align first stage retrieval with our demonstration-based approach @xcite . ## conclusion in this work we present passage ranking with demonstrations (parade), an extensive study on the topic of using demonstrations to improve reranking performance of llms. we show the challenges of applying demonstrations effectively, and that performance heavily relies on selecting "good" demonstrations. we propose a simple yet effective selection method, named difficulty-based selection (dbs), and confirm its effectiveness in both reranking using query likelihood scoring and query generation tasks. for future work, we plan to combine difficulty-based selection with similaritybased selection as an effort to further improve the robustness and effectiveness of the selected demonstrations, and extend dbs to other ranking paradigms @xcite .