Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024)

Abstract: Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets.We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP).Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10.Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

## introduction large language model (llms) such as gpt-3 @xcite and palm @xcite have demonstrated impressive performance on a wide range of natural language tasks, achieving comparable or better performance when compared with their supervised counterparts that are potentially trained with millions of labeled examples, even in the zero-shot setting @xcite @xcite . however, there is limited success for the important text ranking problem using off-the-shelf llms @xcite . existing results usually significantly underperform well-trained baseline rankers (e.g., @xcite ; @xcite ). the only exception is a recent approach proposed by @xcite , which depends on the blackbox commercial gpt-4 system. besides the technical concerns such as sensitivity to input order (ranking metrics can drop by more than 50% when the input document order changes), we argue that relying on such blackbox systems is not ideal for academic researchers due to significant cost constraints and access limitations to these systems, though we do acknowledge the value of such explorations in showing the capabilities of llms for ranking tasks. in this work, we first discuss why it is difficult for llms to perform ranking tasks with existing methods, specifically, the pointwise and listwise formulations. for pointwise approaches, ranking requires llms to output calibrated prediction probabilities before sorting, which is known to be very difficult and is not even supported by the generationonly llm apis (such as gpt-4). for listwise approaches, even with instructions that look very clear to humans, llms can frequently generate conflicting or useless outputs, which happens especially often for moderate-sized llms that are used in our experiments. such observations show that existing popular llms do not fully understand ranking tasks, potentially due to the lack of ranking awareness during their pre-training and (instruction) fine-tuning procedures. we propose the pairwise ranking prompting (prp) paradigm, which uses the query and a pair of documents in the prompt for llms to perform rank-ing tasks, with the motivation to significantly reduce the task complexity for llms and resolve the calibration issue. prp is based on simple prompt design and naturally supports both generation and scoring llms apis. we describe several variants of prp to address efficiency concerns. prp results are the first in the literature that can achieve stateof-the-art ranking performance by using moderatesized, open-sourced llms on standard benchmark datasets. on trec-dl2020, prp based on the flan-ul2 model with 20b parameters outperforms the previous best approach in the literature, based on the blackbox commercial gpt-4 that has (an estimated) 50x model size, by over 5% at ndcg@1. on trec-dl2019, prp is only inferior to the gpt-4 solution on the ndcg@5 and ndcg@10 metrics, but can outperform existing solutions, such as instructgpt which has 175b parameters, by over 10% for nearly all ranking metrics. we also show competitive results using flan-t5 models with 3b and 13b parameters, demonstrating the power and generality of prp. the observations are further validated on seven beir datasets covering various domains, where prp performs competitively with supervised rankers and outperforms other llm based approaches by a large margin. we further discuss other benefits of prp, such as being insensitive to input ordering. we note that "pairwise" paradigm is in itself a very general and classic idea that impacted a wide range of areas. the novelty of our work lies in the important scenario where the technique is introduced, the adaptations to make it practical, the effectiveness it enables, as well as potential changes and insights it inspires. in summary, the contributions of this paper are three-fold: as discussed in section 1, to date there is limited evidence showing off-the-shelf llm-based rankers can outperform fine-tuned smaller rankers. we discuss why this is the case by overviewing and analyzing existing methods, which can be categorized into pointwise or listwise approaches. ## pairwise ranking prompting we propose pairwise ranking prompting (prp) for ranking with llms. we describe the basic pairwise prompting unit, how it supports both generation and scoring apis, and propose several variants of prp with different ranking strategies and efficiency properties. ## ablation studies we perform several ablative studies to gain a deeper understanding of the prp framework in terms of its robustness and generality. robustness to input ordering. we show the robustness of prp to input ordering. one issue of listwise ranking prompting approaches is their sensitivity to input ordering. this is because the ranking will fall back to the initial order when llm prediction fails, which is very common for the difficult listwise formulation. in ## discussion extendability. the design of prp in this paper biases towards simplicity and generality. for example, we decribe the algorithm and report results based on generation api, so prp is applicable to both commercial black-box llms and opensourced white-box llms. the performance may further improve via more sophisticated prompt design, and leveraging extra information such as the score values from the scoring api, which is usually available for white-box llms. we provide some results of prp on a commercial llms in appendix b where performance can be further improved. reproducibility. we used the same prompt template for all 9 datasets evaluated in the paper, showing the generality and power of pairwise ranking prompting in text ranking. as we focus on opensourced llms, and only use standard aggregation methods (win counting, sorting, and sliding window), our experimental results are easy to reproduce. still, we plan to release pairwise inference results on all 9 datasets and the 3 open-source llms to facilitate future research. in specific, we will release the data in json format, which includes query/document information for each pair (including ids, text, label, retrieval rank and scores), together with the actual prompt, the generated text, and its score. the specific prompt template and a data sample can be found at appendix e cost and efficiency. we discussed different efficient variants of prp. also, our results are based on llms that are easily approachable for academic researchers @xcite , alleviating the need to call commercial apis. however, further reducing the number of calls to llms is still an interesting research direction, such as leveraging active learning techniques. the distillation of llm rankers to servable models in large-scale systems is also an important future direction @xcite . data leakage from llms. we note there is minimal label leakage issues as we leverage opensourced llms with clear documentations, while it is not clear for blackbox commercial llms. the comparisons with existing pointwise and listwise approaches on the same llms are also fair. please see a more comprehensive examination on data leakage in appendix c. ## related work we did a detailed review and analysis of the most relevant existing efforts for ranking with llms, including pointwise and listwise approaches in section 2. these works and ours focus on the challenging unsupervised text ranking setting with llms without providing any demonstrations, conducting any fine-tuning, or training of an additional model. prior to the recent efforts on ranking with llms, most work focus on the supervised learning to rank problem @xcite by fine-tuning pre-trained language models (plms) such as t5 @xcite or bert @xcite , which serve as very strong baselines. very recently some work fine-tunes llms or distills from black-box llms @xcite , which is different from our setting. there has been a strong recent interest in exploring information retrieval in general with llms based approaches @xcite , due to the importance of the applications and the power of llms to understand textual queries and documents @xcite @xcite @xcite . several works leverage the generation power of llms to generate training data to train an additional downstream retrieval or ranking model, typically in the few-shot setting @xcite , which is a very different setting from ours. recent methods in this family of methods such as inpars @xcite still significantly underperforms fine-tuned baselines. exaranker @xcite uses llms to generate explanations for ranking decisions, and uses such explanations in ranking model fine-tuning, showing limited ranking performance benefits (the major benefit was on data efficiency). hyde @xcite uses llms to augment queries by generating hypothetical documents for unsupervised retrieval. these works do not directly explore the retrieval or ranking capability of llms, but mainly use llms as auxiliary tools to complement traditional paradigms, possibly limiting the benefits that llms can provide. new paradigms such as differentiable search index (dsi) @xcite directly use transformer memory to index documents for retrieval. using pairwise comparisons with llms is a general paradigm, such as reward modeling using pairwise preferences @xcite @xcite . llms are used as evaluators to compare generative outputs (such as text summary) @xcite . @xcite performs structured comparative reasoning to predict text preferences in various applications. 1sl (macavaney and soldaini, 2023) estimates relevance with reference to an anchor positive query-document pair per query, even for the test set, so the setting may not be practical and is very different from our standard text rank-ing setting. a concurrent work @xcite studied pairwise prompting in recommender systems, which is a substantially different application and their method still largely fall behind state-ofthe-art models with sufficient data. the novelty of our work lies in leveraging the general and simple pairwise prompting paradigm to the important text ranking task, granting llms capabilities that no prior work can, by performing competitively with state-of-the-art fine-tuned models and methods that only work with giant blackbox llms. ## conclusion in this paper, we propose to use pairwise prompting with llms for text ranking tasks. to the best of our knowledge, these are the first published results demonstrating very competitive ranking performance using moderate-sized, open-sourced llms. the key insights are the observation of the difficulties of llms handling ranking tasks in the existing pointwise and listwise formulations. our proposed pairwise ranking prompting (prp) is effective in reducing the burden of llms and shows robust performance on 9 datasets. we also discuss efficiency concerns and ways to mitigate them, and several benefits of prp, such as insensitivity to input ordering and support for both generation and scoring llm apis.