Is C hat GPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023)

Abstract: Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model’s ability to rank unknown knowledge. Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

## introduction large language models (llms), such as chat-gpt and @xcite , are revolutionizing natural language processing with strong zero-shot and few-shot generalization. by pretraining on large-scale text corpora and alignment fine-tuning to follow human instructions, llms have demonstrated their superior capabilities in language understanding, generation, interaction, and reasoning @xcite . figure 1 : average results of chatgpt and gpt-4 (zero-shot) on passage re-ranking benchmarks (trec, beir, and mr.tydi), compared with bm25 and previous best-supervised systems (sota sup., e.g., monot5 @xcite ). as one of the most successful ai applications, information retrieval (ir) systems satisfy user requirements through several pipelined sub-modules, such as passage retrieval and re-ranking @xcite . most previous methods heavily rely on manual supervision signals, which require significant human effort and demonstrate weak generalizability @xcite . therefore, there is a growing interest in leveraging the zero-shot language understanding and reasoning capabilities of llms in the ir area. however, most existing approaches primarily focus on exploiting llms for content generation (e.g., query or passage) rather than relevance ranking for groups of passages @xcite . compared to the common generation settings, the objectives of relevance re-ranking vary significantly from those of llms: the re-ranking agents need to comprehend user requirements, globally compare, and rank the passages based on their relevance to queries. therefore, leveraging the llms' capabilities for passage re-ranking remains a challenging and unanswered task. to this end, we focus on the following questions: to answer the first question, we investigate prompting chatgpt with two existing strategies @xcite . however, we observe that they have limited performance and heavily rely on the availability of the log-probability of model output. thus, we propose an alternative instructional permutation generation approach, instructing the llms to directly output the permutations of a group of passages. in addition, we propose an effective sliding window strategy to address context length limitations. for a comprehensive evaluation of llms, we employ three well-established ir benchmarks: trec @xcite , beir @xcite , and my.tydi @xcite . furthermore, to assess the llms on unknown knowledge and address concerns of data contamination, we suggest collecting a continuously updated evaluation testbed and propose noveleval, a new test set with 21 novel questions. to answer the second question, we introduce a permutation distillation technique to imitate the passage ranking capabilities of chatgpt in a smaller, specialized ranking model. specifically, we randomly sample 10k queries from the ms marco training set, and each query is retrieved by bm25 with 20 candidate passages. on this basis, we distill the permutation predicted by chat-gpt into a student model using a ranknet-based distillation objective @xcite . our evaluation results demonstrate that gpt-4, equipped with zero-shot instructional permutation generation, surpasses supervised systems across nearly all datasets. figure 1 illustrates that gpt-4 outperforms the previous state-of-the-art models by an average ndcg improvement of 2.7, 2.3, and 2.7 on trec, beir, and my.tydi, respectively. furthermore, gpt-4 achieves state-of-the-art performance on the new noveleval test set. through our permutation distillation experiments, we observe that a 435m student model outperforms the previous state-of-the-art monot5 (3b) model by an average ndcg improvement of 1.67 on beir. additionally, the proposed distillation method demonstrates cost-efficiency benefits. in summary, our contributions are tri-fold: ## passage re-ranking with llms ranking is the core task in information retrieval applications, such as ad-hoc search @xcite , web search @xcite , and open-domain question answering @xcite . modern ir systems generally employ a multi-stage pipeline where the retrieval stage focuses on retrieving a set of candidates from a large corpus, and the re-ranking stage aims to re-rank this set to output a more precise list. recent studies have explored llms for zero-shot re-ranking, such as instructional query generation or relevance generation (sachan et al., 2022; liang et al., 2022). however, existing methods have limited performance in re-ranking and heavily rely on the availability of the log probability of model output and thus cannot be applied to the latest llms such as gpt-4. since chatgpt and gpt-4 have a strong capacity for text understanding, instruction following, and reasoning, we introduce a novel instructional permutation generation method with a sliding window strategy to directly output a ranked list given a set of candidate passages. figure 2 illustrates examples of three types of instructions; all the detailed instructions are included in appendix a. ## specialization by permutation distillation although chatgpt and gpt-4 are highly capable, they are also too expensive to deploy in commercial search systems. using gpt-4 to re-rank passages will greatly increase the latency of the search system. in addition, large language models suffer from the problem of unstable generation. therefore, we argue that the capabilities of large language models are redundant for the re-ranking task. thus, we can distill the re-ranking capability of large language models into a small model by specialization. ## datasets our experiments are conducted on three benchmark datasets and one newly collected test set noveleval. ## experimental results of llms 6.1 implementation and metrics in benchmark datasets, we re-rank the top-100 passages retrieved by bm25 using pyserini foot_0 and use ndcg@{1, 5,10} as evaluation metrics. since chatgpt cannot manage 100 passages at a time, we use the sliding window strategy introduced in section 3.2 with a window size of 20 and step size of 10. in noveleval, we randomly shuffled the 20 candidate passages searched by google and re-ranked them using chatgpt and gpt-4 with permutation generation. ## experimental results of specialization ## conclusion in this paper, we conduct a comprehensive study on passage re-ranking with llms. we introduce a novel permutation generation approach to fully explore the power of llms. our experiments on three benchmarks have demonstrated the capability of chatgpt and gpt-4 in passage re-ranking. to further validate llms on unfamiliar knowledge, we introduce a new test set called noveleval. additionally, we propose a permutation distillation method, which demonstrates superior effectiveness and efficiency compared to existing supervised approaches.