How Good are LLM -based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models (2025)

Abstract: In this work, we present a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods, encompassing large language model (LLM)-based, lightweight contextual, and zero-shot approaches, with respect to their performance in information retrieval tasks. We evaluate in total 22 methods, including 40 variants (depending on used LLM) across several established benchmarks, including TREC DL19, DL20, and BEIR, as well as a novel dataset designed to test queries unseen by pretrained models. Our primary goal is to determine, through controlled and fair comparisons, whether a performance disparity exists between LLM-based rerankers and their lightweight counterparts, particularly on novel queries, and to elucidate the underlying causes of any observed differences. To disentangle confounding factors, we analyse the effects of training data overlap, model architecture, and computational efficiency on reranking performance. Our findings indicate that while LLM-based rerankers demonstrate superior performance on familiar queries, their generalisation ability to novel queries varies, with lightweight models offering comparable efficiency. We further identify that the novelty of queries significantly impacts reranking effectiveness, highlighting limitations in existing approaches.

## related work large language models (llms) have transformed information retrieval (ir) by enabling semantic understanding and zero-shot ranking capabilities. retrieval-augmented generation (rag) @xcite integrates retrieval with llm generation to enhance response quality. reranking @xcite refines retrieved documents, prioritizing relevance to improve rag outcomes and reduce llm hallucinations. @xcite highlight reranking's role in evolving rag frameworks, boosting accuracy in tasks like question answering. @xcite note that reranking supports multimodal rag, mitigating data leakage by refining diverse data types. @xcite propose metrics like relevance to evaluate reranking's impact, emphasizing robust strategies for reliable rag performance. fairness in llm-based ranking is critical for equitable applications. @xcite find that llms like gpt and llama2 underrepresent groups on the trec fair ranking dataset, with exposure disparities up to 15%. traditional methods like fa*ir @xcite and exposure metrics @xcite struggle with llms' opaque decisions, underscoring the need for fairness-aware ranking approaches. ## reranking approaches reranking in information retrieval (ir) refines an initial set of retrieved documents to optimize their relevance to a query, a critical step in applications like web search and question answering. with the rise of pretrained language models (plms) and large language models (llms), reranking methods have evolved into three primary categories: pointwise, pairwise, and listwise. these approaches differ in how they score and order documents, balancing effectiveness, efficiency, and generalization. this section presents an overview of key types of reranking methods, detailing their methodologies, and key implementations. ## challenges in reranking with llm reranking refines retrieved documents to match queries in information retrieval (ir). llms enable zero-shot reranking but face challenges due to their general-purpose design, hindering performance compared to fine-tuned rankers. issues include computational complexity, api reliance, and prediction inconsistencies across pointwise, pairwise, and listwise methods. pointwise reranking challenges pointwise methods score query-document pairs independently as classification or regression, sorting by scores with o(n) complexity. a prompt (figure 2a ) yields a relevance judgment (e.g., "yes" or "no"), with scores defined as: pairwise reranking challenges pairwise methods compare document pairs for relative relevance, aggregating results with complexity from o(n 2 ) to o(n). a prompt (e.g., prp) selects the more relevant document, scoring d i as: listwise reranking challenges listwise methods process queries and documents together, outputting reordered lists (figure 2c ) with o(n) complexity. long inputs exceed llm context limits, requiring sliding windows or tournament sorts @xcite @xcite . prediction failures, such as missing documents or inconsistent rankings due to input order sensitivity, and reliance on generation apis @xcite reduce reliability of listwise reranking. ## results and discussion this section evaluates pointwise, pairwise, and listwise reranking methods across ir benchmarks and open-domain datasets, assessing performance, robustness, and generalization to novel queries. it includes three parts: experimental setup, datasets, and performance analysis. ## futurequeryeval reranking models are expected to generalize beyond their pretraining corpora, yet most benchmarks-like trec dl19, dl20, and beir-contain queries and documents that may overlap with the training data of llms @xcite . to ensure evaluations are performed on unseen, truly novel content, we introduce future-queryeval, a dataset designed to test reranking models on queries and documents collected after april 2025. the dataset comprises 148 queries across seven diverse topical categories (e.g., technology, sports, politics). each query is paired with manually annotated documents collected post-april 2025, ensuring they are out-of-distribution for most existing llms. relevance was assigned using a 3-level scale: 0 (irrelevant), 1 (partial), 2 (highly relevant). we validated novelty against gpt-4, confirming queries refer to events beyond the model's knowledge. full construction details, corpus examples, and statistical breakdowns (e.g., document lengths, token cdf, query distributions) are provided in appendix a.1. ## conclusion we presented a comprehensive empirical study of 22 reranking methods across 40 variants, spanning pointwise, pairwise, and listwise paradigms. our evaluation across standard ir benchmarks and the novel futurequeryeval dataset reveals that while llm-based rerankers excel on familiar queries, their generalization to unseen queries remains inconsistent. lightweight models, particularly those fine-tuned on ir data, achieve strong trade-offs between accuracy and efficiency. futurequeryeval, a temporally novel benchmark, exposes critical limitations in current reranking methods when faced with truly unseen data. listwise approaches, especially zephyr-7b and vicuna-7b, achieve the highest effectiveness but at significant computational cost. pointwise rerankers like monot5-3b and twolar-xl offer scalable, high-performing alternatives. pairwise methods provide fine-grained relevance signals yet struggle to scale.