C o R anking: Collaborative Ranking with Small and Large Ranking Agents (2025)

Abstract: Listwise ranking based on Large Language Models (LLMs) has achieved state-of-the-art performance in Information Retrieval (IR).However, their effectiveness often depends on LLMs with massive parameter scales and computationally expensive sliding window processing, leading to substantial efficiency bottlenecks. In this paper, we propose a Collaborative Ranking framework ( CoRanking ) for LLM-based listwise ranking.Specifically, we strategically combine an efficient small reranker and an effective large reranker for collaborative ranking.The small reranker performs initial passage ranking, effectively filtering the passage set to a condensed top-k list (e.g., top-20 passages), and the large reranker (with stronger ranking capability) then reranks only this condensed subset rather than the full list, significantly improving efficiency. We further address that directly passing the top-ranked passages from the small reranker to the large reranker is suboptimal because of the LLM’s strong positional bias in processing input sequences. To resolve this issue, we propose a passage order adjuster learned by RL that dynamically reorders the top passages returned by the small reranker to better align with the large LLM’s input preferences. Our extensive experiments across three IR benchmarks demonstrate that CoRanking achieves superior efficiency, reducing ranking latency by approximately 70% while simultaneously improving effectiveness, compared to the standalone large reranker.

## introduction in recent years, many studies @xcite @xcite have demonstrated strong zero-shot passage ranking capabilities of large language models (llms). a typical approach is listwise ranking, which feeds the query and a list of passages into an llm and instructs it to output * corresponding author. a ranked list of passage ids in descending order of relevance. this approach has established stateof-the-art performance @xcite on major ir benchmarks, including trec @xcite and beir @xcite . despite its effectiveness, llm listwise passage ranking faces critical efficiency challenges, as illustrated in figure 1 (a), primarily stemming from two factors: first, its state-of-the-art performance relies on llms with massive parameters (such as qwen2.5-72b or gpt-4), which introduces significant computational overhead and results in an efficiency issue. second, due to the limitation of the context length of llms, existing listwise ranking methods typically employ a sliding window strategy @xcite . with a fixed window size (e.g., 20 passages) and step size (e.g., 10 passages), this strategy bubbles the relevant passages that are initially lowly ranked to the top. however, the interdependence between windows hinders the parallelization of sliding windows, which results in many repetitive and sequential rankings of sliding windows (e.g., totally 9 sliding windows if window size and step size are set as 20 and 10 when ranking the top 20 from 100 passages) and creates an efficiency bottleneck (especially when employing llms with massive parameters). the sliding window strategy of llm reranker primarily serves to elevate initially low-ranked relevant passages into the final reranking window (e.g., top 20). we believe that this repetitive and inefficient process could be replaced by a small listwise reranker. 1 by utilizing a small listwise reranker to pre-rank the whole list of passages (e.g., 100 passages) using a sliding window strategy, the large listwise reranker, which has stronger ranking capability, only needs to rerank the passages in the final window (i.e., containing the top 20 passages). pre-rank 100 passages with a small listwise reranker using a sliding window strategy, then rerank the top 20 with a large listwise reranker. (c) coranking: pre-rank 100 passages with a small reranker using a sliding window strategy, adjust the order of the top 20 via the passage order adjuster, then rerank them with the large listwise reranker. "large/small reranker calls" represents the number of sliding windows for listwise ranking. this hybrid approach could dramatically reduce computational costs and improve efficiency. however, existing small listwise rerankers are mainly trained on golden lists whose passages are ranked solely based on relevance @xcite . we argue that directly feeding the topranked passages returned by these small listwise rerankers into a large listwise reranker may lead to sub-optimal performance (see figure 1 (b) ). this is because the effectiveness of a listwise reranker has a complex relationship with the order of input passages, rather than being determined exclusively by their relevance, as indicated by existing studies @xcite . for example, @xcite reveal that relevant passages appearing in the middle of the list tend to be ranked at the bottom by the listwise reranker. this stems from llms' pre-training biases toward the positions of input tokens @xcite . while the passage order generated by the small listwise reranker, which is trained using relevancebased ranking, may deviate from the large listwise reranker's order preference. to address this issue, we introduce a passage order adjuster (also a small listwise reranker) to rerank the top-ranked passages from the small listwise reranker so that they align with the order preference of the large listwise reranker (see figure 1 (c) ). more specifically, we propose a collaborative ranking framework, namely coranking, that integrates small and large ranking models to perform passage ranking in a multi-stage manner. in the first stage, we train a small listwise reranker (slr) to pre-rank all the passages to elevate the relevant passages to the top part of the list (e.g., top-20). different from existing methods @xcite ) that solely use teacher-generated ranking lists as training labels, we propose a human-labelenhanced method to improve the ranking quality. in the second stage, we train a passage order adjuster (poa) using the reinforcement learning algorithm dpo. the poa reorders the top-ranked passages to align with the large listwise reranker's order preference. in addition, we design a significance-aware selection strategy s 3 to build high-quality preference pairs for dpo training. in the third stage, a large listwise reranker is used to further rerank the top-ranked passages, enhancing the overall ranking effectiveness. extensive experiments on diverse ir benchmarks demonstrate that our coranking framework achieves a 70% reduction in ranking latency while maintaining superior effectiveness, compared to a standalone large listwise reranker. our contributions are summarized as follows: ## methodology in this section, we present a collaborative ranking framework coranking that integrates both small and large passage rerankers. as illustrated in figure 2 , the framework comprises three key components: (1) a small listwise reranker, (2) a passage order adjuster, and (3) a large listwise reranker. next, we will introduce the definition of the ranking task, the inference process of coranking, and the training details. ## conclusion in this paper, we introduce a novel collaborative ranking framework, namely coranking, to combine small and large listwise rerankers for more efficient and effective passage ranking. specifically, we first train a small listwise reranker slr to move the relevant passages to the top of the list. then we introduce a passage order adjuster poa to rerank the top-ranked passages to better align with the order preference of a large listwise reranker. finally, the large listwise reranker is utilized to solely rerank the top passages. extensive experiments demonstrate the efficiency advantage and superior performance of our coranking.