D e AR : Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation (2025)

Abstract: Large Language Models (LLMs) have transformed listwise document reranking by enabling global reasoning over candidate sets, yet single models often struggle to balance fine-grained relevance scoring with holistic cross-document analysis. We propose DeepAgentRank (DeAR), an open-source framework that decouples these tasks through a dual-stage approach, achieving superior accuracy and interpretability. In Stage 1, we distill token-level relevance signals from a frozen 13B LLaMA teacher into a compact 3, 8B student model using a hybrid of cross-entropy, RankNet, and KL divergence losses, ensuring robust pointwise scoring. In Stage 2, we attach a second LoRA adapter and fine-tune on 20K GPT-4o-generated chain-of-thought permutations, enabling listwise reasoning with natural-language justifications. Evaluated on TREC-DL19/20, eight BEIR datasets, and NovelEval-2306, DeAR surpasses open-source baselines by +5.1 nDCG@5 on DL20 and achieves 90.97 nDCG@10 on NovelEval, outperforming GPT-4 by +3.09. Without fine-tuning on Wikipedia, DeAR also excels in open-domain QA, achieving 54.29 Top-1 accuracy on Natural Questions, surpassing baselines like MonoT5, UPR, and RankGPT. Ablations confirm that dual-loss distillation ensures stable calibration, making DeAR a highly effective and interpretable solution for modern reranking systems.

## introduction document reranking refines top candidate documents retrieved by a first-stage system to improve relevance to a user query. it plays a critical role in tasks like web search @xcite , opendomain qa @xcite , fact verification @xcite , and 1 dataset and code available at https://github. com/datascienceuibk/dear-reranking . retrieval-augmented generation (rag) @xcite , where ranking quality directly impacts downstream results. transformer-based models (e.g., bert @xcite , t5 @xcite ) and instruction-tuned llms (e.g., instructgpt @xcite , llama @xcite , gpt-4 @xcite ) have driven reranking progress. while pointwise @xcite and pairwise @xcite approaches dominate earlier work, listwise reranking with llms @xcite offers global ranking benefits. however, these often rely on expensive proprietary apis and suffer from context-length limitations and brittle reasoning. as shown in figure 1 , our proposed dear-l achieves superior ndcg@5 performance across diverse datasets, outperforming baselines like rankzephyr and rankgpt. to address these challenges, open-source efforts have explored knowledge distillation (kd) to transfer ranking abilities from large llms to smaller, more efficient student models @xcite . however, existing synthetic training sample system prompt: you are rankllm, an intelligent assistant that can rank passages based on their relevancy to the query. instruction: i will provide you with 3 passages, each indicated by a numerical identifier []. rank the passages based on their relevance to the search query: what is a shape of art. the format of the final output should be ' ### final reranking: [] > []', " "e.g., ### final reranking: methods commonly depend on synthetic listwise permutations that risk propagating teacher errors such as hallucinations or misrankings. this motivates two key research questions: (1) can we balance kl divergence with ranking loss to effectively distill logit-level signals from the teacher while mitigating noise? (2) can we incorporate synthetic reasoning chains to retain listwise reasoning benefits without overloading model context windows? we address these questions with a central insight: reranking performance can be enhanced by guiding the student with reasoning chains of thought (cot) derived from synthetic data. we introduce dear, a novel dual-stage reranking framework that combines pointwise and listwise learning with reasoning-augmented supervision. built on a frozen llm backbone (e.g., llama-13b) with lightweight lora adapters @xcite , dear trains on 20,000 synthetic reasoning examples and achieves performance comparable to gpt-4o, while surpassing rankzephyr in inference efficiency (see section 5.3). our contributions are as follows: ## related work reranking methods @xcite are typically pointwise, pairwise, or listwise. pointwise models like monobert @xcite score each document independently using pretrained transformers @xcite , but ignore inter-document context @xcite . pairwise approaches (e.g., duobert, duot5 @xcite ) compare document pairs to infer preferences, at the cost of efficiency @xcite . listwise methods, powered by llms like gpt-3.5 and gpt-4, enable zero-shot ranking through prompting @xcite , though reliance on proprietary apis limits reproducibility. open-source variants like rankvicuna and rankzephyr (pradeep et al., 2023b) address this via distilled listwise models, dataset positive docs negative docs teacher freeze teacher llm logits cross entropy ranknet softmax student trainable student llm logits cross entropy ranknet softmax kl loss 𝐿 = 1 -𝛼 𝑓 𝑆𝑡𝑢𝑑𝑒𝑛𝑡, 𝑇𝑒𝑎𝑐ℎ𝑒𝑟 +𝛼𝐾𝐿 𝐿𝑜𝑠𝑠 pointwise reranking prompt reasoning listwise reranking sorted passages prompt teacher reasoning teacher @xmath0. knowledge distillation (kd) compresses large models into smaller ones by transferring logits or intermediate signals @xcite . in ir, it has been used for both retrieval @xcite and reranking @xcite . rankgpt and rankzephyr apply permutation-based kd using teacher rankings, but such discrete labels may discard confidence information. other work explores reasoning distillation @xcite , where small models benefit from teacher-generated explanations. our method extends this by combining logit-level kd (stage 1) with reasoning-based listwise training (stage 2), preserving fine-grained supervision while improving stability. reasoning improves reranking accuracy and interpretability. r2r @xcite distills direct and comparative explanations from gpt-4 for msmarco and beir. rankgpt also leverages implicit llm reasoning via prompt completions. ## experiments we evaluate dear on standard reranking benchmarks and settings.