Explain then Rank: Scale Calibration of Neural Rankers Using Natural Language Explanations from LLM s (2025)

Abstract: In search settings, calibrating the scores during the ranking process to quantities such as click-through rates or relevance levels enhances a system’s usefulness and trustworthiness for downstream users. While previous research has improved this notion of calibration for low complexity learning-to-rank models, the larger data demands and parameter count specific to modern neural text rankers produce unique obstacles that hamper the efficacy of methods intended for the learning-to-rank setting.This paper proposes exploiting large language models (LLMs) to provide relevance and uncertainty signals for these neural text rankers to produce scale-calibrated scores through Monte Carlo sampling of natural language explanations (NLEs). Our approach transforms the neural ranking task from ranking textual query-document pairs to ranking corresponding synthesized NLEs. Comprehensive experiments on two popular document ranking datasets show that the NLE-based calibration approach consistently outperforms past calibration methods and LLM-based methods for ranking, calibration, and query performance prediction tasks.

## introduction neural ranking models act as the core component of many search systems, often producing the final document scores. however, these scores are usually treated as transient information and only the relative orderings are preserved to produce a ranking. while this approach results in wellperforming systems with respect to common retrieval metrics, such as ndcg and map, it ignores vital information used by end users and downstream applications with real-world impacts, such as fair ranking @xcite , ranked list truncation @xcite , and query perfor-query: "how to put together a scuba regulator" -------------document: "the 1st stage's filter should be positioned against the valve's o-ring, and the 2nd stage hoses should be on the same side as the valve knob. …. don't worry if the 1st stage rotates on the valve." task: to assign a meaningful ranking score to the query-document pair. 0 means irrelevant and 3 means perfectly relevant. ## experiments in this section, we assess our proposed methods using two real-world, extensively used document ranking datasets to evaluate the following research questions (rq): rq1: how do llm-generated natural language explanations impact the calibration and ranking performance of neural rankers? rq2: is there a consistent improvement across different training objectives when using these explanations? rq3: does the mc sampled aggregate-nles from the llm yield empirical improvements? rq4: does the proposed methodology produce meaningful scores in a real-world setting? ## conclusions and future works this study tackled the challenge of scale calibration for neural rankers. our method leverages zero-shot llms' inherent understanding of textual data through nles to enhance scale calibration while maintaining or improving ranking performance. we showed that our approach surpasses established baselines in ranking and calibration metrics, demonstrates consistency across various training objectives, and is effective for downstream post-hoc qpp applications. this work prototypes the idea of leveraging llm-generated nles of query-document pairs to improve the scale calibration of neural ranking models. like many recent works that incorporate llms as an integral part of the document ranking process @xcite @xcite , our approach is not yet efficient enough to be directly applied to an online ranking system with high latency requirements. having validated the feasibility of nles for calibration, future work could incorporate techniques such as distillation to improve run-time efficiency.