Relevance Scores Calibration for Ranked List Truncation via TMP Adapter (2025)

Abstract: The ranked list truncation task involves determining a truncation point to retrieve the relevant items from a ranked list. Despite current advancements, truncation methods struggle with limited capacity, unstable training and inconsistency of selected threshold. To address these problems we introduce TMP Adapter, a novel approach that builds upon the improved adapter model and incorporates the Threshold Margin Penalty (TMP) as an additive loss function to calibrate ranking model relevance scores for ranked list truncation. We evaluate TMP Adapter’s performance on various retrieval datasets and observe that TMP Adapter is a promising advancement in the calibration methods, which offers both theoretical and practical benefits for ranked list truncation.

## introduction determining the appropriate truncation point is a fundamental problem in information retrieval and recommendation systems. an excessively long ranked list can overwhelm users with redundant or less relevant information. conversely, an overly short list risks omitting highly relevant items that could enhance user satisfaction. thus, optimizing the cutoff point is essential to balance relevance, diversity, and usability. the problem of determining the optimal cutoff point in a ranked list, also known as ranked list truncation or relevance filtering, has been approached using two primary methods: adaptive thresholding and global thresholding. adaptive thresholding focuses on predicting an optimal cutoff point for each individual list. bi-cut @xcite leverages a bidirectional lstm to model sequential dependencies and predict truncation points. choppy @xcite employs a transformer architecture for the same task. attncut @xcite further incorporates attention mechanisms and reward augmented maximum likelihood for direct optimization. lecut @xcite improves upon these by adding contextual features from the retrieval task to better model document semantics. in the realm of personalized recommendations, perk @xcite estimates the expected user utility to determine the ideal list size. more recently, genrt @xcite combines reranking and truncation in a joint model using sequence generation. global thresholding aims to calibrate relevance scores, enabling the use of a universal threshold across queries. this approach often involves transforming raw retrieval scores into more interpretable values. tcm @xcite introduces a margin-based loss that facilitates a consistent distance threshold and, rcr @xcite , a regression-compatible ranking approach, ensures alignment between ranking and regression objectives. jrc @xcite consolidates optimization across all samples using a contextualized hybrid model. the cosine adapter @xcite maps cosine similarity scores to interpretable relevance scores and surprise @xcite employs statistical methods to adjust a ranked list using. these methods contrast with adaptive thresholding by seeking a single, universally applicable cutoff. despite the promising progress, we discover that existing methods suffer from three main issues: (i) low capacity, especially for large language models. (ii) unstable training, especially for low-data training. (iii) threshold inconsistency especially in case of distribution shift between the training and test. we address these issues by proposing improved adapter architecture and training method with threshold margin penalty inspired by tcm. ## conclusion in this paper, we introduce threshold margin penalty adapter, a novel approach designed to calibrate ranking model relevance scores for ranked list truncation. proposed tmp adapter extends the improved adapter model by integrating the threshold margin penalty as an additive loss function. this innovation enhances the model's ability to maintain threshold consistency and improves the separation between positive and negative pairs, which is critical for effective ranking list truncation. we evaluate tmp adapter's performance on four datasets and observe a consistent and stable improvement in the f1-score, highlighting the model's effectiveness for score separation. additionally, we observe a significant enhancement in threshold consistency, which underscores the model's in-domain robustness to maintain reliable decision boundaries. these findings show that tmp adapter is a promising advancement in calibration methods, offering both theoretical and practical benefits for ranked list truncation.