Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking (2024)

Abstract: Text ranking has witnessed significant advancements, attributed to the utilization of dual-encoder enhanced by Pre-trained Language Models (PLMs). Given the proliferation of available PLMs, selecting the most effective one for a given dataset has become a non-trivial challenge. As a promising alternative to human intuition and brute-force fine-tuning, Transferability Estimation (TE) has emerged as an effective approach to model selection. However, current TE methods are primarily designed for classification tasks, and their estimated transferability may not align well with the objectives of text ranking. To address this challenge, we propose to compute the expected rank as transferability, explicitly reflecting the model’s ranking capability. Furthermore, to mitigate anisotropy and incorporate training dynamics, we adaptively scale isotropic sentence embeddings to yield an accurate expected rank score. Our resulting method, Adaptive Ranking Transferability (AiRTran), can effectively capture subtle differences between models. On challenging model selection scenarios across various text ranking datasets, it demonstrates significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption.

## introduction ranking relevant documents based on user queries has gained prominence as an essential component for information retrieval (ir) @xcite , retrieval-augmented generation (rag) @xcite , etc. as the volume of online content continues to grow, developing efficient and effective text ranking techniques has become increasingly imperative. fortunately, the advancements in pre-trained language models (plms) have spurred the development of dual-encoder capable of meeting these demands @xcite . usually, practitioners select the plm to be em-ployed by their intuition. however, such decisionmaking becomes unreliable as the number of plms rapidly grows @xcite , introducing a new challenge: the accurate and automatic selection of the most potent plm to achieve the desired performance on given datasets, namely model selection (ms) @xcite . early approaches to ms adopted a brute-force method, fine-tuning each model to identify the optimal one @xcite , which suffers from huge computing costs. hence, it is preferable to estimate model fine-tuning result with computational efficiency. in pursuit of this goal, transferability estimation (te) has garnered increasing attention in recent years, merely leveraging modelencoded features and labels @xcite . currently, advances in te have primarily been developed for classification tasks @xcite . these methods approximate the loglikelihood from features to labels, aligning with the classification objectives well and demonstrating promising results on classification datasets @xcite . however, the text ranking task fundamentally differs from classification task in which the matching scores for relevant query-document pairs are expected to be higher than those for irrelevant pairs @xcite . as a result, applying current classification-oriented te methods directly to text ranking is sub-optimal @xcite . typically, a plm well-suited to a dataset is expected to generate well-distributed sentence embeddings, where matched pairs exhibit high matching scores and vice versa, even before fine-tuning @xcite . hence, the expected rank of documents relevant to queries resulted from initial model can serve as explicit and consistent indicators of model fine-tuning performance in text ranking, in contrast to the classification-oriented te methods. however, a challenge arises from the anisotropy that usually exists in plms from small size to large size, manifesting as entangled feature dimensions and a biased distribution of sentence embeddings @xcite . in this case, even if two sentences are very similar in semantics, their vector representations may be far apart or exhibit directional differences. as a consequence, it leads to inaccurate document rank @xcite and hinders the alignment of expected rank scores with ranking transferability. in solving the anisotropy problem, previous research has proposed isotropization methods such as bert-flow @xcite , bert-whitening @xcite , and simcse @xcite , aiming to transform sentence embeddings into an isotropic space. considering bert-flow and simcse both need careful training @xcite , we select bertwhitening in this work since it only involves efficiently whitening the sentence embeddings. however, the isotropization overlooks the training dynamics involved in adapting model to the text ranking task @xcite , hindering the full expression of the fine-tuning ranking capabilities. to address this issue, we introduce an adaptive scaling mechanism after obtaining isotropic sentence embeddings, aiming to simulate the training dynamics by a simple but effective scaling operation, whose scaling weight can be efficiently solved by the least square method @xcite . then, our approach allows for a more precise reflection of the model's true transferability through improved expected rank scores. in summary, we present a promising alternative to human intuition for ms in text ranking: the adaptive ranking transferability (airtran). in this approach, the expected rank is computed and further enhanced by the proposed adaptive isotropization (adaiso) to estimate ranking transferability. on five widely studied datasets with two challenges pools of small and large candidate plms, covering various text ranking scenarios and dual-encoder backbones, our method shows superior efficiency compared to brute-force fine-tuning and show promising performance compared to previous classification-oriented te methods, human intuition, and chatgpt. ## related work early text ranking methods were primarily based on bag-of-words functions @xcite , which fall short of comprehensively understanding semantic relevance. in recent years, with the advancements in plms, dual-encoder has demonstrated superiority in capturing semantic interaction that goes beyond simple word overlap @xcite . prior research on dualencoder has predominantly focused on improving aspects such as sentence representations @xcite @xcite , negative document sampling @xcite @xcite , matching score computation @xcite @xcite , and supervisory signals construction @xcite @xcite , etc. unlike the above research focus on improving the network structure or training approach, our work explores selecting the most powerful plm to achieve superior fine-tuning performance. the most direct approaches to ms involve either exhaustively trying all available models @xcite , or training a performance predictor to estimate how well a given model will perform on a dataset @xcite , while they are less practical due to the large demands for model training. to address this issue, several te methods have been developed for efficient ms, primarily in classification tasks @xcite . these methods, when provided with model-encoded features and corresponding labels, quantify the degree of compatibility between them, treating this as a measure of transferability. subsequently, they rank candidate models based on their transferability scores to identify the bestperforming model. some of these approaches are grounded in the assumption that an effective model should produce features with a high degree of class separability @xcite @xcite ). another line of research focuses on the approximation of likelihood from features to labels @xcite @xcite @xcite . inspired by these remarkable works, this paper focuses on the selection of text ranking model which is an underexplored and challenging area. ## methodology as illustrated in figure 1 , we introduce the air-tran approach, motivated by the observation that expected rank score is inherently well-aligned with text ranking performance. to further improve this alignment, we propose adaiso to mitigate the anisotropy problem and adapt raw sentence embeddings to the downstream text ranking task. as a result, the resultant expected rank effectively captures the true ranking transferability of the model. in the following sections, we will first describe how to compute expected rank and then elaborate on how adaiso further achieves the enhancement. ## conclusion given the limitations of classification-oriented te methods, we propose the use of airtran in this study. it demonstrates superior ms performance compared to competitive te methods, human experts, and the chatgpt ms agent, indicating its potential as a solution for ms in text ranking tasks. we hope that our work will offer text ranking practitioners valuable guidance and inspiration when selecting models for their datasets of interest in addition to their intuition.