T ransformer R anker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks (2025)

Abstract: Classification tasks in NLP are typically addressed by selecting a pre-trained language model (PLM) from a model hub, and fine-tuning it for the task at hand. However, given the very large number of PLMs that are currently available, a practical challenge is to determine which of them will perform best for a specific downstream task. With this paper, we introduce TransformerRanker, a lightweight library that efficiently ranks PLMs for classification tasks without the need for computationally costly fine-tuning. Our library implements current approaches for transferability estimation (LogME, H-Score, kNN), in combination with layer aggregation options, which we empirically showed to yield state-of-the-art rankings of PLMs (Garbas et al., 2024). We designed the interface to be lightweight and easy to use, allowing users to directly connect to the HuggingFace Transformers and Dataset libraries. Users need only select a downstream classification task and a list of PLMs to create a ranking of likely best-suited PLMs for their task. We make TransformerRanker available as a pip-installable open-source library.

## introduction there currently exists a multitude of pre-trained transformer language models (plms) that are readily available (e.g. through model hubs; @xcite . from a practical perspective, this raises the question of which plm will perform best once finetuned for a specific downstream nlp task. however, since fine-tuning a plm is both computationally costly and sensitive to hyperparameters (such as the learning rate used for fine-tuning), an exhaustive search of all models is infeasible. in practice, this restricts users to the exploration of only a small number of plms and may lead to the best-suited plm for a particular task not being found. (1) the user selects a downstream classification task by selecting a dataset from huggingface datasets. to address this issue, prior work proposed methods for transferability estimation. these methods avoid the high computational costs associated with fine-tuning a plm by keeping the internal states frozen. prominent examples of such methods include h-score @xcite and logme @xcite . in our prior work, we found that these methods can be improved by incorporating features from deeper layers in the estimation process @xcite . we empirically showed that this yields better per-model estimates since, depending on the downstream task, different layers in the transformer model are best suited to provide features. further, we showed that averaging across layers makes the selection process more robust against the different pre-training objectives used in each model, allowing a better comparison over a diverse set of plms. a library for transferability estimation. with this paper, we present transformerranker, a python library that enables users to leverage transferability estimation to identify the best-suited plm for a downstream classification task. with transformerranker, we consolidate work in layer-wise analysis and transferability estimation methods into a single library and provide a threestep interface for ranking any transformer lms available on the huggingface model hub (see figure 1 ). our goal is twofold: transformerranker supports nlp classification tasks of two main families: (1) text classification tasks such as question classification @xcite , sentiment analysis @xcite or textual entailment @xcite in which a classification decision is made for an entire text (or text pair), (2) sequence labeling tasks such as named entity recognition (ner; @xcite and part-of-speech tagging @xcite where classification decisions are made per-word. our library is built to rely solely on py-torch @xcite and huggingface @xcite ecosystems and can be integrated as a fast model selection step in a larger nlp pipeline. we make transformerranker publicly available as a pip-installable open-source project. ## transformerranker we give an overview of transformerranker by first discussing the installation procedure (section 2.1) and the standard three-step workflow for ranking transformer models (section 2.2). we then discuss the implemented estimators (section 2.3) and layer aggregation methods (section 2.4). ## experiments in this section, we present results of different ranking methods and evaluate how downsampling affects runtime and estimation quality. we further conduct experiments to measure runtime of our gpu-accelerated implementations. finally, we show a practical example of transformer-ranker. ## conclusion and outlook in this paper, we introduced transformer-ranker, a library that aids researchers and practitioners in systematically selecting well-suited plms for their downstream nlp tasks. in a fraction of the time, the underlying approach yields a ranking of candidate models that strongly correlates with models ranked by their true downstream performance. future work will extend the scope of the addressed downstream tasks and incorporate new methods for transferability estimation. for instance, we are currently adding experimental support for regression tasks (limited to logme and knn estimators). we invite the community to experiment with transformerranker by ranking different models on preferred datasets.