Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024)

Abstract: There currently exists a multitude of pre-trained transformer language models (LMs) that are readily available. From a practical perspective, this raises the question of which pre-trained LM will perform best if fine-tuned for a specific downstream NLP task. However, exhaustively fine-tuning all available LMs to determine the best-fitting model is computationally infeasible. To address this problem, we present an approach that inexpensively estimates a ranking of the expected performance of a given set of candidate LMs for a given task. Following a layer-wise representation analysis, we extend existing approaches such as H-score and LogME by aggregating representations across all layers of the transformer model. We present an extensive analysis of 20 transformer LMs, 6 downstream NLP tasks, and various estimators (linear probing, kNN, H-score, and LogME). Our evaluation finds that averaging the layer representations significantly improves the Pearson correlation coefficient between the true model ranks and the estimate, increasing from 0.58 to 0.86 for LogME and from 0.65 to 0.88 for H-score.

## introduction current state-of-the-art approaches for classification tasks such as sequence labeling or text classification rely on pre-trained transformer language models (lms; @xcite @xcite . these models are fine-tuned on task-specific training data. however, with the rising availability of such pre-trained lms (e.g. through model hubs; @xcite , this raises the practical challenge of choosing which pre-trained transformer to use for a specific task. however, exhaustively evaluating all available transformer lms is often infeasible due to the high computational costs of fine-tuning. further, since fine-tuning is sensitive to hyperparameters such as the learning rate, a hyperparameter search would need to be executed for each candidate lm. since the concrete choice of the pre-trained model can have a significant impact on the downstream performance, we argue that there is a need for accurate and computationally efficient transferability estimation. here, a given list of candidate lms is automatically ranked with regard to their expected performance on a downstream task. utilizing frozen representations. one promising line of research performs this estimation by extracting static representations from the lm for all data points in the training and testing data. by excluding the parameters of the transformer from further fine-tuning (thus "freezing" the weights of the transformer), the bulk of the computational costs are avoided. various estimators based on frozen features have been proposed. for instance, h-score @xcite considers the inter-class variance and feature redundancy, while the logarithm of maximum evidence (logme; @xcite assumes a linear transformation between representations and layers. the latter was empirically shown to be a strong baseline for task-specific ranking of lms @xcite . figure 1 illustrates how performance estimators such as logme or h-score are used: all task samples are encoded using the frozen lm. using the target labels, a score is estimated for each of the models from which a ranking is produced. contributions. building upon the insights from previous studies on transferability estimation, this work proposes methods to enhance the accuracy of such estimators. our improvements are derived from prior findings in language model probing @xcite , where it was shown that different encoder layers encode different information that can be very task-specific. they demonstrate that, given the variety of nlp datasets and language models available, relying solely on the features of the last layer (see figure 2a ) is insufficient. following this observation, this paper evaluates alternative strategies for extracting representations from frozen transformers with regard to their usefulness in transferability estimation. in more detail: our experiments show that a simple strategy of averaging representations significantly improves transferability estimation for all considered estimators. in particular, we find that the pearson correlation coefficient between the true model ranks and the estimate increases from 0.58 to 0.86 for logme, and from 0.65 to 0.88 for h-score. we believe that our results indicate the viability of using the h-score in combination with layerwise mean representations for accurate and computationally efficient transformer ranking. ## related work our work builds on previous research in the fields of pre-trained model selection and automated machine learning. the following subsections give a short introduction to each of these areas. ## performance estimation the goal of model selection is to estimate the performance of various language models on a given downstream task and rank them such that the order closely aligns with the true order of their downstream performance. in an ideal scenario, for any given classification dataset d, and a list of pretrained models {t m | @xmath0, . . . , m }, there would be a method capable of ranking these models without the necessity of fine-tuning each model t on d. this approach should offer not only a speed advantage over the exhaustive fine-tuning of all potential model candidates, but also maintain sufficient accuracy to reliably identify the most suitable models. ## experimental setup we fine-tune each model on specific downstream tasks to assess their performance. for each dataset, we arrange the models in ascending order by their fine-tuned scores to accurately determine their rankings. then, we compare the scores of each estimator by comparing them to the true ranking. in the following, we describe the selection of language models and datasets. language models. we select 20 pre-trained language models from the transformers @xcite library's model hub, expanding upon the most recent work by @xcite , which utilized 7 different models. the selection was broadened to include models that were pre-trained with other tasks than masked language modeling (mlm), such as deberta @xcite , spanbert @xcite , and two sentence-transformer models: all-mpnet-base and all-minilm @xcite . the complete list of models used in the experiments, along with their number of parameters and pre-training tasks, can be found in ## results and discussion we report the pearson ρ correlations between the true fine-tuned and the estimated performances in each of the setups in low performance when using the last layer. due to increasing the number of models to 20, we report a decreased correlation when only the last layer is used (compared to @xcite . when performing the best layer search and estimating each layer in language models, we notice that models such as electra and sentence-transformers underperform on ner tasks using the last layer compared to their deeper layers . this leads to an incorrect order of their final rankings (for detailed plots and best layer indices, see figure 4 and layer-aggregation improves all tested estimation methods. by selecting the best-performing layer within the encoder, we achieve an improvement in knn's performance by 0.54, elevating its ρ value from 0.14 to 0.68. similarly, logme's performance is enhanced by 0.26, increasing its ρ from 0.58 to 0.84 and h-score by 0.23 (from 0.65 to 0.88). the performance difference between the layer mean and the best layer is negligible. specifically, when using logme, the difference is merely 0.0086, with the layer mean surpassing the best layer in two of the six datasets (namely sciner and cola). thus, the layer mean approach is a viable alternative to selecting the best layer, offering comparable performance across datasets, with certain datasets showing benefits while for other datasets the performance difference remains insignificant. aggregating layer-wise representations using the layer mean also shows superiority against averaging between the scores of each layer with a difference of 0.88 vs. 0.82 for h-score, 0.86 vs. 0.80 using logme, and 0.63 vs. 0.63 for knn. since estimation with the layer mean representation is more time-efficient than identifying the optimal layer for each encoder or averaging layerspecific scores, as it requires a single estimation, the layer-mean representation appears to be the most promising. ## conclusion we propose the aggregation of layer-wise representations to enhance the performance of an existing encoder transferability estimation techniques h-score and logme. we demonstrate that any representation aggregation method (across all layers of the transformer model) can improve the ranking of pre-trained language models. the straightforward approach to average the representations across all layers (layer mean) improves the correlation coefficient between the true model ranks and the estimate by 0.13 for h-score and 0.28 for logme. searching for the layer with the best estimate is shown to perform slightly worse on average across all tasks. furthermore, it comes with the additional cost induced by the need to compute the score for each layer individually. we therefore recommend using the layer mean variant of h-score or logme.