How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023)

Abstract: Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computational cost the performance of a model when transferred from a source task to a given target task. Considering the effectiveness of such estimations, the communities of natural language processing also began to study similar problems for the selection of pre-trained language models. However, there is a lack of a comprehensive comparison between these estimation methods yet. Also, the differences between vision and language scenarios make it doubtful whether previous conclusions can be established across fields. In this paper, we first conduct a thorough survey of existing transferability estimation methods being able to find the most suitable model, then we conduct a detailed empirical study for the surveyed methods based on the GLUE benchmark. From qualitative and quantitative analyses, we demonstrate the strengths and weaknesses of existing methods and show that H-Score generally performs well with superiorities in effectiveness and efficiency. We also outline the difficulties of consideration of training details, applicability to text generation, and consistency to certain metrics which shed light on future directions.

## introduction recent advances in the community of natural language processing (nlp) are heavily built on the effectiveness of pre-trained language models (plms), especially on large ones (llms) @xcite @xcite . as the number of available plms continually grows, a critical question arises: "which plm can make the performance of a downstream task best?". the fine-tuning result on a task usually varies across different plms, and this variation becomes more pronounced in low-resource scenarios @xcite . basically, the key to such model selection is to figure out the transferability between the model and the target task. pioneering works conducted finetuning on every candidate model in a brute-force manner @xcite . though the true fine-tuning performance can be obtained in this way, expensive parameter optimization is practically prohibitive @xcite . thus, there is an urgent need to quantify the transferability at a low cost of computation. to this end, transferability estimation (te), as an essential task of transfer learning (tl), has emerged as a key challenge with several solutions proposed in computer vision (cv) fields initially @xcite . recently, some of these remarkable approaches have also been applied to nlp tasks which show promising results on plm selection @xcite . despite a great number of surveys established for tl and plms @xcite @xcite , there is no comprehensive survey on te yet, especially with the purpose of plm selection. therefore, this paper aims to fill this gap by providing a comprehensive and well-structured summary of recent progress. to ensure comprehensive coverage, a multi-stage approach is employed to identify and select the studies included in this review. firstly, an extensive literature search was carried out using online databases, such as google scholar and dblp. the search terms used were carefully chosen to capture the key concepts and themes related to te and plms. after retrieving an initial pool of nearly 100 articles, a thorough screening of titles, abstracts, and keywords was conducted to exclude irrelevant studies, leading to a final selection of 20 studies that met the predetermined criteria for inclusion. based on these research, we present a method taxonomy. as shown in fig. 1 , according to the need for training on target task, we divide te methods into: (1) model similarity-based methods that assume the inter-model similarity reflects the transferability which require the model trained on target task @xcite . (2) training-free methods that accelerate the estimation process by computing metrics free of target model training to examine the compatibility of the plm's feature space on the target dataset @xcite . then we conduct qualitative analysis for the applicability and provide empirical results on the glue benchmark @xcite to manifest specific strengths and weaknesses in existing methods. we show that model similarity-based methods have the superiority of applicability to different target tasks, and training-free methods have the advantage over fast estimation. and for the methods simulating the dynamics of fine-tuning, they generally perform better. besides, we analyze some factors that can affect the estimation effectiveness and efficiency including task type, sample size, feature dimension, target model as well as sample affinity function. the empirical observations demonstrate that h-score @xcite generally shows desired usability. based on these investigations, we further exhibit some under-explored aspects to shed light on the future directions foot_0 . ## qualitative analysis to examine the applicability of each method, we qualitatively compare them as shown in task agnostic a widely applicable method should be able to deal with multiple task types such as classification, regression, and generation. currently, only model similarity-based methods satisfy this property. however, the inter-model similarity is only aware of sample features and does not consider the output space of the task. dataset |train| |dev| task metric sentence classification tasks cola 8.5k 1k acceptability mcc. sst-2 67k 872 sentiment acc. paraphrase tasks mrpc 3.7k 408 paraphrase acc. qqp 364k 40.4k paraphrase acc. inference tasks mnli 393k 9.8k nli acc. qnli 105k 5.5k qa/nli acc. rte 2.5k 277 nli acc. wnli 635 71 coref./nli acc. ## quantitative analysis effectiveness and efficiency the overall metric scores of all methods are reported in ## conclusion and future directions this paper reviews the recent advances in te that can be applied to plm selection and presents a method taxonomy based on a thorough analysis. moreover, comprehensive qualitative and quantitative comparisons between different approaches are provided to help understand their applicability in a number of aspects. we hope this survey can help people for the purpose of research or industry to choose desired plm by appropriate te methods. although lots of efforts have been made as surveyed, there still remain some directions that deserve further investigation: