Evidence > Intuition: Transferability Estimation for Encoder Selection (2022)

Abstract: With the increase in availability of large pre-trained language models (LMs) in Natural Language Processing (NLP), it becomes critical to assess their fit for a specific target task a priori—as fine-tuning the entire space of available LMs is computationally prohibitive and unsustainable. However, encoder transferability estimation has received little to no attention in NLP. In this paper, we propose to generate quantitative evidence to predict which LM, out of a pool of models, will perform best on a target task without having to fine-tune all candidates. We provide a comprehensive study on LM ranking for 10 NLP tasks spanning the two fundamental problem types of classification and structured prediction. We adopt the state-of-the-art Logarithm of Maximum Evidence (LogME) measure from Computer Vision (CV) and find that it positively correlates with final LM performance in 94% of the setups.In the first study of its kind, we further compare transferability measures with the de facto standard of human practitioner ranking, finding that evidence from quantitative metrics is more robust than pure intuition and can help identify unexpected LM candidates.

## introduction advances in deep learning-based nlp and cv build on expressive representations from encoder models pre-trained on massive corpora. downstream models make use of latent information in these representations to extract relevant features for the task at hand. within this paradigm, deciding which pre-trained encoder to use in any taskspecific architecture is crucial, however training a model using each encoder candidate is infeasible. in absence of prior heuristics (e.g., via related work), the choice of encoder has therefore prevailingly been based on practitioner intuition rather than quantitative evidence. the authors contributed equally to this work. in nlp, prior work has examined the different yet related task of performance prediction @xcite , surveyed and categorized lms @xcite , and used probing to predict lm performance specifically for dependency parsing @xcite , but has yet to extensively investigate how to rank the increasingly large number of pre-trained lm encoders across various tasks and domains. preliminary work by @xcite shows that the logme estimator holds promise, including the first steps for encoder selection in nlp. with their main focus being on cv, however, they evaluate only a limited set of tasks and models for nlp and use self-reported benchmark scores instead of running controlled experiments which should include, e.g., the variance across initializations, domains, and fine-tuning strategies (section 2). as such, we seek to answer: how well can we estimate the transferability of pre-trained lms to specific nlp tasks? to do so, we contribute: ## transferability estimation transferability estimation aims to quantify the ability of a model to transfer knowledge learned from one task to another @xcite . formally, given a pool of l pretrained lms {ϕ l } l @xmath0, we calculate a predictive score s l (d) for each ϕ l which ideally correlates with the model's final performance p l (d). s l (d) is computed without fine-tuning ϕ l on d such that the optimal ϕ * l can be chosen from a large model pool at a low computational cost. the cv community has begun to explore methods for encoder pre-selection and ranking through metrics such as logme and the log expected empirical prediction (leep; @xcite . these are widely-used state-of-the-art methods in cv. recent work introduced the gaussian bhattacharyya coefficient (gbc; @xcite and optimal transport based conditional entropy (otce; @xcite , the exploration of which we leave for future work. however, in the nlp field, related work focus on choosing a task and not an lm encoder for transferability @xcite , leaving the ranking of encoders an unexplored question. logme logme measures the suitability of all encoded dataset features @xmath1.g., embeddings with dimensionality h) to predict all scalar labels @xmath2. as this density is intractable, it is estimated by mapping @xmath3; this is akin to training a linear probe with optimal param-eters w * and using the likelihood p(y|f, w * ) as a proxy for feature suitability. because a simple linear model will overfit on the training data, it would be beneficial to obtain the marginal likelihood, or evidence, by integrating over all possible values of w: p(y|f ) = p(y|f, w)p(w)dw. to once again make this computation tractable, @xcite reformulate it as an efficient, iterative evidence maximization problem where both w as well as y are drawn from lightly parametrized, isotropic gaussian distributions. the normalized logarithm of the maximized evidence (logme) can then be used as s l (d) to rank encoder models directly. nlp setting logme has shown promise for cv, and an initial study on the glue benchmark @xcite indicate the same for nlp @xcite . however, for nlp, there are notable differences in setups across tasks. we adapt and apply logme extensively to a wide range of nlp settings to identify empirically grounded guidelines. in particular, we investigate variations concerning the task, instance granularity, domain, and tuning strategy. first, compared to most image classification tasks, nlp tasks are subject to differences in granularity, i.e., classification (c) and structured prediction (sp). furthermore, there is less clarity than for individual images as to which representation best captures the full language input @xcite . therefore, for c setups we experiment with two representations: i.e., using [cls]/<@xmath4. second, depending on differences in the data domain, nlp practitioners are often faced with a pool of domain-adapted lms in addition to more general-purpose encoders-the correct choice of which may not be immediately apparent. finally, the best performance in nlp is often achieved using full fine-tuning, while cv models usually do not fine-tune the encoder @xcite . it will therefore be crucial to investigate whether the predictive performance of s l (d) holds when it is computed based on untuned f while p l (d) is based on fully fine-tuned representations. ## experimental setup applying seven architecturally and domain-diverse pre-trained lms with up to four configurations each to 10 datasets and a wide variety of tasks, we investigate logme's predictive power for transferability estimation in nlp-for a total of 280 setups. we refer to language models we pick seven pre-trained lms with a wide domain and architectural variety from the transformers library's model hub @xcite . three are "generalpurpose" models, namely bert base @xcite , roberta base @xcite , and distilbert base @xcite . four models are pre-trained on domain-specific corpora, namely clinical-biobert @xcite , biobert @xcite , twitter-roberta base @xcite , and scibert base @xcite . note that for biobert variants domain-adaptive pre-training has been applied @xcite . ## analysis of results our results across all setups are consolidated in figure 1 and figure 2 (c: blue, sp: beige). 2,3 the left of each figure plots the performance using frozen lm embeddings ( ) against logme scores, while on the right, full lm fine-tuning is applied ( ). 4 figure 1 shows the results of using mean-pooled embeddings in both c/sp settings. for , we obtain ρ > 0.8 on 8/10 tasks and τ @xmath5.7 on 6/10 tasks, indicating a strong relationship between model performance and logme. after fine-tuning ( ), we observe a general reduction in ρ and τ w (most on crossner, en-ewt), however overall correlations remain positive to a significant degree. for c setups using the alternative [cls]/<@xmath6, logme correlates highly at ρ > 0.95 on 5/6 tasks and τ @xmath7.7 on 4/6 tasks when using head-only tuning ( ). after full fine-tuning ( ), scierc, rte and agnews have lower correlations, particularly with the high-variance roberta model. however, the remaining tasks maintain a stable correlation, with ρ > 0.6 and τ @xmath8.3 across 5/6 tasks. overall, logme has a positive correlation with final performance in 30/32 cases. in more detail, logme has a τ @xmath9.41 in 20/32 setups, meaning that selecting a higher ranked model is the better choice 71% of the time. logme both identifies intuitive, domain-specific scenarios (e.g., twitter-roberta performing well on airline twitter), but also finds cases that may be unintuitive, such as distilbert's occasionally high performance for crossner and jobstack. this finding holds across c, sp, domains as well as different input representations. for the latter, we note that, surprisingly, even the untuned representation of [cls]/<@xmath10. ## human performance given the lack of prior work examining transferability estimation of pre-trained lm encoders, the most common method for encoder selection employed today is practitioner intuition. as such, we conduct a small-scale study with 12 nlp practitioners and ask them to perform the same ranking as in section 3. despite having access to model details and associated papers, this task is difficult even for experts. while for logme, the range of τ w is in [-0.20; 1.00], human rankings fall into a wider range of [-0.54; 1.00], indicating higher uncertainty. similarly, we observe that human correlations are negative thrice as often as for logme. additionally, logme provides a continuous scale for comparing models, while human rankings offer no indication of relative performance differences. at the same time, they are more inaccurate for tasks without an associated domain-specific model (e.g., news, mixture of genres in ewt). moreover, even when domains are clear (e.g., twitter, science), logme tends to be more accurate than the predictions of most human participants. finally, the high variance between practitioners and the fact that no single person was an expert in all setups further reinforces the necessity of quantitative transferability scores. ## conclusion we show the value of transferability estimation for selecting high-performing lms before full model fine-tuning in experiments, covering the two fundamental nlp settings of classification and structured prediction. by adopting the stateof-the-art logme scoring method, we are able to rank lms on a continuous scale which correlates with final performance-with the better encoder being chosen in 71% of cases. additionally, we identify nlp-specific guidelines for transferability estimation: in particular, predicting the best lm for tasks/domains which greatly deviate from an encoder's pre-training setup and require large amounts of full fine-tuning may require larger pre-selections of lms due to the higher uncertainty of the scoring methods. finally, our human study showed that practitioners frequently misconstrue the performance of lms even on domain-specific tasks. as such transferability quantification methods provide valuable evidence over intuition.