Where to start? Analyzing the potential value of intermediate models (2023)

Abstract: Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

## introduction finetuning pretrained models @xcite , is currently the standard and best approach for adjusting such models to perform a downstream task @xcite . the resulting finetuned models are typically used for inferring the labels of new examples that are reminiscent of the data used for finetuning. however, it was shown @xcite that finetuned models, trained on some source dataset, may represent better base models, namely a better starting point for a new finetuning process on a desired target dataset. this scheme, often referred to as intertraining, is the focus of the present work. given a target dataset, one may wonder what could be the intertraining gain, to determine whether it is worthwhile spending resources on selecting a base model. assuming the potential gain is high, the following natural question is which base models are most promising, out of countless options available through hubs huggingface (e.g. @xcite . we propose pragmatic methods to answer both questions, supported by extensive experiments. we begin with two observations: (i) some target datasets are intertraining-sensitive, i.e., have the potential to gain significantly from intertraining, while others are not, and are typically indifferent to the base model selection. furthermore, revealing this property of the target dataset can be done efficiently, by examining the gains obtained when using a single representative base model as a starting point; (ii) some base models are of high quality, i.e. finetuning on them provides consistent improvements on target datasets, but most base models are inferior and degrade performance. furthermore, ranking base models by quality can be done on one target task -and efficiently, via linear probing, namely training only the base model classification head, over a single representative dataset. thus, we argue that a preferable base model can be selected independently of the target dataset. this is in contrast to the common perception (c.f. §7) that the alignment of the target dataset and the source dataset -used to generate the base modelis a major factor in determining intertraining success. we substantiate our observation of independence by conducting experiments on a comprehensive set of target datasets and base models, comprising models obtained under controlled conditions as well as models from huggingface. in addition to these findings, we analyze attributes of the source and target datasets that affect gains ( §6). as some models are just better, not due to the choice of a current dataset, it makes sense to rank the models once and pick the best ones. but even ranking a thousand models is costly. in §8, we rely on our analysis to propose a practical approach to efficiently select models in a real-world setting. moreover, instead of expecting others to rank the models, we share an updating site site featuring the best models currently found. so far, we tested over 2.5k models. ## preliminaries in this paper, we use the following terminology. a dataset is a set of examples and labels. our goal is to maximize accuracy on the test of the target dataset, "target" for short. we discuss the difference between domain, task, and dataset in app. a. a pretrained (pt) model is a self-supervised model, e.g., roberta @xcite . a finetuned model is a pt model that was further trained over some source dataset, denoted henceforth as "source". we assume access to many such models, e.g., through huggingface. a base model can be either a pt model or a finetuned model. when finetuning over the target train data, one can start from any base model. intertraining refers to starting from a finetuned model as a base model, and in this case, we refer to this base model as an intermediate model. we denote by s t m , the accuracy score obtained over the target test set, t, after finetuning some base model m over the target train set. the intertraining gain of model m w.r.t. using the pt model, is thus defined via gain (m, t) = s t ms t p t . note that the gain may be negative. given a set of intermediate models, @xmath0. . . m n , the intertraining max-gain is defined as max @xmath1. thus, theoretically, max-gain is achieved by finetuning all the available intermediate models and picking the one best performing on the target test set. to avoid overfitting and reduce costs, our goal is to find an intermediate model with a gain that is as close as possible to the max-gain, without explicitly finetuning all the intermediate models. ## experimental setup our experimental setup is described next. the parameters for reproducibility are detailed in app. b. ## results most models are worse than pt and about 1 in 6 are better, providing positive intertraining gain. the in-house models/targets results are depicted in fig. 1 and stds and reference results in app. §d. app. §e reports results with off-the-shelf roberta and t5 intermediate models. the rows and columns in fig. 1 are similarly ordered -first the general datasets, then the nli datasets, and last the twitter datasets. loosely speaking, we do not recognize an approximate green block structure across the diagonal; namely, we do not observe clear intertraining gain for similar tasks (nli); nor for similar domains (twitter). however, some columns and some rows depict higher intertraining gains, while for others, the impact is minor. taken together, these observations suggest little dependence between the source used to generate the intermediate model and the performance over the target. this is in contrast to the common assumption ( §7) that the source and target need to be similar for intertraining to work. next, we delve deeper into these observations. ## source and target interaction analysis next, we analyse the interaction between the source dataset and the target dataset. obviously, such interaction may impact the gain. on one extreme, since finetuning on the same data twice does not improve performance, intertraining is not valuable when the source and target data are identical. on another extreme, consider partitions of the same data. obviously, training over half the data as the source would be beneficial for the other half, the target. thus, we do not question that interaction may exist, but rather investigate how common and strong it is. interaction between dataset groups. our general dataset group consists of diverse datasets, while the other groups share a domain (twitter) or a task (nli). we analyze whether datasets that share a domain or task with the target represent better source datasets than others. comparing sources (table rows), while nli is best improved by nli models, nli models im-general nli twitter general -0.37 -2.68 -0.54 nli 1.26 0.63 -0.03 twitter -0.4 -2.39 0.53 twitter models, however, do seem to improve twitter targets more (anova @xmath3.01), by 1 point. hence, it seems the effects are mixed. in summary, as a group, a similar source tends to yield greater gains than an unrelated source. however, in the rest of this section, we find this effect is of secondary importance to predicting model gains. a similar source may be more beneficial than a random one, but a well chosen source produces larger benefits regardless of similarity. symmetry as a similarity bound. we consider dataset similarity from another perspective. similarity is a symmetric relation. hence, if sourcetarget similarity was a main reason for gains, we would expect that when the source a helps the target b, the source b would help a as well. we assess the symmetry of the in-house models/targets experiment. we find that gains are far from symmetric (details in app. §g). thus, (symmetric) similarity seems like a weak explanation of which source data would help which target. 3 regression. as additional support for the relatively small impact of the source-target interaction, we show that the interaction is hardly needed to predict the scores. specifically, a simple regressor can approximate the gain without considering such interactions. the regression fits 3 sets of coefficients. for each target two coefficients t i , t ′ i -which one may think of as capturing average gain and sensitivity to inter-training, per target; and for each base model b j -which one may think of as capturing average inter-training gain, per base model. we then define gai n(base i , target j ) = (b i + t j ) t ′ j where i and j are the base model and target indices, respectively. note that by construction, this regressor has 2 ṅ + n parameters, while trying to model n ṅ interactions; thus, it does not have enough degrees of freedom to explicitly model all source/target interactions. still, it obtains satisfactory performance, as shown next. as a baseline, we fit the same regressor after randomly shuffling all the gains in the experiment. this shuffling ensures no information comes from the specific source and target, while maintaining the gain value distribution. we minimize mean squared error (mse) and fit with sgd until convergence. before we present the results, it would be beneficial to give some intuition on mse. as mse is the square of the error, an average mse of 4, means that the average prediction was accurate up to 2 ( √ 4) points on average or more accurate than that (as outliers have more effect on mse than on the average error rate). we obtain an mse of 4.2 on the in-house models/targets experiment (3.3), versus 9.6, σ = 0.9 when the gains are shuffled. thus, knowing the source id and the target id provides significant information about the expected gain, with no need of keeping a designated parameter for each source-target combination. we further compare this regressor with two other regressors, one that considers only base model information (gai n(base i , target j ) = b i ) and one that considers only target related information (gai n(base i , target j ) = t j ). the mse fit is 10.4 and 8.2, respectively, compared to 10.8, σ = 0.4 on shuffled gains. this suggests both the source and the target impact the intertraining gain, with potentially somewhat stronger influence by the target. in conclusion, our observations suggest that when considering intertraining potential, loosely speaking it is enough to consider two separate issues -(i) choosing a base model; and (ii) determining the expected effect on the target. ## factors contributing to gains so far, we showed the effects of the intermediate models are largely separate from those of the target. we proceed to examine how specific factors in each contribute to intertraining gains. ## related work various works use intertraining, often following the assumption of task alignment necessity @xcite @xcite , namely, that the source acts as weak labeled data @xcite . while we consider intertraining through finetuning, adaptation to the target @xcite or the domain @xcite was also suggested. such adaptation may be applied to any base model, and is complementary to the choice among base models. the need for alignment was also previously debated in the context of pretraining tasks @xcite @xcite . the properties of intertraining were studied in other contexts. @xcite suggested the intertraining scheme. @xcite probed the linguistic knowledge changes after intertraining, noting correlations to some target tasks, and hypothesized why some source tasks are good for intertraining. @xcite ; @xcite replicated the existence of good sources, but rejected the hypothesis. others tried to find which tasks have an affinity to each other @xcite @xcite , or when to prefer multitask @xcite . we study a much larger number of sources and targets, aiming to describe their natural distribution (c.f. §9) and also find that while specific choice may help, given enough models, it is safe to identify the models that just excel generally. recent work focuses on fusing multiple base models rather than picking just one @xcite @xcite . we expect our understanding to aid in choosing a subset of models to fuse as well. multi-task learning is another related field. it studies finetuning on different tasks at once @xcite and recently also a way to recycle models to do so was proposed @xcite . in contrast to our analysis, similarity between tasks aids multi-task learning @xcite . deciding which information should accompany publications is an active research field, covering datasets @xcite , human annotation sheets @xcite , and models @xcite . our work proposes to report upon sharing a model the aspects shown to affect its quality, such as lp (m, m n li). for datasets, we propose to report intertraining sensitivity. ## practical recommendations based on the observations ( §4) and analysis ( §5), we propose a methodology for efficiently utilizing intertraining in real-world settings. we suggest to collaboratively rank all available base models for intertraining and to utilize this list whenever intertraining is applied to a new target. new model. when a new model becomes available, we encourage practitioners to assess and share its quality. this can be done efficiently by linear probing on mnli ( §4.2) or comprehensively (app. §f) by finetuning on various datasets. we created statically ranked lists for robertabase and t5-small in app. §e. moreover, we apply our methods to many architectures and 36 test sets in an updating site , reporting the best models. new target. when considering intertraining on a new task, we recommend checking the target dataset sensitivity, and then choosing the base model. since the model's rank hardly depends on the target dataset, we suggest using the static ranking. specifically, we propose to finetune the top-ranked model, and compare the results to those obtained when finetuning the respective pt model. assuming the gain justifies it, one should consider a few additional intermediate models, in descending order, according to the allocated resources. ## discussion in §8, we highlighted our practical recommendations for intertraining. those, together with a systematic analysis of what affects intertraining, cover our main contributions. we hope this analysis would also aid future work on interactions between datasets; intertraining practices; and reusing finetuned models. our experiments mainly characterize what is probable rather than what is possible. we do not create specific models or aim to improve a specific task. instead, we investigate what is likely to be found in typical practical settings. accordingly, our findings are probabilistic in nature: most models are not beneficial as intermediate models, but there are enough that are. mostly, beneficial models are beneficial for many targets. as a side effect, we do identify specific strong source models. mnli was already considered a beneficial source dataset @xcite , a finding which we corroborate in a comprehensive manner. furthermore, when considering off-theshelf models we find models that outperform it (e.g., sts-b based for roberta, and quora for t5). to facilitate finding additional and better base models we will continuously report in the website website the best models per architecture. ## limitations one obvious limitation is that our work is empirical in nature. as such, we report observations, sampled or common, with no theoretical guarantees, and one should recognize the existence of exceptions. specifically, even though we have not observed it -there might exist target tasks that benefit greatly from a certain type of intermediate models; or intermediate models that help greatly in many targets while degrading performance in others. moreover, while testing 22 source datasets, many of which previously untested, we did not find a new top source for intertraining. the best one we found for roberta was already known to be good (mnli; @xcite . with that, by checking dozens of offthe-shelf models, we did uncover new intermediate models that seem to outperform mnli (e.g., sts-b for roberta and qqp for t5 -c.f. app. §e). more work is needed to assess the potential intertraining gain of the available datasets and models. we ran thousands of finetuning experiments, spanning a vast number of tasks and base models. thus, while we believe it is unlikely that reproducing our experiments will result in different outcomes, the large scale of our experiments places a heavy burden on trying to replicate our results. moreover, the off-the-shelf models used in our experiments might not be hosted publicly in the future. another limitation is that we could not upload all of the models to a shared location. this project was very computationally demanding, but more than that, it was demanding in terms of disk space, hence we had to delete many models along the way. finally, for practical reasons, our results are limited to classification tasks in english. we hope that future work will aim to test our conclusions beyond this scope. overall, in the space of classification, we see our results as robust, testing on 22 datasets (about double the amount of previous works @xcite ). we hope the diversity of targets brings large enough differences between datasets that the results would hold in other scopes.