Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024)

Abstract: Intermediate task transfer learning can greatly improve model performance. If, for example, one has little training data for emotion detection, first fine-tuning a language model on a sentiment classification dataset may improve performance strongly. But which task to choose for transfer learning? Prior methods producing useful task rankings are infeasible for large source pools, as they require forward passes through all source language models. We overcome this by introducing Embedding Space Maps (ESMs), light-weight neural networks that approximate the effect of fine-tuning a language model. We conduct the largest study on NLP task transferability and task selection with 12k source-target pairs. We find that applying ESMs on a prior method reduces execution time and disk space usage by factors of 10 and 278, respectively, while retaining high selection performance (avg. regret@5 score of 2.95).

## introduction the current default approach for supervised learning in nlp involves directly fine-tuning a pretrained transformer using labeled data of the target task. however, prior work showed that in some cases it is beneficial to perform two consecutive fine-tunings in a row: first, on an intermediate task, and then on the target task @xcite . this may be particularly effective when little training data exists for the target task, while much exists for the intermediate task. however, whether and how much performance is gained with intermediate task transfer learning heavily depends on the chosen intermediate task. worse, finding the best intermediate task for a given target task is a non-trivial problem given the large amount of labeled datasets that exist for nlp. for instance, the huggingface hub alone contains more than 160k datasets and 700k models. this renders an exhaustive search for the best possible intermediary task infeasible. the problem of finding promising intermediate tasks for a target task is called intermediate task selection. 1 prior work investigates approaches for finding suitable intermediate tasks given a source transformer lm and a target task @xcite @xcite @xcite . these approaches rely on the local availability of (large) source models or a space-intense representation of source datasets. methods also require resource-intensive computation for each sourcetarget pair @xcite . thus, most approaches are infeasible in real-world scenarios, i.e., with large source pools and constrained resources. while the large pool of available models and datasets is a valuable resource, user cannot optimally utilize it @xcite . this paper makes two contributions. first, we propose embedding space maps (esms), linear transformations of the embedding space, to be used in combination with logme @xcite , a source selection method that achieves high selection performance but suffers from its dependency on forward passes through each source model. we overcome this by approximating the embeddings of fine-tuned language models with esms. the resulting source selection method esm-logme reduces execution time by a factor of 10 and disk space usage by a factor of 278, and thus enables efficient source selection also on large source pools. second, we compare the performance of esm-logme to prior methods in the to date largest study on transferability across nlp tasks with more than 1.5k source datasets from huggingface hub and 8 target datasets across several task types and languages. the results show that esm-logme is the best-performing source selection method that is feasible in real-world scenarios. we release source code and all resources under the apache 2 license. our python package also allows to share and find esms to facilitate efficient source selection among researchers and practitioners. the repository is available at: https://github.com/davidschulte/hfdataset-selector 2 related work transfer learning is a common paradigm in nlp. with bert, @xcite propose encoders that are trained on a large corpus and then finetuned on individual target tasks. phang et al. (2018) show that language models can benefit from adding an intermediate fine-tuning step. this procedure is called intermediate task transfer learning. one of its challenges is finding the right intermediate task. source selection methods determine transfer suitability of source tasks for a given target task. the resulting rankings enable users to pick the potentially best sources, e.g., to perform transfer learning using these top picks to find the actual best source. methods typically consist of two resource-intense phases. in a one-time process, for each source a target-independent representation is created (p1). then, for a given target task, a ranking is produced using these representations (p2). textemb @xcite and taskemb @xcite embed datasets into a vector space and compute the distance of their representations. taskemb shows good performance in the literature, but its vector representation is in general as large as the language model itself. the vectors produced by textemb are small, but describe only the task domain and not the relation of inputs and labels. nce @xcite , leep (nguyen et al., 2020), and logme @xcite rank source models by evaluating pseudo-labels, their distributions, and target embeddings. these methods show state-of-the-art performance in source selection, but require forward passes through each source model. for scenarios with many source models, such approaches may thus be infeasible. prior studies evaluate the effect of intermediate task transfer learning and the performance of source selection methods on nlp tasks @xcite @xcite . but these studies do not represent real-world scenarios. employed source pools are small (@xmath0, whereas users can and have to choose from a very large pool of source datasets and models. also, studies largely do not evaluate execution time and disk space usage. we argue that efficiency is crucial for practical use of source selection methods. in sum, prior work can achieve high ranking accuracy, but does not explore source selection in real-world scenarios. studies largely neglect efficiency and use benchmarks that do not resemble source pools available on popular model hubs. ## embedding space maps fine-tuning a base language model f 0 on a task t using dataset d t results in a fine-tuned language model f t , which embeds texts differently than the base language model. we describe this effect as a function h 0→t on the embedding space and approximate this function using a neural network, which we call esm, ϕ 0→t . specifically, an esm's inputs are embeddings produced by the base model f 0 and its outputs are d-dimensional approximations of the embeddings produced by the fine-tuned model f t . attaching this network to a base model allows us to approximate how the respective fine-tuned model embeds a given text x. for phase p1 (cf. section 2), i.e., to train an esm, we embed each text x of a dataset d with both f 0 and f t . we train the esm ϕ 0→t by using the resulting embeddings (f 0 (x), f t (x)) as train examples. although any dataset could be used in this step, we choose to embed d t (the dataset that f 0 was fine-tuned on to obtain f t ), as its input texts describe task t and its effect on the language model best. 2 for p2, we once compute embeddings for the inputs of a target task using the base model and transform them using an esm for each intermediate task. following this, we rank sources by the logme score of their esm-transformed embeddings and target labels. we call this workflow esm-logme. since esms approximate the embeddings produced by the intermediate model, esm-logme can be viewed as an approximation of logme. we design esms as a single forward layer to minimize their size and computational complexity. therefore, esms are linear transformations. this design choice greatly reduces the amount of parameters needed to describe fine-tuning f 0 on a task, e.g., from 110m to less than 0.6m for bert. thus, esms drastically reduce compute cost and disk space usage of source representations. since h 0→t is the result of changing (many) parameters inside the language model, ϕ 0→t underfits this function. our evaluation shows that-albeit their simplicity-esms can encode abstract characteristics of their corresponding task. for a more intuitive understanding of the concept of esms, we visualize how well they approximate the effect of fine-tuning in an experiment (see figure 2 ). esms are parameter-efficient representations of transfer learning that are attached to a base language model. this modular design of esms resembles that of adapters (houlsby et al., 2019; pfeiffer 2 we train esms for 10 epochs with a learning rate of 0.001 and weight decay of 0.01. et al., 2020). while adapter blocks are inserted between transformer layers of a language model, esms are placed solely on top. using adapters for source selection requires a forward pass through the entire language model for each source. this also holds for state-of-the-art methods such as log-me. in contrast, with esms, only a single forward pass through the base language model is required to compute the base embeddings of the target task. these can then quickly be transformed using an esm for each intermediate task. in turn, esms significantly decrease computational effort in p2. ## experimental setup in contrast to prior studies, we aim to evaluate ranking performance in a real-world scenario. we parse datasets from the huggingface hub and heuristically determine their input and label columns to gather as many intermediate tasks as possible. this process includes searching for common column names, analyzing column types and contents foot_1 . the resulting pool consists of 1553 datasets (1496 classification and 57 regression tasks). we manually curate a selection of target datasets that is diverse as to task type, domain, and language. it consists of datasets or subsets from imdb @xcite , tweeteval with emotion and sentiment subsets (tes, tss) @xcite , j-sts @xcite , multi-dimensional gender bias classification (mdgb) @xcite , the english subset of paws-x @xcite , query wellformedness (gqw) @xcite , and civil comments (gcc) @xcite . 4 we artificially reduce the train size of target datasets to 1k rows to simulate data scarcity and of source datasets to 10k rows for evaluation efficiency. we use bert (bert-base-multilingual-uncased), perform transfer learning for all source-target pairs 5 , and evaluate the rankings of several source selection methods using the realized performance gains on a validation dataset as ground truth. 6 we calculate source rankings using esm-logme, logme, nce, leep, textemb, taskemb, vocabulary overlap (jacard index of the sets of tokenized inputs), and fine-tuning source models while freezing the parameters of the language model. model performance is measured in accuracy for classification tasks and as mean of pearson correlation and spearman's rank corr. coefficient for regression. we follow prior work @xcite and measure the quality of source rankings using ndcg @xcite and regret@k @xcite with @xmath1, 3, 5 (all reported as pp.). effectively, r@k expresses how well the best task in the selected k tasks performs compared to using the best task from the entire pool. the metric assumes that users employ transfer learning on all k selected tasks to find the actual best from those. we use r@5 as the primary metric. ## conclusion we show that a linear transformation of the embedding space suffices to describe a source task well enough for source selection. although esm-logme yields less accurate rankings than logme, our results show that the esm-logme workflow performs well on most target tasks (avg. regret@5 score of 2.95). at the same time, esm-logme is substantially more efficient than all wellperforming state-of-the-art methods. this makes it the best-performing source selection method that is feasible in a real-world scenario.