Efficiently Tuned Parameters Are Task Embeddings (2022)

Abstract: Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets. However, it is computationally infeasible to experiment with all intermediate transfer combinations, making choosing a useful source task a challenging problem. In this paper, we anticipate that task-specific parameters updated in parameter-efficient tuning methods are likely to encode task-specific information. Therefore, such parameters can be predictive for inter-task transferability. Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer. We experiment with 11 text classification tasks and 11 question answering tasks. Experimental results show that our approach consistently outperforms existing inter-task transferability prediction methods while being conceptually simple and computationally efficient. Our analysis also reveals that the ability of efficiently tuned parameters on transferability prediction is disentangled with their in-task performance. This allows us to use parameters from early checkpoints as task embeddings to further improve efficiency.

## introduction the pretraining then fine-tuning paradigm @xcite @xcite @xcite @xcite has substantially improved the state-of-the-art on a wide range of natural language processing (nlp) tasks. in this paradigm, we first pretrain a large language model on large-scale corpora in a general domain, and then fine-tune the pretrained model to be a task-specific model on the target dataset. in addition to directly transferring from a general pretrained language model, prior work @xcite also shows that intermediate-task transfer, i.e., fine-tuning on intermediate source tasks before the target task, can further improve target task performance. however, the success of intermediate-task transfer heavily relies on the selection of a proper source dataset while an inappropriate source dataset often leads to performance degradation compared to plain finetuning. therefore, some recent works @xcite investigate methods to efficiently predict inter-task transferability without actually trying out all intermediate-task combinations. the current state of the art @xcite ) on predicting inter-task transferability is built on task2vec @xcite , which considers the fisher information matrix of a model finetuned on a task as the "task embedding", and predicts inter-task transferability by computing the cosine similarity between the task embedding of the source and target tasks. despite empirically performing well, this approach requires fine-tuning the full model and (inefficiently) computing the fisher matrix of the model. moreover, the resulting task embeddings generally have a high dimensionality similar to the size of the underlying model. therefore, intermediate task selection, which requires storing task embeddings for each source/target task, can be space-consuming, especially when experi-menting with large language models. in this work, we opt for parameter-efficient tuning approaches @xcite @xcite @xcite for the efficient and accurate prediction of inter-task transferability. our key insight is that task-specific parameters updated in parameter-efficient tuning methods are likely to encode high density task-specific information since they are used as a query for retrieving task-related knowledge in a frozen pretrained language model. therefore, we propose to directly use task-specific parameters learned via parameter-efficient tuning on source/target datasets as task embeddings, as shown in figure 1 . compared to task embeddings obtained by calculating the fisher matrix of the fine-tuned model @xcite , efficiently tuned parameters are of much lower dimensionality and do not suffer from noise from uninformative weights in the model parameters, thus leading to more accurate transferability prediction. also, our method only requires parameter-efficient tuning on the tasks and stores task-specific parameters, making both computing and storing task embeddings more efficient. moreover, with the development of open-source parameter-efficient tuning platforms like adapter-hub @xcite , we can easily access off-the-shelf parameters of the source and target datasets downloaded from the model zoo and then compute the similarity between the downloaded parameters. we empirically verify the effectiveness of our approach by experimenting with 11 text classification tasks and 11 question answering tasks, following @xcite . our results show that our approach consistently outperforms existing intertask transferability prediction methods while being simpler and more efficient. in addition, we find that the ability of efficiently tuned parameters on transferability prediction is not strongly correlated with their in-task performance. therefore, task-specific parameters tuned with a relatively small number of steps are already highly predictive for inter-task transferability, allowing us to further improve the efficiency of intermediate task selection. ## related work prior work @xcite shows that positive transfer can be elicited by training a model on intermediate source tasks before fine-tuning on the target task. however, the choice of an appropriate source task is crucial for effective transfer. @xcite show that the size of the source dataset is an good prior for source task selection. @xcite propose to use task requiring complex reasoning and inference as source tasks. besides these heuristics, a number of work also focuses on systematic prediction of intermediate task transferability. @xcite propose to used task2vec to construct task embeddings based on the input text or fisher information matrix of a fine-tuned model. @xcite further extend similar ideas for adapter-based transfer learning. more recently, @xcite explore prompt-based transfer and propose to use prompt similarity as a predictor for prompt transferability to select proper soft prompts for initialization. this can be viewed as a special case of our proposed method where the parameter-efficient tuning method is restricted to vanilla prompt tuning @xcite and the transfer method is restricted to prompt transfer instead of general intermediate-task transfer. ## conclusion in this paper, we show that efficiently tuned parameters are highly predictive for inter-task transferability and thus can be used as off-the-shelf task embeddings for source task selection in intermediatetask transfer learning. our empirical investigation with three parameter-efficient tuning methods on 22 nlp tasks demonstrates that our approach outperforms prior works on inter-task transferability prediction despite being more efficient.