T ask W eb: Selecting Better Source Tasks for Multi-task NLP (2023)

Abstract: Recent work in NLP has shown promising results in training models on large amounts of tasks to achieve better generalization. However, it is not well-understood how tasks are related, and how helpful training tasks can be chosen for a new task. In this work, we investigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TaskWeb, a large-scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design a new method TaskShop based on our analysis of TaskWeb. TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target task, and to choose a subset of helpful training tasks for multi-task training. Our method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. We also use TaskShop to build much smaller multi-task training sets that improve zero-shot performances across 11 different target tasks by at least 4.3%.

## introduction recent studies have revealed that large language models are able to generalize to unseen tasks when jointly trained on many different tasks, with their performance scaling to the size and diversity of the training data @xcite @xcite @xcite . as more and more tasks are added to build general-purpose models, it has been noted that knowing inter-task relationships may be helpful but that it remains unclear how to select helpful tasks for multi-task learning @xcite @xcite . in this work, we investigate whether quantifying the relationship between different nlp tasks via 1. assign score from source task to target task 2. select helpful source tasks for multi-task learning pairwise task transfer helps task selection, which we define as choosing one or more source tasks that better initialize a model for an unseen target task as shown in figure 1 . we begin from a pairwise setup as it is often used to quantify task relationships @xcite and is more tractable than larger combinations of tasks. first, we construct taskweb, a large-scale benchmark for pairwise task transfers across different model architectures (encoder-only, decoderonly, encoder-decoder), parameter count (60m to 770m) and adaptation methods including finetuning, adapter-tuning @xcite and @xcite , resulting in 25,000 transfers. from our results, we discover a transitive property where having strong, positive transfers @xmath0, b and c makes it more likely that @xmath1. then, we introduce a new method taskshop that predicts the transferability from a source task to a target task associated with only a few examples. taskshop builds upon the transitive behavior to construct different paths with "pivot" tasks between the source and target tasks. it combines taskweb scores between the source and pivot and textual similarity scores between the pivot and target to estimate (@xmath2. we evaluate our methods in both single-task and multi-task settings. first, we show that taskshop assigns better transferability scores both in terms of the overall ranking and identifying top helpful tasks. then, we demonstrate that models trained on small multi-task sets built with taskshop outperform models trained on larger sets of tasks. we perform additional analyses and discover that there is a tradeoff for building multitask sets of varying sizes with taskshop, and that the proportion of helpful tasks in the training set affects performance. to summarize, our contributions are as follows: ## background and overview we use pairwise task transfer to quantify task similarities, select better source tasks for unseen tasks and improve performance via multi-task finetuning. ## conclusion in this work, we investigate how using prior knowledge of task relationships quantified via pairwise task transfer aids selecting helpful source tasks for multi-task nlp. we build taskweb, a benchmark and repository of pairwise task transfers across different tasks, models and adaptation methods in nlp. based on our analysis of taskweb, we propose taskshop, our method for selecting helpful source tasks for a new target task. we show that taskshop outperforms existing methods in choosing helpful source tasks for different target tasks. moreover, we use taskshop and taskweb to build small multi-task training sets and outperform other methods that use much larger training sets.