Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022)

Abstract: Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.

## introduction following the wide success of unsupervised language model pre-training @xcite @xcite , recent work on transfer learning has shown that additional supervised multi-task training further improves performance on various downstream nlp tasks @xcite @xcite . there are two distinct ways in which the supervised data has been used: increasing the scale of the multi-task step to incorporate more tasks @xcite and developing task similarity metrics to incorporate tasks related to the target task @xcite . @xcite show that a multi-task training step, or pre-finetuning step, with a sufficiently large, diverse set of tasks is an effective taskagnostic second stage of model pre-training before finetuning on target tasks. in particular, they find that using a large number of tasks (e.g., roughly 15 tasks) is crucial in achieving good downstream performance, while pre-finetuning with fewer tasks causes a small performance drop. meanwhile, work on task transferability @xcite has shown that the choice of individual intermediate tasks significantly affects downstream fine-tuning performance-predicting the transfer is challenging and there is high variance depending on the choice of intermediate task. this motivates us to ask the question if prefinetuning on a small group of tasks related to the target tasks can obtain comparable performance to large-scale multi-task training. in this work, we present an empirical study to answer this question by extending the task transferability experiment to groups of tasks. we follow the two-step experimental pipeline from @xcite , where a pre-trained model is first pre-finetuned on a set of tasks and then separately finetuned on various target tasks on which we report performance. in addition, we group our set of 29 pre-finetuning tasks based on task format into 3 groups-classification tasks, sequence labelling tasks and extractive question answering tasks (figure 1 ). we perform model pre-finetuning on every combination of these task groups and report performance on target tasks that belong to each group. this allows us to systematically study how the size of the multi-task step and the choice of pre-finetuning tasks affects downstream task performance. we observe that, on average, large-scale multitask pre-finetuning results in improved performance on downstream target tasks. we also see that a model trained on related foot_0 pre-finetuning tasks obtains comparable downstream task performance to the large-scale model, at a reduced computational cost foot_1 , but pre-finetuning on an unrelated grouping pre-trained model multi-task pre-finetuning target task finetuning pre-finetuning tasks target task data report performance figure 2: high level workflow of the multi-task setup. our findings show the interplay between multitask scaling and task selection-when the target tasks are unknown then multi-task scaling is an effective, if expensive, intermediate step but if the goal is to improve performance on a specific target task set then multi-task training on a smaller set of related tasks is an effective cheaper alternative. these results also hint at the need to better study modeling techniques that mitigate negative transfer between tasks in large-scale multi-task training. ## multi-task setup our multi-task experiments follow the two step approach from @xcite , shown in figure 2 . a pre-trained model is first pre-finetuned on a set of tasks to obtain a single shared encoder. this model is then finetuned separately on various target tasks on which we report performance. grouping of tasks figure 1 shows our list of 29 pre-finetuning tasks and 12 unseen target tasks (listed in appendix a). informed by prior work @xcite @xcite , we divide these datasets into three groups based on task format-sequence labelling (sl), extractive question answering (qa) and classification (c). as noted in @xcite , grouping of nlp tasks is an imperfect heuristic. prior work @xcite formalizes the notion of task similarity using learned task embeddings so an another in terms of computational cost when it involves multitask training on fewer examples (appendix b.3) alternate formulation would be to divide the tasks into groups based on these learned embeddings. in this work we focus on a simple, intuitive grouping based on the task format, or output space, of the task. ## results a large-scale target task-agnostic multi-task step improves downstream performance from related tasks transfer better to identify the role of transferability, we aggregate the results on target tasks based on task groups in tasks interact differently, so selecting an optimal subset is hard when we look at prefinetuning runs on pairs of task groups taken together, we see that the sl+c and qa+c prefinetuning setups perform worse on qa and sl tasks than even the baseline model but the sl+qa setup is competitive with the best sl+qa+c setup across all unseen tasks. this shows that selecting an optimal combination of tasks can be challenging based on task group heuristics. @xcite also observed a similar result that, at larger scales, a random subset of tasks often outperforms we also report the average over all tasks and the average of the standard deviation values in separate rows for analysis. we observe the effect of scale by seeing that the sl+qa+c setup has the best average performance across all tasks. we see that multi-task training results in reduced variability across multiple runs. ## conclusion in this work, we bring together the lines of exploration on transferability of tasks and large-scale multi-task training. our results show that when the target tasks are unknown then multi-task scaling offers an effective way to obtain good downstream performance but if the goal is to improve performance on a specific target task set then a smaller set of related tasks is an effective, cheaper alternative. we observe that task groups interact differently when combined and that selecting an optimum subset becomes harder as the size increases. we also see that variability across multiple random restarts decreases on related target tasks and also reduces on increasing the size of the multi-task step.