When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022)

Abstract: Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL) to train jointly on a supplementary task and the target task (pairwise MTL), or simply using MTL to train jointly on all available datasets (MTL-ALL). In this work, we compare all three TL methods in a comprehensive analysis on the GLUE dataset suite. We find that there is a simple heuristic for when to use one of these techniques over the other: pairwise MTL is better than STILTs when the target task has fewer instances than the supporting task and vice versa. We show that this holds true in more than 92% of applicable cases on the GLUE dataset and validate this hypothesis with experiments varying dataset size. The simplicity and effectiveness of this heuristic is surprising and warrants additional exploration by the TL community. Furthermore, we find that MTL-ALL is worse than the pairwise methods in almost every case. We hope this study will aid others as they choose between TL methods for NLP tasks.

## introduction the standard supervised training paradigm in nlp research is to fine-tune a pre-trained language model on some target task @xcite @xcite . when additional non-target supervised datasets are available during fine-tuning, it is not always clear how to best make use of the supporting data @xcite @xcite . although there are an exponential number of ways to combine or alternate between the target and supporting tasks, three predominant methods have emerged: application papers that use these methods generally focus on only one method @xcite @xcite @xcite @xcite , while a limited amount of papers consider running two. those that do examine them do so with a limited number of configurations: @xcite examines stilts and one instance of mtl, @xcite ; @xcite ; @xcite compare mtl with mtl all , and @xcite ; @xcite ; @xcite ; @xcite use mtl all and stilts but not pairwise mtl. in this work we perform comprehensive experiments using all three methods on the 9 datasets in the glue benchmark @xcite . we surprisingly find that a simple size heuristic can be used to determine with more than 92% accuracy which method to use for a given target and supporting task: when the target dataset is larger than the supporting dataset, stilts should be used; otherwise, mtl should be used (mtl all is almost universally the worst of the methods in our experiments). to confirm the validity of the size heuristic, we additionally perform a targeted experiment varying dataset size for two of the datasets, showing that there is a crossover point in performance between the two methods when the dataset sizes are equal. we believe that this analysis will help nlp researchers to make better decisions when choosing a tl method and will open up future research into understanding the cause of this heuristic's success. ## results we provide three different analyses: a comparison of pairwise mtl vs stilts, experiments varying dataset size to validate our findings, and a comparison of pairwise approaches vs mtl all . ## related work a large body of recent work @xcite @xcite @xcite exists that examines when these transfer learning methods are more effective than simply fine-tuning on the target task. oftentimes, these explanations involve recognizing catastrophic forgetting @xcite @xcite although recent work has called for them to be reexamined @xcite . this paper is orthogonal to those, as we examine when you should choose mtl or stilts, rather than when they are more effective than the standard fine-tuning case (in fact, these strategies could be combined to predict transfer and then use the best method). as our task is different, theoretical explanations for how these methods work in relation to each other will need to be explored in future work. potential theories suggested by our results are discussed in appendix c, and are left to guide those efforts. ## conclusion we examined the three main strategies for transfer learning in natural language processing: training on an intermediate supporting task to aid the target task (stilts), training on the target and supporting task simultaneously (mtl), or training on multiple supporting tasks alongside the target task (mtl all ). we provide the first comprehensive comparison between these three methods using the glue dataset suite and show that there is a simple rule for when to use one of these techniques over the other. this simple heuristic, which holds true in more than 92% of applicable cases, states that multi-task learning is better than intermediate fine tuning when the target task is smaller than the supporting task and vice versa. additionally, we showed that these pairwise transfer learning techniques outperform the mtl all approach in almost every case. pairwise oracle uses the best supplementary task for the given target task using the best pairwise method (stilts or mtl). all scores are the average of 5 random seeds. note that mtl all was run with three different sampling methods (top half). we find that on almost every task, pairwise approaches are better than mtl all . bold scores indicate the best score in the column for the given section. wei zhu, xiaofeng zhou, k. wang, x. luo, xiepeng li, y. ni, and g. xie. 2019. panlp at mediqa 2019: pre-trained language models, transfer learning and knowledge distillation. in bionlp@acl.