Exploring and Predicting Transferability across NLP Tasks (2020)

Abstract: Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve performance? In this paper, we conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even with low-data source tasks that differ substantially from the target task (e.g., part-of-speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size. Overall, our experiments reveal that factors such as data size, task and domain similarity, and task complexity all play a role in determining transferability.

## introduction with the advent of methods such as elmo @xcite and bert @xcite , the dominant paradigm for developing nlp models has shifted to transfer learning: first, pretrain a large language model, and then fine-tune it on the target dataset. prior work has explored whether fine-tuning on intermediate source tasks before the target task can further improve this pipeline @xcite , but the conditions for successful transfer remain opaque, and choosing arbitrary source tasks can even adversely impact downstream performance @xcite . our work has two part of this work was done during an internship at microsoft research. given a target task, we first compute its task embedding and then identify the most similar source task embedding (in this example, wikihop) from a precomputed library via cosine similarity. finally, we perform intermediate fine-tuning of bert on the selected source task before fine-tuning on the target task. 1 main contributions: (1) we perform a large-scale empirical study across 33 different datasets to shed light on the transferability between nlp tasks, and (2) we develop task embeddings to predict which source tasks to use for a given target task. our study includes over 3,000 combinations of tasks and data regimes within and across three broad classes of problems (text classification, question answering, and sequence labeling), which is considerably more comprehensive than prior work @xcite @xcite . our results show that transfer learning is more beneficial than previously thought @xcite , especially for lowdata target tasks, and even low-data source tasks that are on the surface very different than the target task can result in transfer gains. while previous work has recommended using the amount of labeled data as a criterion to select source 1 credit to jay alammar for creating the bert image. motivated by these results, we move on to a more practical research question: given a particular target task, can we predict which source tasks (out of some predefined set) will yield the largest transfer learning improvement, especially in lowdata settings? we address this challenge by learning embeddings of tasks that encode their individual characteristics (figure 1 ). more specifically, we process all examples from a dataset through bert and compute a task embedding based on the model's gradients with respect to the task-specific loss, following recent meta-learning work in computer vision @xcite . we empirically demonstrate the practical value of these task embeddings for selecting source tasks (via simple cosine similarity) that effectively transfer to a given target task. to the best of our knowledge, this is the first work that builds explicit representations of nlp tasks to investigate transferability. we publicly release our task library, which consists of pretrained models and task embeddings for the 33 nlp tasks we study, along with a codebase that computes task embeddings for new tasks and identifies source tasks that will likely yield positive transferability. 2 ## exploring task transferability to shed light on the transferability between different nlp tasks, 3 we perform an empirical study with 33 tasks across three broad classes of problems: text classification/regression (cr), question answering (qa), and sequence labeling (sl). 4 in each experiment, we follow the stilts pipeline of @xcite by taking a pretrained bert model, 5 fine-tuning it on an intermediate source task, and then fine-tuning the resulting model on a target task. we explore in-class and out-ofclass transfer in both data-rich and data-constrained regimes and demonstrate that positive transfer can occur in a more diverse array of settings than previously thought @xcite . ## predicting task transferability the above analysis suggests that no single factor (e.g., data size, task and domain similarity, task complexity) is predictive of transfer gain across all of our settings. given a novel target task, how can we identify the single source task that maximizes transfer gain? one straightforward but extremely expensive approach is to enumerate every possible (source, target) task combination. work on multitask learning within nlp offers a more practical alternative by developing feature-based models to identify task and dataset characteristics that are predictive of task synergies @xcite . here, we take a different approach, inspired by recent computer vision methods @xcite , by computing task embeddings from layerwise gradients of bert. our approach generally outperforms baseline methods that use the data size heuristic @xcite and the gradients of the learning curve @xcite in terms of selecting the most transferable source tasks across settings. ## related work we build on existing work in exploring and predicting transferability across tasks. transferability between nlp tasks: sharing knowledge across different tasks, as in multitask/transfer learning, often improves over standard single-task learning @xcite . within multitask learning, several works (e.g., @xcite @xcite combine multiple tasks for better regularization and transfer. more related to our work, @xcite explore intermediate fine-tuning and find that transferring from data-rich source tasks boosts target task performance for text classification, while @xcite observe transfer gains between related sequence labeling tasks. expanding from single to multi-source transfer, @xcite show that pretraining on multiple datasets improves generalization on qa tasks. nevertheless, exploiting synergies between tasks remains difficult, with many combinations of tasks negatively impacting downstream performance @xcite @xcite , and the factors that determine successful transfer still remain murky. concurrent work indicates that intermediate tasks that require high-level inference and reasoning abilities tend to work best @xcite . ## conclusion we conduct a large-scale empirical study of the transferability between 33 nlp tasks across three broad classes of problems. we show that the benefits of transfer learning are more pronounced than previously thought, especially when target training data is limited, and we develop methods that learn vector representations of tasks that can be used to reason about the relationships between them. these task embeddings allow us to predict source tasks that will likely improve target task performance. our analysis suggests that data size, the similarity between the source and target tasks and domains, and task complexity are crucial for effective transfer, particularly in data-constrained regimes. in this work, we experiment with 33 datasets across three broad classes of problems (text classification/regression, question answering, and sequence labeling). below, we briefly describe the datasets, and summarize their characteristics in