Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020)

Abstract: While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate-target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level inference and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abilities such as coreference resolution. However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings.

## introduction unsupervised pretraining-e.g., bert @xcite or roberta @xcite -has recently pushed the state of the art on many natural language understanding tasks. one method of further improving pretrained models that has been shown to be broadly helpful is to first finetune a pretrained model on an intermediate task, before fine-tuning again on the target task of interest @xcite @xcite , also referred to as * equal contribution. stilts. however, this approach does not always improve target task performance, and it is unclear under what conditions it does. this paper offers a large-scale empirical study aimed at addressing this open question. we perform a broad survey of intermediate and target task pairs, following an experimental pipeline similar to @xcite and @xcite . this differs from previous work in that we use a larger and more diverse set of intermediate and target tasks, introduce additional analysis-oriented probing tasks, and use a better-performing base model roberta @xcite . we aim to answer the following specific questions: the first question is the most straightforward: it can be answered by a sufficiently exhaustive search over possible intermediate-target task pairs. the second and third questions address the why rather than the when, and differ in a crucial detail: a model might learn skills by training on an intermediate task, but those skills might not help it to succeed on a target task. our search for intermediate tasks focuses on natural language understanding tasks in english. in particular, we run our experiments on 11 intermediate tasks and 10 target tasks, which results in a total of 110 intermediate-target task pairs. we use 25 probing tasks-tasks that each target a narrowly defined model behavior or linguistic phenomenonto shed light on which skills are learned from each intermediate task. our findings include the following: (i) natural language inference tasks as well as qa tasks which involve commonsense reasoning are generally useful as intermediate tasks. (ii) socialiqa and qqp as intermediate tasks are not helpful as a means to teach the skills captured by our probing tasks, while finetuning first on mnli and cosmosqa result in an increase in all skills. (iii) while a model's ability to learn skills relating to input-noising correlate with target task performance, low-level skills such as knowledge of a sentence's raw content preservation skills and ability to detect various attributes of input sentences such as tense of main verb and sentence length are less correlated with target task performance. this suggests that a model's ability to do well on the masked language modelling (mlm) task is important for downstream performance. furthermore, we conjecture that a portion of our analysis is affected by catastrophic forgetting of knowledge learned during pretraining. ## experiments training and optimization we use the largescale pretrained model roberta large in all experiments. for each intermediate, target, and probing task, we perform a hyperparameter sweep, varying the peak learning @xmath0, 1 × 10 -5 , 5 × 10 -6 , 3 × 10 -6 } and the dropout @xmath1.2, 0.1}. after choosing the best learning rate and dropout rate, we apply the best configuration for each task for all runs. for each task, we use the batch size that maximizes gpu usage, and use a maximum sequence length of 256. aside from these details, we follow the roberta paper for all other training hyperparameters. we use nvidia p40 gpus for our experiments. a complete pipeline with one intermediate task works as follows: first, we fine-tune roberta on the intermediate task. we then fine-tune copies of the resulting model separately on each of the 10 target tasks and 25 probing tasks and test on their respective validation sets. we run the same pipeline three times for the 11 intermediate tasks, plus a set of baseline runs without intermediate training. this gives us 35×12×3 = 1260 observations. we train our models using the adam optimizer (kingma and ba, 2015) with linear decay and early stopping. we run training for a maximum of 10 epochs when more than 1,500 training examples are available, and 40 epochs otherwise to ensure models are sufficiently trained on small datasets. we use the jiant @xcite ) nlp toolkit, based on pytorch @xcite , hugging face transformers @xcite , and allennlp @xcite , for all of our ## related work within the paradigm of training large pretrained transformer language representations via intermediate-stage training before fine-tuning on a target task, positive transfer has been shown in both sequential task-to-task @xcite and multi-task-to-task @xcite formats. @xcite perform an extensive study on transfer with bert, finding language modeling and nli tasks to be among the most beneficial tasks for improving target-task performance. @xcite perform a similar cross-task transfer study on reading comprehension datasets, finding similar positive transfer in most cases, with the biggest gains stemming from a combination of multiple qa datasets. our work consists of a larger, more diverse, set of intermediate task-target task pairs. we also use probing tasks to shed light on the skills learned by the intermediate tasks. among the prior work on predicting transfer performance, bingel and søgaard (2017) is the most similar to ours. they do a regression analysis that predicts target-task performance on the basis of various features of the source and target tasks and task pairs. they focus on a multi-task training setting without self-supervised pretraining, as opposed to our single-intermediate task, three-step procedure. similar work @xcite has been done on cross-lingual transfer-the analogous challenge of transferring learned knowledge from a highresource to a low-resource language. many recent works have attempted to understand the knowledge and linguistic skills bert learns, for instance by analyzing the language model surprisal for subject-verb agreements (goldberg, 2018), identifying specific knowledge or phenomena encapsulated in the representations learned by bert using probing tasks @xcite @xcite @xcite , analyzing the attention heads of bert @xcite @xcite , and testing the linguistic generalizations of bert across runs @xcite . however, relatively little work has been done to analyze fine-tuned bert-style models @xcite . ## conclusion and future work this paper presents a large-scale study on when and why intermediate-task training works with pretrained models. we perform experiments on roberta with a total of 110 pairs of intermediate and target tasks, and perform an analysis using 25 probing tasks, covering different semantic and syntactic phenomena. most directly, we observe that tasks like cosmos qa and hellaswag, which require complex reasoning and inference, tend to work best as intermediate tasks. looking to our probing analysis, intermediate tasks that help roberta improve across the board show the most positive transfer in downstream tasks. however, it is difficult to draw definite conclusions about the specific skills that drive positive transfer. intermediate-task training may help improve the handling of syntax, but there is little to no correlation between target-task and probing-task performance for these skills. probes for higherlevel semantic abilities tend to have a higher correlation with the target-task performance, but these results are too diffuse to yield more specific conclusions. future work in this area would benefit greatly from improvements to both the breadth and depth of available probing tasks. we also observe a worryingly high correlation between target-task performance and the two probing tasks which most closely resemble roberta's masked language modeling pretraining objective. thus, the results of our intermediate-task training analysis may be driven in part by forgetting of knowledge acquired during pretraining. our results therefore suggest a need for further work on efficient transfer learning mechanisms.