Investigating Transferability in Pretrained Language Models (2020)

Abstract: How does language model pretraining help transfer learning? We consider a simple ablation technique for determining the impact of each pretrained layer on transfer task performance. This method, partial reinitialization, involves replacing different layers of a pretrained model with random weights, then finetuning the entire model on the transfer task and observing the change in performance. This technique reveals that in BERT, layers with high probing performance on downstream GLUE tasks are neither necessary nor sufficient for high accuracy on those tasks. Furthermore, the benefit of using pretrained parameters for a layer varies dramatically with finetuning dataset size: parameters that provide tremendous performance improvement when data is plentiful may provide negligible benefits in data-scarce settings. These results reveal the complexity of the transfer learning process, highlighting the limitations of methods that operate on frozen models or single data samples.

## introduction despite the striking success of transfer learning in nlp, remarkably little is understood about how these pretrained models improve downstream task performance. recent work on understanding deep nlp models has centered on probing, a methodology that involves training classifiers for different tasks on model representations @xcite @xcite @xcite @xcite . while probing aims to uncover what a network has already learned, a major goal of machine learning is transfer: systems that build upon what they have learned to expand what they can learn. given that most † atamkin@stanford.edu recent models are updated end-to-end during finetuning (e.g. @xcite @xcite , it is unclear how, or even whether, the knowledge uncovered by probing contributes to these models' transfer learning success. in a sense, probing can be seen as quantifying the transferability of representations from one task to another, as it measures how well a simple model (e.g., a softmax classifier) can perform the second task using only features from a model trained on the first. however, when pretrained models are finetuned end-to-end on a downstream task, what is transferred is not the features from each layer of the pretrained model, but its parameters, which define a sequence of functions for processing representations. critically, these functions and their interactions may shift considerably during training, potentially enabling higher performance despite not initially extracting features correlated with this task. we refer to this phenomenon of how layer parameters from one task can help transfer learning on another task as transferability of parameters. in this work, we investigate a methodology for measuring the transferability of different layer parameters in a pretrained language model to different transfer tasks, using bert @xcite as our subject of analysis. our methods, described more fully in section 2 and figure 1 , involve partially reinitializing bert: replacing different layers with random weights and then observing the change in task performance after finetuning the entire model end-to-end. compared to possible alternatives like freezing parts of the network or removing layers, partial reinitialization enables fairer comparisons by keeping the network's architecture and capacity constant between trials, changing only the parameters at initialization. through experiments across different layers, tasks, and dataset sizes, this approach enables us to shed light on multiple dimensions of the transfer learning process: are the early layers of the network more important than later ones for transfer learning? do individual layers become more or less critical depending on the task or amount of finetuning data? does the position of a particular layer within the network matter, or do its parameters aid optimization regardless of where they are in the network? we find that when finetuning on a new task: our first set of experiments aims to uncover how many pretrained layers are sufficient for accurate learning of a downstream task. to do this, we perform a series of incremental reinitialization experiments, where we reinitialize all layers after the kth layer of bert-base, for values @xmath0, 1, . . . 12}, replacing them with random weights. we then finetune the entire model end-toend on the target task. note that @xmath1, while @xmath2. we do not reinitialize the bert word embeddings. as bert uses residual connections @xcite around layers, the model can simply learn to ignore any of the reinitialized layers if they are not helpful during finetuning. we use the bert-base uncased model, implemented in pytorch @xcite via the transformers library @xcite . we finetune the network using adam (kingma and ba, 2015), with a batch size of 8, a learning rate of 2e-5, and default parameters otherwise. more de-tails about reinitialization, training, statistical significance, and other methodological choices can be found in the appendix. we conduct our experiments on three english language tasks from the glue benchmark, spanning the domains of sentiment, reasoning, and syntax @xcite : sst-2 stanford sentiment treebank involves binary classification of a single sentence from a movie review as positive or negative @xcite . qnli question natural language inference is a binary classification task derived from squad @xcite . the task requires determining whether for a given (question, answer) pair the question is answered by the answer. cola the corpus of linguistic acceptability is a binary classification task that requires determining whether a single sentence is linguistically acceptable @xcite . because pretraining appears to be especially helpful in the small-data regime @xcite , it is crucial to isolate task-specific effects from data quantity effects by controlling for finetuning dataset size. to do this, we perform our incremental reinitializations on randomly-sampled subsets of the data: 500, 5k, and 50k examples (excluding 50k for cola, which contains only 8.5k examples). the 5k subset size is then used as the default for our other experiments. to ensure that an unrepresentative sample is not chosen by chance, we run multiple trials with different subsamples. confidence intervals produced through multiple trials also demonstrate that trends hold regardless of intrinsic task variability. while similar reinitialization schemes have been explored by @xcite ; @xcite in computer vision and briefly by @xcite in an nlp context, none investigate these data quantity-and task-specific effects. figure 2 shows the results of our incremental reinitialization experiments. these results show that the transferability of a bert layer varies dramatically based on the finetuning dataset size. across all but the 500 example trials of sst-2, a more specific trend holds: earlier layers provide more of an improvement on finetuning performance when the finetuning dataset is large. this trend suggests that larger finetuning datasets may enable the network to learn a substitute for the parameters in the middle and later layers. in contrast, smaller datasets may leave the network reliant on existing feature processing in those layers. however, across all tasks and dataset sizes, it is clear that the pretrained parameters by themselves do not determine the impact they will have on finetuning performance: instead, a more complex interaction occurs between the parameters, optimizer, and the available data. ## does probing predict layer transferability? what is the relationship between transferability of representations, measured by probing, and transferability of parameters, measured by partial reinitialization? to compare, we conduct probing experiments for our finetuning tasks on each layer of the pretrained bert model. our probing model averages each layer's hidden states, then passes the pooled representation through a linear layer and softmax to produce probabilities for each class. these task-specific components are identical to those in our reinitialization experiments; however, we keep the bert model's parameters frozen when training our probes. our results, presented in figure 2 (dashed lines), show a significant difference between the layers with the highest probing performance and reinitialization curves for the data-rich settings (darkest solid lines). for example, the probing accuracy on all tasks is near chance for the first six layers. despite this, these early layer parameters exhibit significant transferability to the finetuning tasks: preserving them while reinitializing all other layers enables large gains in finetuning accuracy across tasks. interestingly, however, we observe that the smallest-data regime's curves are much more similar to the probing curves across all tasks than the larger-data regimes. smaller finetuning datasets enable fewer updates to the network before overfitting occurs; thus, it may be that finetuning interpolates between the extremes of probing (no data) and fully-supervised learning (enough data to completely overwrite the pretrained parameters). we leave a more in-depth exploration of this connection to future work. while the incremental reinitializations measure each bert layer's incremental effect on transfer learning, they do not assess each layer's contribution in isolation, relative to either the full bert model or an entirely reinitialized model. measuring this requires eliminating the number of pretrained layers as a possible confounder. to do so, we conduct a series of localized reinitialization experiments, where we take all blocks of three consecutive layers and either 1) reinitialize those layers or 2) preserve those layers while reinitializing the others in the network. 1 these localized reinitializations help determine the extent to which bert's different layers are either necessary (performance decreases when they are removed) or sufficient (performance is higher than random initialization when they are kept) for a specific level of performance. again, bert's residual connections permit the model to ignore reinitialized layers' outputs if they harm finetuning performance. these results, shown in figure 3 , demonstrate that the earlier layers appear to be generally more helpful for finetuning relative to the later layers, even when controlling for the amount of finetuning data. however, there are strong task-specific effects: sst-2 appears to be particularly damaged by removing middle layers, while the effects on cola are distributed more uniformly. the effects on qnli appear to be concentrated almost entirely in the first four layers of bert-suggesting opportunities for future work on whether sparsity of this sort indicates the presence of easy-to-extract features correlated with the task label. these results support the hypothesis that different kinds of feature processing learned during bert pretraining are helpful for different finetuning tasks, and provide a new way to gauge similarity between different tasks. we also investigate whether the success of bert depends mostly on learned inter-layer phenomena, such as learned feature processing pipelines (tenney et al., 2019a), or intra-layer phenomena, such as a learned feature-agnostic initialization scheme which aid optimization (e.g. @xcite . to approach this question, we perform several layer permutation experiments, where we randomly shuffle the order of bert's layers before finetuning. the degree that finetuning performance is degraded in these runs indicates the extent to which bert's finetuning success is dependent on a learned composition of feature processors, as opposed to providing better-initialized individual layers which would help optimization anywhere in the network. these results, plotted in figure 4 , show that scrambling bert's layers reduces their finetuning ability to not much above a randomly-initialized network, on average. this decrease suggests that bert's transfer abilities are highly dependent on the intra-layer interactions learned during pretraining. we also test for correlation of performance between tasks. we do this by comparing task-pairs for each permutation, as we use the same permutation for the nth run of each task. the high correlation coefficients for most pairs shown in ## conclusion we present a set of experiments to better understand how the different pretrained layers in bert influence its transfer learning ability. our results reveal the unique importance of transferability of parameters to successful transfer learning, distinct from the transferability of fixed representations assessed by probing. we also disentangle important factors affecting the role of layers in transfer learning: task vs. quantity of finetuning data, number vs. location of pretrained layers, and presence vs. order of layers. while probing continues to advance our understanding of linguistic structures in pretrained models, these results indicate that new techniques are needed to connect these findings to their potential impacts on finetuning. the insights and methods presented here are one contribution toward this goal, and we hope they enable more work on understanding why and how these models work.