What Happens To BERT Embeddings During Fine-tuning? (2020)

Abstract: While much recent work has examined how linguistic information is encoded in pre-trained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks. Using a suite of analysis techniques—supervised probing, unsupervised similarity analysis, and layer-based ablations—we investigate how fine-tuning affects the representations of the BERT model. We find that while fine-tuning necessarily makes some significant changes, there is no catastrophic forgetting of linguistic phenomena. We instead find that fine-tuning is a conservative process that primarily affects the top layers of BERT, albeit with noteworthy variation across tasks. In particular, dependency parsing reconfigures most of the model, whereas SQuAD and MNLI involve much shallower processing. Finally, we also find that fine-tuning has a weaker effect on representations of out-of-domain sentences, suggesting room for improvement in model generalization.

## introduction unsupervised pre-training of deep language models has led to significant advances on many nlp tasks, with the popular bert model @xcite and successors (e.g. @xcite dominating the glue leaderboard @xcite and other benchmarks over the past year. many recent works have attempted to better understand these models and explain what makes them so powerful. particularly, behavioral studies (e.g. @xcite , diagnostic probing classifiers (e.g. @xcite @xcite , and unsupervised techniques (e.g. @xcite have shed light on the representations from the pretrained models and have shown that they encode a wide variety of linguistic phenomena @xcite . however, in the standard recipe for models such as bert @xcite , after initializing with pre-trained weights, they are then trained for a few epochs on a supervised dataset. considerably less is understood about what happens during this fine-tuning stage. current understanding is based largely on the models' performance. while fine-tuned transformers achieve state-ofthe-art accuracy, they also can end up learning shallow heuristics @xcite @xcite , suggesting a disconnect between the richness of features learned from pre-training and those used by finetuned models. thus, in this work, we seek to understand how the internals of the model-the representation space-change when fine-tuned for downstream tasks. we focus on three widelyused nlp tasks: dependency parsing, natural language inference (mnli), and reading comprehension (squad), and ask: we approach these questions with three complementary analysis techniques. supervised probing classifiers @xcite @xcite provide a means of explicitly testing for the presence of pre-specified linguistic phenomena, while representational similarity analysis (rsa; @xcite gives a task-agnostic measurement of the change in model activations. finally, we corroborate our results with two types of layerbased ablations-truncation and partial freezingand measure their effect on end-task performance. taken together, we conclude that fine-tuning involves primarily shallow model changes, evidenced by three specific observations. first, linguistic features are not lost during fine-tuning but tasks can differ in how they either surface or obfuscate different phenomena. second, fine-tuning tends to affect only the top few layers of bert, albeit with variation across tasks: squad and mnli have a relatively shallow effect, while dependency parsing involves deeper changes to the encoder. we confirm this by partial-freezing experiments which test how many layers need to change to do well on each task and relate this to an estimate of task difficulty (with respect to the pre-training regime) via layer ablations. finally, we observe that fine-tuning induces large changes on in-domain examples, yet on out-of-domain sentences, the representations more closely resemble those of the pre-trained model. more generally, supervised probing models and diagnostic classifiers make few assumptions be-yond the existence of model activations and can test for the presence of a wide variety of phenomena. tenney et al. (2019b); liu et al. (2019); peters et al. (2018b) introduced task suites that probe for high-level linguistic phenomena such as partof-speech, entity types, and coreference, while @xcite showed that these phenomena are represented in a hierarchical order within the layers of bert. @xcite used a geometrically-motivated probe to explore syntactic structures, and @xcite and @xcite designed informationtheoretic techniques that can measure the model and data complexity. 1 while probing models depend on labelled data, parallel work has studied the same encoders using unsupervised techniques. @xcite used a form of canonical correlation analysis (pw-cca; @xcite to study the layerwise evolution of representations, while saphra and lopez (2019) explored how these representations evolve during training. @xcite used representational similarity analysis (rsa; @xcite to study the effect of context on encoder representations, while @xcite correlated them with syntax. fine-tuning comparatively few analyses have focused on understanding the fine-tuning process. initial studies of fine-tuned encoders have shown state-of-the-art performance on benchmark suites such as glue @xcite and surprising sample efficiency @xcite . however, behavioral studies with challenge sets @xcite @xcite have shown limited ability to generalize to out-of-domain data and across syntactic perturbations. van @xcite focused on question-answering models with taskspecific probes. @xcite analyzed the effects of fine-tuning with respect to the performance of diagnostic classifiers. @xcite studied fine-tuning via rsa, finding a significant divergence between the representations of models fine-tuned on different tasks. concurrent work by @xcite investigated the transferability of pre-trained language models and performed an number of layer ablations. consistent with our observations in section 5.2, they find 1 see @xcite and @xcite for a survey of probing methods. ## experimental setup bert we focus on the popular bert model @xcite , focusing on the 12-layer base uncased variant. 2 we denote the pretrained model as base and refer to fine-tuned versions by the name of the task. mnli a common benchmark for natural language understanding, the mnli dataset @xcite contains over 433k sentence pairs annotated with textual entailment information. we fine-tune bert using the architecture and parameters of @xcite , using a softmax layer on [cls] representation to predict the output label. across three trials, the evaluation accuracy of our bert base model is 83.3 ± 0.1, slightly lower but comparable to the published score of 84.6. ## where do the representations change? the supervised probes from the previous section are highly targeted: as trained models, they are sensitive to particular linguistic phenomena, but they also can learn to ignore everything else. if the supervised probe is closely related to the finetuning task-such as for syntactic probes and dependency parsing-we observe significant changes in performance, but otherwise we see little effect. nonetheless, we know that something must be changing during fine-tuning-at minimum because, as shown in @xcite , performance degrades significantly if the encoder is completely frozen. to explore this change, we turn to an unsupervised technique, representational similarity analysis (rsa; @xcite , which is sensitive to the global structure of the embedding space, and corroborate our findings with layer-based ablations. while these techniques are not targeted to specific linguistic phenomena, they do provide a powerful exploratory tool that can illuminate which parts of the model change and how they vary across datasets. ## out-of-domain behavior finally, we ask whether the effects of fine-tuning are general: do they apply only to inputs that look like the fine-tuning data, or do they lead to broader changes in behavior? this is usually explored by behavioral methods, in which a model is trained on one domain and evaluated on anotherfor example, the mismatched evaluation for mnli @xcite -but this analysis is limited by the availability of labeled data. by using rsa, we can test this in an unsupervised manner. we use rsa to compare the fine-tuned model to base and observe the degree of similarity when inputs are drawn from different corpora. we use random samples from the development sets for mnli (as premise [sep] hypothesis) and squad (as question [sep] passage) as in-domain for their respective models, 9 and as the out-of-domain control we use random wikipedia sentences (which resemble the pretraining domain). as in section 5.1, we use the representations of @xmath0. 10 results for the mnli and squad fine-tuned models are shown in figure 5 . although we see that all models diverge from bert base in the top layers, there is a significantly larger change in the representations on indomain examples. this suggests that fine-tuning is specific to the target domain. for other examples, such as the wikipedia sentences which resemble the pre-training data, the similarity score with bert base is much higher. this suggests that fine-tuning leads the model to change its representations for the new domain but to continue to behave more like the base model otherwise. this final result again shows that fine-tuning is conservative and suggests room for improvement in model generalization to out-of-domain sentences. ## conclusions in this paper, we employ three complementary analysis methods to gain insight into effects of fine-tuning on the representations produced by bert. from supervised probing analyses, we find that the linguistic structures discovered during pretraining remain available after fine-tuning, though this information is not strengthened by tuning on benchmark tasks such as mnli and squad. in light of prior studies @xcite which have shown that end-task models often fall back on simple heuristics, our results are especially interesting: they suggest that the model has the option of using stronger features, but chooses to use heuristics instead. next, our results using rsa and layer ablations show that the changes from fine-tuning alter a fraction of the model capacity, specifically within the top few layers (up to some variation across tasks). also, although fine-tuning has a significant effect on the representations of in-domain sentences, the representations of out-of-domain examples remain much closer to those of the pre-trained model. overall, these conclusions suggest that finetuning-as currently practiced-is a conservative process: preserving linguistic features, affecting only a few layers, and specific to in-domain examples. while the standard fine-tuning recipe undeniably leads to strong performance on many tasks, there appears to be room for improvement: an opportunity to refine this transfer step-potentially by utilizing more of the model capacity-to better the generalization and transferability. finally, in this work, we pulled from a range of analysis techniques to understand very finegrained aspects of model representations (via probing classifiers) and coarse-grained ones (via rsa). an important direction for future work is the development of new techniques which allow for more exploration of the middle ground. given available techniques, we can illuminate broadly that models are changing and test hypotheses about specific features (with probing tasks or attention analyses). new principled methods for discovering which features change will be invaluable for a deeper understanding of these models.