Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERT oids’ Representations (2021)

Abstract: Most of the recent works on probing representations have focused on BERT, with the presumption that the findings might be similar to the other models. In this work, we extend the probing studies to two other models in the family, namely ELECTRA and XLNet, showing that variations in the pre-training objectives or architectural choices can result in different behaviors in encoding linguistic information in the representations. Most notably, we observe that ELECTRA tends to encode linguistic knowledge in the deeper layers, whereas XLNet instead concentrates that in the earlier layers. Also, the former model undergoes a slight change during fine-tuning, whereas the latter experiences significant adjustments. Moreover, we show that drawing conclusions based on the weight mixing evaluation strategy—which is widely used in the context of layer-wise probing—can be misleading given the norm disparity of the representations across different layers. Instead, we adopt an alternative information-theoretic probing with minimum description length, which has recently been proven to provide more reliable and informative results.

## introduction with the impressive success of pre-trained language models, such as bert @xcite , and their significant advances in transfer learning, a wave of interest has recently been directed toward understanding the knowledge encoded in their representations @xcite . one of the analytical tools which is widely used for this investigation is probing: training a shallow supervised classifier that attempts to predict specific linguistic properties or reasoning abilities, based on representations obtained from the model @xcite 2019b,a; @xcite @xcite @xcite . however, most of the previous studies have focused on bert only, neglecting other models in the family. this leaves open the question of how training objectives (which are fundamentally different for some models) and architectural choices would impact the resulting representations and the knowledge encoded in them. in this work, we carry out an analysis on three popular language models with totally different pretraining objectives: bert (masked language modeling), xlnet (permuted language modeling, @xcite , and electra (replaced token detection, @xcite . we also show that the "weight mixing" evaluation strategy of @xcite , which is widely used in the context of probing @xcite @xcite , inter alia), might not be a reliable basis for drawing conclusions in the layer-wise cross model analysis as it does not take into account the norm disparity across the representations of different layers. instead, we perform an information-theoretic probing analysis using minimum description length proposed by @xcite . based on a series of experiments, we find that language models derived from bert have different behaviors in encoding linguistic knowledge. specifically, we show that, unlike bert, xlnet encodes linguistic information in the earlier layers during pre-training, while electra tends to carry this information to the higher layers. we also extend our probing experiments to the finetuned setting to assess the extent of change in the encoded knowledge upon fine-tuning. using representation similarity analysis @xcite , we show that representations from the higher layers in xlnet-that do not well encode specific linguistic knowledge-undergo substantially bigger changes during fine-tuning when compared to the other models. in summary, our main contributions are as follows: ## background and pilot analysis in this section, we review bert and two of its popular derivatives: xlnet and electra, highlighting their differences as well as two commonly used probing methods that are essential to our discussion. we then conduct a pilot experiment to show the limitations of the evaluation metric used in edge probing and then justify our choice of an information-theoretic alternative. ## probing pre-trained representations we begin with probing the representations of each pre-trained model on a variety of core nlp tasks, including dependency labeling, named entity recognition, semantic role labeling, coreference resolution, and relation classification (see an example for each task in statistics of the datasets are provided in table a .1. for probing evaluation, we perform mdl probing on frozen representations obtained from each model. in order to obtain a fixed-length representation, following voita and titov (2020), we project the representations to 256-dimensional vectors, and then apply a self-attention pooling on them. foot_3 ## probing fine-tuned representations after observing that the amount and distribution of encoded linguistic knowledge can differ in these models, we aim to investigate how this information might be affected after the fine-tuning process. to this end, we repeated mdl probing as described in section 3 on the fine-tuned representations. next, to validate our results, we will expand our experiments to complementary analyses and measure the final quality of these representations on downstream tasks at different layers. we opted for the mnli @xcite dataset for fine-tuning all models. we used the same hyperparameters for fine-tuning all three models: 32 as the batch size, max length of 128, the learning rate of 2e-5, and five epochs of training. center of gravity in fine-tuned models. figure 4 demonstrates the difference in the average layer that most captures the information related to a particular task between the pre-trained and the fine-tuned models. we measure the difference of the two centers of gravity to evaluate the extent to which the concentration of knowledge ( ēc ) shifts for each model on a specific probing task after finetuning: first, we show that the concentration of information in fine-tuned models is usually in earlier layers compared to the pre-trained models. this can be attributed to the significant loss of linguistic knowledge in the final layers of fine-tuned models in favor of the specific information of the fine-tuning task. we show that xlnet in most tasks falls back to earlier layers than the two other models because it forgets the most linguistic knowledge in the final layers. this suggests that xlnet is going through a more extensive change in its representations which we investigate in the following sections. we also indicate that electra is falling back more than bert. we hypothesize that since electra focuses its most capable representations in the final layers, and those are the layers that change the most in fine-tuning, its center of gravity changes more than bert. full mdl compression and codelength results are reported in the appendix (table a .2). global rsa. after fine-tuning each model, we leverage representational similarity analysis (rsa) to investigate the overall amount of changes in the representations of each layer. rsa is a technique borrowed from neuroscience @xcite which is used for comparing two different representation spaces. to be specific, we sampled 5000 english sentences from opus dataset @xcite and used global rsa foot_5 , introduced by @xcite . this is a better choice since we used average pooling in our finetuning, discussed next. as a measure of intra-and inter-space similarities, we used cosine similarity and pearson correlation, respectively. figure 5 shows the results of the rsa measure applied to the representations of our models. we observe that xlnet has changed drastically during fine-tuning, while in bert and electra, only the top layers are primarily affected. we also see that bert shows a conservative pattern in the fine-tuning process which is consistent with findings of @xcite . as seen in figure 2 , we hypothesize that higher layers in xlnet are more open to change since they have relatively less specific knowledge. on the contrary, electra, which enjoys more linguistic information in its representations, especially in the higher layers, does not need to change very much. ## related work while extensive research has been devoted to probing bert @xcite @xcite , other popular models, such as xlnet and electra which have significant differences in their pre-training objectives, are less thoroughly investigated. only a few probing studies similar to our work exist which cover other models within the family. mosbach et al. (2020) mostly focused on the impact of pooling strategy in probing sentence-level tasks. based on the accuracy, they found that fine-tuning can affect the linguistic knowledge encoded in the representations. @xcite investigated the distribution of linguistic knowledge across individual neurons. in particular, they found that neurons in xlnet are more localized in encoding individual linguistic information compared to bert, where neurons are shared across multiple properties. by adopting the method of @xcite , @xcite investigated whether pre-trained language models encode semantic information, for instance by checking their representations against the lexico-semantic structure of wordnet @xcite . the above studies mainly rely on the accuracy metric for their probing evaluation, which is re-cently shown to fail in adequately reflecting the differences among representations @xcite . to our knowledge, this is the first time that an information-theoretic probing method is employed for conducting a cross-model and layerwise analytical study. ## conclusions in this paper, we aimed to extend probing studies on bert to the other models in the family to investigate how training objectives and architectural choices would affect the resulting representations and the linguistic knowledge encoded in them. to this end, we leveraged mdl probing method, which has recently proven to provide more reliable and informative results when compared with conventional probes. to the best of our knowledge, this is the first time mdl probing has been employed to analyze such state-of-the-art pre-trained language models as bert. by probing three stateof-the-art language models, i.e., bert, xlnet, and electra, we found considerable differences in the extent and distribution of the core linguistic knowledge in their representations. specifically, we demonstrate that xlnet accumulates linguistic knowledge in the earlier layers than bert, whereas that of electra is mainly in the final layers. moreover, from probing and employing rsa similarity measure on fine-tuned models, we illustrate that xlnet is more susceptible to forgetting linguistic knowledge in final layers and undergoes substantial adjustments to its representations when compared to the other models. based on differential downstream performance observations for before and after fine-tuning, we confirm that the changes in representations are proportional to the provided gain in the downstream task, which consequently indicates that xlnet injects more information during fine-tuning into its representations than the two other models.