From BERT ‘s P oint of V iew: R evealing the P revailing C ontextual D ifferences (2022)

Abstract: Though successfully applied in research and industry large pretrained language models of the BERT family are not yet fully understood. While much research in the field of BERTology has tested whether specific knowledge can be extracted from layer activations, we invert the popular probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space. By extracting coarse features from masked token representations and predicting them by probing models with access to only partial information we can apprehend the variation from ‘BERT’s point of view’. By applying our new methodology to different datasets we show how much the differences can be described by syntax but further how they are to a great extent shaped by the most simple positional information.

## introduction by taking on the perspective of bert and presenting the methodology to explore this point of view we contribute a new approach to bertology research, a field that has emerged for a number of good reasons. ever since the original bert paper @xcite combined masked language modeling (mlm) with massive pretraining and the transformer architecture @xcite , models of the bert family have achieved a variety of new natural language processing (nlp) benchmarks. while their success is driven by the contextualization of words it is clear that these models do not yet have a real understanding of language @xcite . still, researchers are struggling to find out what it is exactly that they learn and how they perform so well. challenges are the high number of parameters over which the model knowledge is distributed, and the innumberable different patterns the models can potentially gather from text. bertology takes on this quest of understanding the inner workings of these large pretrained models to drive further improvements and identify the next steps towards general ai. though the training of ever greater models with ever more data has been criticized because of the societal costs and risks these models bring with, including bias and discrimination @xcite . nevertheless, because of their high performance they are already employed in research but also industryapplications, which exacerbates the need for their explainability. the black box of bidirectional encoder representations from transformers (bert) and its relatives commonly consists of 6-24 identical transformer encoder layers. each layer comprises a multi-head-attention block followed by a fully connected block, with both being bypassed by residual connections @xcite . this stack of layers is primarily pretrained with mlm, the task of predicting a randomly masked (or replaced) word in an input text, and can afterwards be finetuned to specific tasks. next to attention scores, layer activations are a popular choice for analysis, as they conflate the information from attention heads and skip-connections and represent the stages of the contextualization process. bert layer activations are most prominently scrutinized by the so-called edge probing design, in which they are treated as the fixed input to another neural network trained on specific nlp tasks @xcite . previous research has employed this method to test them for information on word senses or grammatical properties, but this does not reveal how much the information shapes the space. by inverting the probing design we present a new way to analyze layer activations, specifically their prevailing patterns, complementing the existing methodology. in contrast to the regular process, we do not use them as input but as the output of a model. we reduce their dimensionality by clustering and principal component extraction to capture the predominating differences within a dataset. by declaring these differences as our ground truth and explaining them in a second step, we can render visible the salient patterns and groupings from 'bert's point of view'. we chose this term to signal the shift of the perspective, from any contextual information a human might deem important to the information that actually predominates the representative space of the models. for different datasets we extract the layer activations of masked tokens, meaning that all analyzed tokens start out with identical representations. with this setup we can be sure that the differences we analyze are derived from context and are not caused by different pretrained embeddings. as masked language modeling continues to be a popular and successful pretraining objective, it is of particular interest which patterns are exploited when a model is determining the identity of a masked token. for the prediction of the representative features we train three types of probing models that receive as input a simplified version of the token context: bag-of-words, ordered part-of-speech tags and simply the position of the token in a sentence. by the disentanglement of these information types, the probing results provide indications about their importance for shaping the representative space. social science shows that contrastive explanations are more relevant to humans than complete explanations @xcite , which affirms the necessity of methods that focus on the contrasts perceived by black box models. for a specific dataset of masked tokens our methodology reveals the most salient differences between their contexts. contributions with our methodology we offer a new perspective on the contextual representations inside masked language models: the contrasts within a dataset from bert's point of view. by its application we render visible how well syntax describes the coarse patterns of the space but further how much of this description is possible by mere simplistic positional information. finally we demonstrate the danger of misinterpreting the learned patterns of the models due to the correlational nature of separated information types which may also lead to an overestimation of the models' sophistication. ## related work in the field of bertology (see @xcite for a general overview) much research has focused on three components; the self-attention mechanism, a key component of the transformer architecture that provides intuitive explanations (see e.g. @xcite @xcite , individual neurons (e.g. @xcite and the layer activations that are scrutinized in our work. frequently the edge probing design @xcite has been deployed to analyze the contents of these activations, in different settings such as after finetuning @xcite and with various modifications. amnesic probing measures what information gets used in the probing tasks by removing selected properties (e.g. part-ofspeech) from the activations @xcite . similarly o'connor and andreas (2021) measured usable information when increasing context size and ablating features of this additional context, e.g. by shuffling. in a parameter-free approach @xcite analyzed the output representation of a masked token by additionally masking other tokens in its proximity to determine their impact. another stream of research explores the geometrical space of layer activations. a common approach is the direct measurement of similarities, e.g. between instances of the same token and tokens of the same sentences @xcite or between instances of homonyms and synonyms @xcite . further work analyzes the separability of predefined categories (e.g. word senses) by manifold analysis @xcite , by measuring categorical cohesion with silhouette scores @xcite or a nearest-neighbor classifier @xcite , or by searching for clustering solutions that correspond to the categories @xcite . the similarities to our work are the focus on word level representations and the search for categories, though our clusters are not predefined by us but are the groupings inherent to our datasets from bert's point of view. much of the described work concerns only representations of unmasked tokens, except for e.g. @xcite and @xcite , but as masked language modeling continues to be a popular training objective the study of contextual information of masked tokens is highly relevant. ## experimental setup because of the various components of probing classifiers and their respective interactions, the design of such is non-trivial @xcite . this is also true for this new, inverted type of probing process that we present here. ## results for the investigation of prevailing differences discerned by the models, we are starting with a manual inspection of the pca plots for bert-base-uncased in figure 1 and deberta-base in figure 3 , observing that the contextual space for some combinations of datasets and layers exhibits quite distinct clusters. the presence of further clusters is indicated by their optimal number as determined by silhouette scores, shown in the lower right corner of the individual plots. the datapoints are colored according to positional information, here simply defined as the first character of the masked token divided by the number of characters in the sentence. from these visuals alone we can already learn that positional information greatly shapes the principal components and visible clusters. some clusters are completely defined by a specific position while others are internally arranged by this feature. the probing results for the test datasets are shown in figure 2 for bert-base-uncased and figure 4 for deberta-base (evaluation results can be found in appendix e). for almost all studied representations the part-of-speech models perform best or are on par to the bag-of-words models. the performance gap is more distinct for the explained variance of the principal components with an 0.21 average difference in r² but only 0.1 for the macro-f1 scores of the cluster predictions. notably, while the position models can never outperform the pos models, as they receive only the position-related subset of their contextual inputs, they achieve a large percentage of their performance for many settings, corroborating the finding of the visibly prevailing positional information. here the performance gaps are 0.40 for r² and 0.26 for the macro-f1 scores, averaged over all studied settings, showing how much the part-of-speech tags add to the explanations. for some datasets the plots of bert and de-berta closely resemble one another, especially for the noun-synsets. strikingly, though the data consists of three equally-sized groups of synsets, there are exactly two clusters visible in 2d. the cluster assignment plots for best values of k in appendix f show that for some layers of bert and deberta the k-means algorithm does manage to differentiate all three of the synsets. since the probing results are similar as well, we can conclude that bert's and deberta's point of view do correspond for this dataset. qualitative inspection found that the synsets person.n.01 and manner.n.01 adjoin while line.n.16 is spatially far removed. while positional information visibly permeates the clusters the distance between them is described almost perfectly by the pos models, thus by syntactic contextual differences. however, the almost equal performance of the bow models shows the correlation of part-of-speech with bag-of-words patterns that can be exploited. for the dataset of masked person.n.01 synsets the bert and deberta 2d-projections appear less alike, especially for layer 6. in this setting the pos model's performance for the regression of bert's principal components greatly exceeds all others with 88% of their variances explained. further analysis showed that the cluster in the upper half of the plot contains only instances of masked tokens that were followed by an apostrophe, showing that his specific syntactic pattern is perceived as significantly different. because of the very low performance of the bow regression model, we can be sure that this difference is indeed caused by syntax and not by co-occurences. the diverg-ing results for the k-means-2 model expose that the clustering algorithm found a different one than the visible grouping solution (see also plots with cluster assignments for @xmath0. for both the twitter datasets of random masks and masked activist tokens the final layer of bert singles out those that appeared at the end of the sentence. qualitative investigation showed that this is the case regardless if the token was the final one or followed by a punctuation character. for deberta the space of cctweets-activist is characterized by position to a greater extent than that of cctweets-random, as evident from the visuals and the performances of the position models. the numbers of clusters as suggested by silhouette scores are much higher for the random masks. while the bert and deberta perspectives on the dataset with equally sized groups of synsets seem quite identical, the cases of one synset or random masks reveal rather different perceived contextual differences. ## discussion & conclusion in contrast to much recent bertology work of predicting specific syntactic and semantic information from layer activations, we invert the probing design to instead predict features of the representative space itself. from masked token representations we extract clusters and principal components of contextual information and explore their nature by probing models that receive as input detangled types of information. we thus paint a picture of the dissimilarities and groupings within a dataset from bert's point of view, thereby expanding existing probing methodology by a crucial perspective. our analysis shows that the representative space of contextual information does exhibit clusters. most clusters and principal components of our datasets are best described by the part-of-speech models, however, for many settings the positional probing models can achieve 50% or more of the pos performance. this shows how the representative space of both bert and deberta is greatly shaped by the most simple positional information, even though these models handle positional embeddings differently. as demonstrated by @xcite , neural networks are prone to shortcut learning, and thus position may be one such shortcut. on the other hand, for the standard bert it was shown that the representative space is anisotropic due to outlier neurons capturing positional information, which was attributed to layer normalization @xcite . the usual probing classifier architecture that receives representations as input and predicts a specified linguistic property cannot clarify, if the representations are actually informed by the linguistic property of interest or by other, correlating properties of the training data @xcite . in our analysis the for some cases equal performance of detangled semantics (bag-of-words) and syntax (ordered part-of-speech tags) shows as well their correlational nature and the difficulty of pinpointing which features are actually utilized by large masked language models. when simplistic and meaningful features correlate this provides the danger of assuming that the models are much more sophisticated than they actually are. we do not attempt to answer the question of what information should predominate the represen-tational space, but it is likely that the optimum is not reached with features as simple as the position of a token in a sentence. we expect the best solutions to be defined by more sophisticated features that are not obtainable with simple string analysis, and which might even be utilizable for data analysis and hence other fields of research. concluding, our methodology delivers clues about the shortcomings of language models and the shortcuts that they are exploiting, to highlight directions of further adjustments of training objectives and processes in the future. we hope that this work inspires more researchers to look at the world from bert's point of view, to understand how it differs from ours. by recognizing the nature of their current primitivity we can generate new ideas on how to improve these large language models, gradually moving in the direction of a more general ai. limitations the prevailing contextual patterns that are revealed by this method are not universal but are always contingent on the analyzed datasets. accordingly these have to be chosen and controlled depending on the research objective. for the extraction of categories by clustering, selecting the appropriate number of clusters is nontrivial. here the number of clusters was set to equal numbers to allow for a comparison between datasets and layers, but these may not reflect the inherent number of groupings. lastly this method, as with any method that analyzes individual parts of a network in isolation, does not explain how the identified prevailing information is utilized during task performance. future work a promising extension of this work will be to enlarge the set of probing models to even better partition the different types of information, to better understand their contributions. examples of further relevant input types are a windowed bag-of-words or bag-of-words filtered by word types. furthermore it would be highly interesting to compare the pos model to other embedding models (e.g. simply word embeddings) with identical structures. the settings that may be analyzed by this method are various, such as the finetuning process, to explore how the prevailing patterns shift when models adapt to particular tasks. a comparison between models of different languages may reveal different focuses and varying correlations of information types. the described method is applicable also for the analysis of unmasked tokens though then the process of contextualization will differ from the very first layer depending on the token. the masked and unmasked contextualizations are moreover shaped by different objectives, predicting masked tokens and predicting potentially perturbed tokens, which may result in attention to different contextual patterns. finally it may be fruitful to utilize gradientbased attribution methods to pinpoint not just the relevance of input types but the relevance of specific inputs and positions from bert's point of view.