Positional Artefacts Propagate Through Masked Language Model Embeddings (2021)

Abstract: In this work, we demonstrate that the contextualized word vectors derived from pretrained masked language model-based encoders share a common, perhaps undesirable pattern across layers. Namely, we find cases of persistent outlier neurons within BERT and RoBERTa’s hidden state vectors that consistently bear the smallest or largest values in said vectors. In an attempt to investigate the source of this information, we introduce a neuron-level analysis method, which reveals that the outliers are closely related to information captured by positional embeddings. We also pre-train the RoBERTa-base models from scratch and find that the outliers disappear without using positional embeddings. These outliers, we find, are the major cause of anisotropy of encoders’ raw vector spaces, and clipping them leads to increased similarity across vectors. We demonstrate this in practice by showing that clipped vectors can more accurately distinguish word senses, as well as lead to better sentence embeddings when mean pooling. In three supervised tasks, we find that clipping does not affect the performance.

## introduction a major area of nlp research in the deep learning era has concerned the representation of words in low-dimensional, continuous vector spaces. traditional methods for achieving this have included word embedding models such as word2vec @xcite , glove @xcite , and fasttext @xcite . however, though influential, such approaches all share a uniform pitfall in assigning a single, static vector to a word type. given that the vast majority of words are polysemous @xcite , static word embeddings cannot possibly represent a word's changing meaning in context. in recent years, deep language models, like elmo @xcite , bert @xcite and roberta @xcite , have achieved great success across many nlp tasks. such models introduce a new type of word vectors, deemed the contextualized variety, where the representation is computed with respect to the context of the target word. since these vectors are sensitive to context, they can better address the polysemy problem that hinders traditional word embeddings. indeed, studies have shown that replacing static embeddings (e.g. word2vec) with contextualized ones (e.g. bert) can benefit many nlp tasks, including constituency parsing @xcite , coreference resolution @xcite and machine translation @xcite . however, despite the major success in deploying these representations across linguistic tasks, there remains little understanding about information embedded in contextualized vectors and the mechanisms that generate them. indeed, an entire research area central to this core issue -the interpretability of neural nlp models -has recently emerged @xcite @xcite . a key theme in this line of work has been the use of linear probes in investigating the linguistic properties of contextualized vectors @xcite . such studies, among many others, show that contextualization is an important factor that sets these embeddings apart from static ones, the latter of which are unreliable in extracting features central to context or linguistic hierarchy. nonetheless, much of this work likewise fails to engage with the raw vector spaces of language models, preferring instead to focus its analysis on the transformed vectors. indeed, the fraction of work that has done the former has shed some curious insights: that untransformed bert sentence representations still lag behind word embeddings across a variety of semantic benchmarks @xcite and that the vector spaces of language models are explicitly anisotropic @xcite . certainly, an awareness of the patterns inherent to models' untransformed vector spaces -even if shallow -can only benefit the transformation-based analyses outlined above. in this work, we shed light on a persistent pattern that can be observed for contextualized vectors produced by bert and roberta. namely, we show that, across all layers, select neurons in bert and roberta consistently bear extremely large values. we observe this pattern across vectors for all words in several datasets, demonstrating that these singleton dimensions serve as major outliers to the distributions of neuron values in both encoders' representational spaces. with this insight in mind, the contributions of our work are as follows: ## finding outliers in this section, we demonstrate the existence of large-valued vector dimensions across nearly all tokens encoded by bert and roberta. to illustrate these patterns, we employ two well-known datasets -sst-2 @xcite and qqp 1 . sst-2 (60.7k sentences) is a widely-employed sentiment analysis dataset of movie reviews, while qqp (727.7k sentences) is a semantic textual similarity dataset of quora questions, which collects questions across many topics. we choose these datasets in order to account for a reasonably wide distributions of domains and topics, but note that 1 https://www.quora.com/q/quoradata/ first-quora-dataset-release-question-pairs any dataset would illustrate our findings well. we randomly sample 10k sentences from the training sets of both sst-2 and qqp, tokenize them, and encode them via bert-base and roberta-base. all models are downloaded from the huggingface transformers library @xcite , though we replicated our results for bert by loading the provided model weights via our own loaders. when discounting the input embedding layers of each model, we are left with 3.68m and 3.59m contextualized token embeddings for bert-base and roberta-base, respectively. in order to illustrate the outlier patterns, we average all subword vectors for each layer of each model. in examining bert-base, we find that the minimum value of 96.60% of vectors lies in the 557 th dimension. figure 1 displays the averaged subword vectors for each layer of bert-base, corroborating that these patterns exist across all layers. for roberta-base, we likewise find that the maximum value of all vectors is the 588 th element. interestingly, the minimum element of 88.19% of vectors in roberta-base is the 77 th element, implying that roberta has two such outliers. figure 2 displays the average vectors for each layer of robertabase. our observations here reveal a curious pattern that is present in the base versions of bert and roberta. we also corroborate the same findings for the large and distilled @xcite variants of these architectures, which can be found in the appendix a. indeed, it would be difficult to reach any sort of conclusion about the representational geometry of such models without understanding the outliers' origin(s). in this section, we attempt to trace the source of the outlier dimensions in bert-base and robertabase (henceforth bert and roberta). similarly to the previous section, we can corroborate the results of the experiments described here (as well as in the remainder of the paper) for the large and distilled varieties of each respective architecture. thus, for reasons of brevity, we focus our forthcoming analyses on the base versions of bert and roberta and include results for the remaining models in the appendix b.2 for the interested reader. in our per-layer analysis in §2, we report that outlier dimensions exist across every layer in each model. upon a closer look at the input layer (which features a vector sum of positional, segment, and token embeddings), we find that the same outliers also exist in positional embeddings. figure 3 shows that the 1st positional embedding of bert has two such dimensions, where the 557 th element is likewise the minimum. interestingly, this pattern does not exist in other positional embeddings, nor in segment or token embeddings. furthermore, figure 4 shows that the 4th positional embedding of roberta has four outliers, which include the aforementioned 77 th and 588 th dimensions. we also find that, from the 4th position to the final position, the maximum element of 99.8% positional embeddings is the 588 th element. digging deeper, we observe similar patterns in the layer normalization @xcite ) parameters of both models. recall that ln has two learnable parameters -gain (γ) and bias (β)both of which are 768-dimension vectors (in the case of the base models). these are designed as an affine transformation over dimension-wise normalized vectors in order to, like most normalization strategies, improve their expressive ability and to aid in optimization. every layer of bert and roberta applies separate lns post-attention and pre-output. for bert, the 557 th element of the γ vector is always among the top-6 largest values for the first ten layers' first ln. specifically, it is the largest value in the first three layers. for roberta, the 588 th element of the first ln's β vector is always among the top-2 largest values for all layersit is largest in the first five layers. furthermore, the 77 th element of the second ln's γ are among the top-7 largest values from the second to the tenth layer. it is reasonable to conclude that, after the vector normalization performed by ln, the outliers observed in the raw embeddings are lost. we hypothesize that these particular neurons are somehow important to the network, such that they retained after scaling the normalized vectors by the affine transformation involving γ and β. indeed, we observe that, in bert, only the 1st position's embedding has such an outlier. however, it is subsequently observed in every layer and token after the first ln is applied. since layernorm is trained globally and is not token specific, it happens to rescale every vector such that the positional information is retained. we corroborate this by observing that all vectors share the same γ. this effectively guarantees the presence of outliers in the 1st layer, which are then propagated upward by means of the transformer's residual connection @xcite . also, it is important to note that, in the case of bert, the first position's embedding is directly tied to the requisite [cls] token, which is prepended to all sequences as part of the mlm training objective. this has been recently noted to affect e.g. attention patterns, where much of the probability mass is distributed to this particular token alone, despite it bearing the smallest norm among all other vectors in a given layer and head @xcite . ## clipping the outliers in §3, we demonstrated that outlier neurons are related to positional information. in this section, we investigate the effects of zeroing out these dimensions in contextualized vectors, a process which we refer to as clipping. ## discussion the experiments detailed in the previous sections point to the dangers of relying on metrics like cosine similarity when making observations about models' representational spaces. this is particularly salient when the vectors being compared are taken off-the-shelf and their composition is not widely understood. given the presence of model idiosyncracies like the outliers highlighted here, mean-sensitive, l2 normalized metrics (e.g. cosine similarity or pearson correlation) will inevitably weigh the comparison of vectors along the highestvalued dimensions. in the case of positional artefacts propagating through the bert and roberta networks, the basis of comparison is inevitably steered towards whatever information is captured in those dimensions. furthermore, since such outlier values show little variance across vectors, proxy metrics of anisotropy like measuring the average cosine similarity across random words (detailed in §4.1) will inevitably return an exceedingly high similarity, no matter what the context. when cosine similarity is viewed primarily as means of semantic comparison between word or sentence vectors, the prospect of calculating cosine similarity for a benchmark like wic or sts-b becomes erroneous. though an examination of distance metrics is outside the scope of this study, we acknowledge similar points as having been addressed in regards to static word embeddings @xcite as well as contextualized ones @xcite . likewise, we would like to stress that our manual clipping operation was performed for illustrative purposes and that interested researchers should employ more systematic post-hoc normalization strategies, e.g. whitening @xcite , when working with hidden states directly. relatedly, the anisotropic nature of the vector space that persists even after clipping the outliers suggests that positional artefacts are simply part of the explanation. per this point, @xcite prove that, in training any sort of model with likelihood loss, the representations learned for tokens being predicted will be naturally be pushed away from most other tokens in order to achieve a higher like-lihood. they relate this observation to the zipfian nature of word distributions, where the vast majority of words are infrequent. @xcite extend this insight specifically to bert and show that, while high frequency words concentrate densely, low frequency words are much more sparsely distributed. though we do not attempt to dispute these claims with our findings, we do hope our experiments will highlight the important role that positional embeddings play in the representational geometry of transformer-based models. indeed, recent work has demonstrated that employing relative positional embeddings and untying them from the simultaneously learned word embeddings has lead to impressive gains for bert-based architectures across common benchmarks @xcite . it remains to be seen how such procedures affect the representations of such models, however. beyond this, it is clear that layernorm is the reason positional artefacts propagate though model representations in the first place. indeed, our experiments show that the outlier dimension observed for bert is tied directly to the [cls] token, which always occurs at the requisite 1st position --despite having no linguistic bearing on the sequence of observed tokens being modeled. however, the fact that roberta (which employs a similar delimiter) retains outliers originating from different positions' embeddings implies that the issue of artefact propagation is not simply a relic of task design. it is possible that whatever positional idiosyncrasies contribute to a task's loss are likewise retained in their respective embeddings. in the case of bert, the outlier dimension may be granted a large negative weight in order to differentiate the (privileged) 1st position between all others. this information being reconstructed by the layernorm parameters, which are shared for all positions in the sequence length, and then propagated up through the transformer network is a phenomenon worthy of further attention. ## related work in recent years, an explosion of work focused on understanding the inner workings of pretrained neural language models has emerged. one line of such work investigates the self-attention mechanism of transformer-based models, aiming to e.g. characterize its patterns or decode syntactic structure @xcite @xcite @xcite . another line of work analyzes models' internal representations using probes. these are often linear classifiers that take representations as input and are trained with supervised tasks in mind, e.g. pos-tagging, dependency parsing @xcite @xcite @xcite . in such work, high probing accuracies are often likened to a particular model having "learned" the task in question. most similar to our work, @xcite investigate the extent of "contextualization" in models like bert, elmo, and gpt-2 @xcite . mainly, they demonstrate that the contextualized vectors of all words are non-isotropic across all models and layers. however, they do not indicate why these models have such properties. also relevant are the studies of @xcite , who introduce a neuron-level analysis method, and durrani et al. (2020) 2020 ) train a linear probe to predict linguistic information stored in a vector. they then employ the weights of the classifier as a proxy to select the most relevant neurons to a particular task. in a similar vein, @xcite demonstrate the existence of syntactic and semantic subspaces in bert representations. ## conclusion in this paper, we called attention to sets of outlier neurons that appear in bert and roberta's internal representations, which bear consistently large values when compared to the distribution of values of all other neurons. in investigating the origin of these outliers, we employed a neuronlevel analysis method which revealed that they are artefacts derived from positional embeddings and layer normalization. furthermore, we found that outliers are a major cause for the anisotrophy of a model's vector space @xcite . clipping them, consequently, can make the vector space more directionally uniform and increase the similarity between words' contextual representations. in addition, we showed that outliers can distort results when investigating word sense within contextualized representations as well as obtaining sentence embeddings via mean pooling, where removing them leads to uniformly better results. lastly, we find that "clipping" does not affect models' performance on three supervised tasks. it is important to note that the exact dimensions at which the outliers occur will vary pending different initializations and training procedures (as evidenced by our own roberta model). as such, future work will aim at investigating strategies for mitigating the propagation of these artefacts when pretraining. furthermore, given that both bert and roberta are masked language models, it will be interesting to investigate whether or not similar artefacts occur in e.g. autoregressive models like gpt-2 @xcite or xlnet @xcite . per the insights of @xcite , it is very likely that the representational spaces of such models are anisotropic, but it is important to gauge the extent to which this can be traced to positional artefacts.