Outlier Dimensions that Disrupt Transformers are Driven by Frequency (2022)

Abstract: While Transformer-based language models are generally very robust to pruning, there is the recently discovered outlier phenomenon: disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI. We replicate the original evidence for the outlier phenomenon and we link it to the geometry of the embedding space. We find that in both BERT and RoBERTa the magnitude of hidden state coefficients corresponding to outlier dimensions correlate with the frequencies of encoded tokens in pre-training data, and they also contribute to the “vertical” self-attention pattern enabling the model to focus on the special tokens. This explains the drop in performance from disabling the outliers, and it suggests that to decrease anisotopicity in future models we need pre-training schemas that would better take into account the skewed token distributions.

## introduction the current transformer-based language models are heavily overparametrized, which explains why it is possible to prune these models by up to 30-40% @xcite @xcite without a significant drop in performance. however, it has recently been shown that multiple transformer-based language models (lms) are highly sensitive to removal of outlier dimensions @xcite : the parameters (weights and biases) in the output element of a transformer layer, the magnitude of which is unusually large within the layer (consistently in the same dimension across the model layers). for bert model family the output element is the layernorm, as shown in figure 1 . although these parameters constitute less than 0.0001% of the full bert @xcite model, removing them significantly degrades bert's performance. @xcite find that the same parameters are particularly relevant in several linguistic probing tasks. these dimensions affect the vector representation of different tokens in the same way, making the embedding space less isotropic and thus reducing its representational power @xcite . outlier dimensions have also been found to make model quantization challenging @xcite as they need to be treated separately from others when defining quantization schemes. thus there are both conceptual and practical reasons supporting a deeper study of this phenomenon. what is not clear at this point is the mechanism behind the emergence of outliers. we replicate the original findings in bert and roberta, and we contribute new evidence directly linking the outlier phenomenon with the frequency of encoded tokens in the pre-training data, as well as the self-attention pattern focusing on special tokens. we also present evidence for two kinds of outliers: some of them affect the masked language model (mlm) performance the most in the middle layers (where the correlation with token frequency is at its peak), and for others the impact grows towards bert-base-uncased cola mnli mnli-mm mrpc qnli qqp rte sst2 stsb baseline 56.9 +/-1.5 84.5 +/-0.2 84.8 +/-0.4 84.3 +/-1.1 91.4 +/-0.1 91.1 +/-0.1 66.3 +/-1.6 92.8 +/-0.5 89.0 +/-0.3 1 random removed 56.5 +/-1.5 84.5 +/-0.2 84.8 +/-0.4 84.5 +/-0.9 91.3 +/-0.1 91.1 +/-0.1 66.6 +/-1.7 92.8 +/-0.4 89.0 +/-0.3 w/o 308 47.3 +/-1.2 81.4 +/-1.1 82.2 +/-1.1 54.0 +/-12.3 88.9 +/-0.8 89.1 +/-1.6 62.1 +/-3.2 90.8 +/-1.1 56.9 +/-17.4 w/o 381 33.8 +/-9.4 73.2 +/-2.1 73.8 +/-2.0 64.6 +/-15.2 80.3 +/-1.5 79.8 +/-3.2 55.8 +/-1.6 87.9 +/-1.0 78.1 +/-4.8 2 random removed 56.4 +/-1.5 84.5 +/-0.2 84.8 +/-0.4 84.3 +/-0.9 91.3 +/-0.1 91.1 +/-0.1 66.6 +/-1.7 92.8 +/-0.5 88.9 +/-0.3 w/o 308 & 381 15.9 +/-4.2 58.4 +/-3.3 59.0 +/-3.5 55.1 +/-16.3 74.5 +/-1.3 74.6 +/-4.6 55.3 +/-4.5 76.0 +/-2.4 35.7 +/-15. 3 ## methodology according to @xcite , outliers are parameters (both weights and biases) in the final element of a transformer layer (layernorm for bert family, final mlp for gpt-2), which have unusually high magnitude foot_0 within the layer. outlier dimensions are those dimensions at which outlier parameters are found consistently across the model layers. the reason kovaleva et al. study these parameters is that when they are disabled, the model performance on downstream tasks is greatly reduced. since not all parameters that can be identified by magnitude and position criteria have that effect, we add this property to the definition. in this work the term outlier dimension refers to the dimensions with parameters meeting the magnitude criteria across layers and having at least 5x more damaging effect on accuracy on a representative downstream task, for which we choose mnli (see §3.1). to disable the outliers, unless stated otherwise, we set to zero both the layernorm weight and bias parameters for all layers (24 parameters in total for one outlier dimension in bert and robertabase) foot_1 . see app. a for the full list of outliers identified for all models in this study. we use the notation o to refer to specific lay-ernorm outlier parameters (in bert model family): e.g. o381 to indicate "an outlier with index 381". since outlier indices are a constant for a given model, in this study we will also discuss hidden state outlier dimensions: the coefficient of the hidden state with the same index as the outlier. we experiment with bert-base @xcite ("bert-base-uncased"), roberta-base @xcite ("roberta-base") and vision transformer @xcite ("google/vit-base-patch16-224-in21k") from the transformers library foot_2 . for the experiments on pre-training dynamics we rely on the checkpoints with seed 1 provided by @xcite 4 . hardware, implementation and energy expenditure details are outlined in app. b. we release the code to replicate our experiments foot_5 . ## conclusion to the best of our knowledge, this is the first work to directly link the outlier dimension phenomenon in transformer-based models (in particular bert and roberta) to encoded token frequency in pretraining data. we also find that the magnitude of hidden state dimensions corresponding to outliers correlates with the vertical self-attention pattern, which enables the attention to the classification tokens. furthermore, we find that there are two types of outliers: some of them affect the mlm performance the most in the middle layers (where the correlation with token frequency is also at its peak), and for others the impact grows towards the final layers (even though the correlation with token frequency decreases). our findings suggest that outliers are due not to the transformer architecture per se, but rather to the highly skewed token frequency distribution in textual pre-training data. in that case, to mitigate anisotropy we might need to design a pre-training scheme that better accounts for such distributions.