All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021)

Abstract: Similarity measures are a vital tool for understanding how language models represent and process language. Standard representational similarity measures such as cosine similarity and Euclidean distance have been successfully used in static word embedding models to understand how words cluster in semantic space. Recently, these measures have been applied to embeddings from contextualized models such as BERT and GPT-2. In this work, we call into question the informativity of such measures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate these measures. Moreover, we find a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model. We show that simple postprocessing techniques such as standardization are able to correct for rogue dimensions and reveal underlying representational quality. We argue that accounting for rogue dimensions is essential for any similarity-based analysis of contextual language models.

## introduction by mapping words into continuous vector spaces, we can reason about human language in geometric terms. for example, the cosine similarity of pairs of word embeddings in word2vec @xcite and glove @xcite shows a robust correlation with human similarity judgments, and embeddings cluster into natural semantic classes in euclidean space @xcite . in recent years, static embeddings have given way to their contextual counterparts, with language models based on the transformer architecture @xcite such as bert @xcite , roberta @xcite , xlnet @xcite and @xcite achieving state of the art results on many language understanding tasks. despite their success, relatively little is known about how these models represent and process language. recent work has employed measures such as cosine similarity and euclidean distance to contextual representations with unclear and counterintuitive results. for example, similarity/distance measures in bert are extremely sensitive to word position, leading to inconsistent results on evaluation benchmarks @xcite . additionally, representational quality appears to degrade severely in later layers of each network, with the final layers of bert, roberta, gpt-2 and xlnet showing little to no correlation with the semantic similarity/relatedness judgments of humans @xcite . recent work which probes the representational geometry of contextualized embedding spaces using cosine similarity has found that contextual embeddings have several counterintuitive properties @xcite . for example: 1) word representations are highly anisotropic: randomly sampled words tend to be highly similar to one another when measured by cosine similarity. in the final layer of gpt-2 for example, any two words are almost perfectly similar. 2) embeddings have extremely low self-similarity: in later layers of transformer-based language models, random words are almost as similar to one another as instances the same word in different contexts. in this work, we critically examine the informativity of standard similarity/distance measures (particularly cosine similarity and euclidean distance) in contextual embedding spaces. we find that these measures are often dominated by 1-5 dimensions across all the contextual language models we tested, regardless of the specific pretraining objective. it is this small subset of dimensions which drive anisotropy, low self-similarity, and the apparent drop in representational quality in later layers. these dimensions, which we refer to as rogue dimensions are centered far from the origin and have disproportionately high variance. the presence of rogue dimensions can cause cosine similarity and euclidean distance to rely on less than 1% of the embedding space. moreover, we find that the rogue dimensions which dominate cosine similarity do not likewise dominate model behavior, and show a strong correlation with absolute position and punctuation. finally, we show that these dimensions can be accounted for using a trivially simple transformation of the embedding space: standardization. once applied, cosine similarity more closely reflects human word similarity judgments, and we see that representational quality is preserved across all layers rather than degrading/becoming task-specific. taken together, we argue that accounting for rogue dimensions is essential when evaluating representational similarity in transformer language models. foot_0 ## background standard measures such as cosine similarity or euclidean distance in contextual embedding spaces have been used in a wide range of applications: to understand how the representational similarity of word embedding spaces corresponds to human semantic similarity/relatedness judgments @xcite @xcite , human brain activation patterns/cross-model similarity @xcite , syntax structure (chrupała and alishahi, 2019), semantic shift @xcite , compositionality/idomaticity of word vectors @xcite , polysemy (soler and apidianaki, 2021), context-sensitivity @xcite , social bias @xcite , changes to the embedding space during fine-tuning @xcite , and as an evaluation metric for text generation @xcite . however, a number of works have questioned the appropriateness of cosine similarity. @xcite found that static embedding models encode a substantial degree of word frequency information, which leads to a frequency bias in cosine similarity. @xcite questioned the adequacy of cosine similarity in sentence encoders after finding contextual discrepancies in bias measures. perhaps most relevant to the present work is @xcite which treats individual word embed-dings as statistical samples, shows the equivalence of cosine similarity and pearson correlation, and notes that pearson correlation (and therefore cosine similarity) is highly sensitive to outlier dimensions. they further suggest the use of non-parametric rank correlation measures such as spearman's ρ, which is robust to outliers. our work investigates the sensitivity of cosine similarity to outlier dimensions in contextual models, and further characterizes the behavioral correlates of these outliers. our goal in this work was not causal explanation of degenerate embedding spaces or post-processing for task performance gains, but rather to empirically motivate trivially simple transformations to enable effective interpretability research with existing metrics. however, we refer interested readers to @xcite who studied degeneration toward anisotropy in machine translation. similarly, @xcite suggested a learned transformation of transformer embedding spaces which resulted in increased performance on semantic textual similarity tasks. ## rogue dimensions and model behavior in this section, we address the question of whether the dimensions which dominate cosine similarity likewise dominate model behavior. specifically, if similarity measures are dominated by only a few dimensions, as shown in the previous sections, then those dimensions should be the only ones the model actually uses, otherwise, the measures only reflect a small subset of what the model is doing. we find that dimensions which dominate cosine similarity do not likewise dominate model behavior. ## postprocessing and representational quality while we have shown that the representational geometry of contextualized embeddings makes cosine similarity uninformative, there are several simple postprocessing methods which can correct for this. in this section we outline three such methods: standardization, all-but-the-top @xcite , and ranking (via spearman correlation). we evaluate representational quality of the postprocessed embeddings on several word similarity/relatedness datasets and show that the underlying representational quality is obscured by the rogue dimensions. when we correct for rogue dimensions, correlation with human similarity judgments improves across the board. we also find that representational quality is preserved across all layers, rather than giving way to degraded/task specific representations as argued in previous work. ## discussion and future work perhaps the most important direction for future work is designing and implementing language models which do not develop rogue dimensions in the first place. @xcite introduce a cosineregularization term during pretraining which improved the performance of transformer models on machine translation. perhaps bert or gpt models could similarly benefit from such regularization. a prerequisite for designing models without rogue dimensions is understanding how these dimensions arise over time. contemporaneous work from @xcite provides a useful characterization of how degenerate representations may be learned, which largely focuses on token frequency, while @xcite provide a characterization of how outliers impact model performance, attributing much of the problem to scaling factors in layer normalization, and luo et al. (2021) make observations about the contribution of positional embeddings. in the present work, we observe strong correlations with specific tokens and positions. unifying these accounts is an important task for future work. with the recent release of the multibert checkpoints @xcite , future work can uncover whether rogue dimensions are a coincidental property of some models, or whether they are a requisite for good performance. the multi-berts may also elucidate how these dimensions emerge during pretraining. while we empirically motivate a trivially simple transformation which corrects for rogue dimensions, we believe the most fruitful direction for future work is to build models whose representations require no post-hoc transformations. this would result in more interpretable embedding spaces and may additionally lead to models with better performance. ## conclusion in this work, we showed that similarity measures in contextual language models are largely reflective of a small number of rogue dimensions, not the entire embedding space. consequently, a few dimensions can drastically change the conclusions we draw about the linguistic phenomena a model actually captures. we showed that the previously observed anisotropy in contextual models is essentially an artifact of rogue dimensions and is not a global property of the entire embedding space. we also showed that variability in similarity is driven by just 1-5 dimensions of the embedding space. in many cases, removing just a single dimension completely changed which token pairs were similar to one another. however, we found that model behavior was not driven by these rogue dimensions, and that these dimensions seem to handle a small subset of a model's linguistic abilities, such as punctuation and positional information. in summary, standard similarity measures such as cosine similarity and euclidean distance are not informative measures of how contextual language models represent and process language. we argue that measures of similarity in contextual language models must account for rogue dimensions using techniques such as standardization. these techniques should not just be viewed as avenues to improve downstream performance, but as prerequisites for any analysis involving representational similarity.