Anisotropy Is Inherent to Self-Attention in Transformers (2024)

Abstract: The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.

## introduction in recent years, deep learning models based on transformers have led to significant breakthroughs in the field of natural language processing (nlp). these models have demonstrated state-of-the-art performance across a range of tasks, such as language modeling, machine translation, and sentiment analysis. however, despite their successes, they suffer from a phenomenon known as the representation degeneration problem. specifically, this degeneration is characterized by anisotropy, a property of hidden representations that makes them all close to each other in terms of angular distance (cosine-similarity). anisotropy has been widely observed among self-supervised models based on transformers, and literature currently suggests that it may be a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens @xcite . however, it remains uncertain whether anisotropy is a fundamental property of transformers-based models or a consequence of the pre-training process. in this paper, we investigate the anisotropy problem in depth, and we make several contributions: ## related work the general phenomenon of anisotropy in tokenbased transformers for language models has been shown in @xcite . figure 1 extends one of their experiment to more architectures. @xcite shows that the degeneration of representations comes from the distributions of subwords in natural language, namely the existence of unused and rare tokens that tend to push all representations away from the origin towards a specific direction. other works have established a connection between word frequency and distortions of the latent spaces @xcite @xcite . @xcite have shown that anisotropy in lms could be explained by a global drift of the representations in the same direction, thus unifying conclusions from @xcite and @xcite . the authors propose that this drift is caused by the persistent updating of the representation of rare and unused tokens in a consistent direction, due to the nature of the softmax operation in the cross-entropy loss. they show that removing the average component to all representations leads to a nearly perfect isotropy. several methods have been proposed to reduce anisotropy in transformers-based lms at tokenlevel @xcite , or at sentence-level @xcite @xcite . they usually consist in post-processing the representations, and lead to downstream performance boosts. we argue that these positive results are paving the way for the search of pre-training objectives that do not introduce anisotropy in the first place, in the hope that the resulting models will also perform better without any post-processing, and potentially be trained more efficiently. this motivates us to gain a deeper understanding of the underlying factors that induce anisotropy, whether they belong in data, architectures, or training procedures. ## exploring the representation drift in this section, we focus on some intrinsic properties of the transformer block in a modalityagnostic fashion, i.e. with minimal assumptions on the data distribution, and without training. we analyze experimentally the behavior of the untrained transformer block t when a common bias term b is added to untrained input representations x. this allows us to mimic the common drift as mentioned in @xcite and to identify some properties induced by this artificial drift on the output representations. ## queries and keys: training dynamics we have established that manually pushing for driftbased anisotropy on untrained transformers models leads to sharper (i.e. low-entropy) self-attention patterns. in this section, we show that this evolution of self-attention values actually takes place during training, and we explore the mechanism behind their appearance. as pointed out in section 4, the self-attention scores result from the qk t operation, which computes scalar products between query and key representations corresponding to each pair of positions. thus, in this section, we study the evolution of these query and key representations along training, and explore the mechanism behind the increase of the scalar products leading to self-attention scores. we use the multibert checkpoints @xcite with seed 0 to retrieve q and k distributions at different pretraining steps, and we use 128 samples from wikitext-103 as input data. along this section, q s and k s refer to query and key representations extracted at a specific layer and head at a given step s, and qs and ks are the average representations, taken over all tokens in the sampled batch. by studying qs and ks , we aim at exploring the common (or context-agnostic) drifts of keys and queries distributions. in figure 11 and figure 12 , we compute a svd of the union of q s and k s for all steps s, so that the projection makes sense for both distributions across steps for visualization purposes foot_0 . as shown in our selected examples, we observe that the dynamics of qs and ks tend to align along training, making the average of the distributions drift in either similar or opposite directions. the first dimension of the svd seems to describe this common drift. note that in r d h (d @xmath0, such an alignment is very unlikely to happen randomly. interestingly, figure 12a shows that the common direction dynamics appear in the first few steps, while the opposite direction dynamics of figure 12b only starts after 8% of the total training steps. to consolidate our observations, we compute the evolution of the cosine-similarity between qs and ks along training in figure 13 . we also display (a) similar (b) opposite figure 12: evolution of qs and ks along training for two different heads in the network, projected via common svd. each arrow represents a checkpoint in the multibert suite. we display typical examples of dynamics in same/opposite direction. figure 13 shows that the first layers display a common direction dynamic, as the cosinesimilarity tends to increase, thus showing that the key and query distributions drift along a similar direction in average. the last layers seem to adopt an opposite direction dynamic, as the cosinesimilarity between their mean key and query representations gets negative along training. as shown in figure 14 , this drift induces an increase in the magnitude of scalar products obtained in the self-attention qk t operation, thus facilitating the emergence of sharp patterns where attention focuses on specific tokens. finally, figure 15 describes the evolution of the average entropy in self-attention distributions. we observe that training induces an overall decay of the entropy for all layers, with different dynamics. this corresponds to sharper self-attention distributions. it is interesting to notice that the distributions in the first layers remain sharper than the ones in the last layers. overall, this section shows that drift anisotropy emerges in the query and key representations during the training of multibert, as self-attention distributions become sharper. the drifts of queries and keys tend to align, thus increasing the magnitude of scalar products, and the general sharpness of self-attention. although this section focuses on the case of token-based nlp, we believe that strong attention patterns may be required when training transformers across all modalities, potentially generating distortions in query and key distributions that account for the final observed anisotropy of the models. however, we could not extend experiments to other modalities due to the lack of released intermediate checkpoints, to the best of our knowledge. ## discussion in this work, we argue that the nature of data distributions is not solely responsible for the anisotropy observed in most hidden representations of transformers-based models across modalities. as section 4 shows, untrained transformers layers display a tendency towards anisotropy. biased inputs tend to increase the variance of the attention scores and thus facilitate the emergence of sharp patterns in the self-attention mechanisms. we also show in section 5 that along training, query and key distributions drift in parallel directions, which increases anisotropy in the inner representations of the transformer layers, while allowing sharper attention patterns. as discussed in @xcite , outlier dimensions in transformers are also involved in the emergence of strong attention patterns. consistency of the svd in section 5, we use an svd on the union of q s and k s for visualization purposes (see figure 11 and figure 12 ). it may be argued that this approach favors the emergence of a discriminative singular direction, that helps distinguish between keys and queries, thus supporting the findings in a less convincing way. to address this concern, we display alternative projections in appendix c, where we compute the svd on q s or (a) similar (b) opposite figure 14: evolution of the scalar product between qs and ks along training. steps are in thousands. k s only, and then project all representations using this svd. our observations show that our findings are consistent for these alternative projections.