Too Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications (2021)

Abstract: The success of language models based on the Transformer architecture appears to be inconsistent with observed anisotropic properties of representations learned by such models. We resolve this by showing, contrary to previous studies, that the representations do not occupy a narrow cone, but rather drift in common directions. At any training step, all of the embeddings except for the ground-truth target embedding are updated with gradient in the same direction. Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models we have empirically tested. Our experiments show that isotropy can be restored using a simple transformation.

## introduction word embeddings, both static @xcite and contextualized @xcite , have been instrumental to the progress made in natural language processing over the past decade @xcite @xcite @xcite . in recent years, language models based on transformer architecture @xcite have led to state-of-the-art performance on problems such as machine translation @xcite , question answering @xcite , and word sense disambiguation @xcite , among others. however, it has been observed that representations from transformers exhibit undesirable properties, such as anisotropy, that is tend to occupy only a small subspace of the embedding space. the observation has been documented by a number of studies @xcite @xcite . a similar property has been identified in the past in static word embeddings @xcite . to address the issues, postprocessing methods @xcite , and regularization terms have been proposed @xcite @xcite . however, the mechanism that leads to undesirable properties remains unclear. without understanding the mechanism, it is going to be difficult to address the fundamental issue properly. the deficiencies are most pronounced in the representations of rare words, as we will show in section 4. performance of pretrained language models is inconsistent and tends to decrease when input contains rare words @xcite . @xcite observe that replacing a portion of words in the mnli @xcite entailment data set with less frequent synonyms leads to decrease in performance of bert-base and roberta-large by 30% and 21.8% respectively. 2 after enriching rare words with surface-form features and additional context, @xcite decrease the performance gap to 20.7% for bert and 17% for roberta, but the gap remains large nonetheless. why do even the large-scale, pretrained language models struggle to learn good representations of rare words? consider a language model with an embedding matrix shared between the input and output layers, a standard setup known as weight tying trick @xcite . intuitively, at any training step t, optimization of the cross-entropy loss can be characterized as "pulling" the target embedding, w t , closer to the model's output vector h t , while "pushing" all other embeddings, w \w t , in the same direction, away from the output vector h t . this leads to what we call common enemies effect -the effect of the target words producing gradients of the same direction for all of the nontarget words. compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models we have empirically tested; see figure 1 . although @xcite report a closely related phenomenon and call it representation degeneration, their analysis is based on an assumption that the embedding matrix is learned after all other parameters of the model are well-optimized and fixed, which is not the case in practice. we conduct our analysis in a more realistic setting, and arrive at different conclusions. we show that embeddings do not occupy a narrow cone, but are shifted in one common direction and only appear as a cone when projected to a lower dimensional space (section 4.1). in fact simply removing the mean vector of all embeddings, thus centering them, shifts the embeddings back onto a more spherical shape. we evaluate embeddings, before and after centering, on four standard benchmarks and observe significant performance improvement across all of them. why is removing the mean so effective? we find that the common enemies effect applies to most, if not all, words in the vocabulary but in non-uniform manner. as language is known to follow an approximately zipfian distribution @xcite @xcite even common words will not occur frequently in a text corpus, and in result will be often "pushed" by other target words in the same direction as rare words. consequently, all embeddings share a significant common direction. we will focus on the analysis of auto-regressive gpt-2 @xcite and two masked language models, bert @xcite and roberta @xcite . our contributions can be summarized as follows: ## experiments we validate our theoretical analysis through a series of experiments on geometric properties of noncontextualized embeddings. ## discussion there has been a body of literature demonstrating substantial benefits of improved quality of word embeddings on downstream performance (e.g., @xcite @xcite @xcite . in particular, @xcite propose to add a cosine similarity regularization to the crossentropy loss to increase the aperture of the cone in which embeddings are distributed, and report improved performance on machine translation and language modeling. it is straightforward to demonstrate that the cosine regularization proposed by @xcite is equivalent to minimizing the squared norm of the mean direction of embeddings, hence constraining the most significant drift direction. we provide the derivation of the equivalence in appendix a. large-margin classification has been studied extensively, both in nlp @xcite and machine learning in general @xcite . as substantial shared components of embeddings will lead to a decreased classification margin in the output softmax layer, our work offers explanation for the fragility of pretrained language models reported in the literature (e.g., @xcite . our analyses show clearly that shifting of the embeddings in the embedding space is due to the dynamic interactions between the representations and the embedding vectors. as the embeddings become more similar, the resulting representations become closer, creating a positive feedback mechanism for the representations to drift collectively. in addition, while isotropy of representations is desirable and has an overall positive impact on performance, the relationships between isotropy and performance in a considerable effort has been made to improve performance of language systems on rare words, but the focus has been on either injecting subword information in non-contextual representations @xcite @xcite , replacing rare words' representations through exploiting their context @xcite , or both @xcite . in comparison, we strive to provide an explanation of the underlying problem, which is necessary to render such post-hoc fixes no longer necessary. ## conclusion we find that the embeddings learned by gpt-2, bert, and roberta do not degenerate into a narrow cone, as has been suggested in the past, but instead drift in one shared direction. we recognize that target words produce gradients in the same direction for all the non-target words at each training step. combined with the unbalanced distribution of word frequencies, any two words' embeddings will be repeatedly updated with gradients of the same direction. as such updates accumulate, the embeddings drift and share common components. our experiments show that simply centering the embeddings restores a nearly perfectly isotropic distribution of tested models' embeddings and simultaneously improves embeddings' ability to reflect semantic relations. this understanding of the learning process dynamics opens exciting avenues for future work, such as improving the most affected embeddings of rare words and formulation of more computationally efficient training objectives. neural machine translation system: bridging the gap between human and machine translation. in arxiv:1609.08144. zhilin yang, zihang dai, ruslan salakhutdinov, and william w. cohen. 2018. breaking the softmax bottleneck: a high-rank rnn language model. in international conference on learning representations. zhilin yang, zihang dai, yiming yang, jaime carbonell, russ r salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems, volume 32. curran associates, inc. canlin zhang, xiuwen liu, and daniel bis. 2019. an analysis on the learning rules of the skip-gram model. 2019 international joint conference on neural networks (ijcnn), pages 1-8. george kingsley zipf. 1949. human behaviour and the principle of least effort: an introduction to human ecology.