Learned Transformer Position Embeddings Have a Low-Dimensional Structure (2024)

Abstract: Position embeddings have long been essential for sequence-order encoding in transformer models, yet their structure is underexplored. This study uses principal component analysis (PCA) to quantitatively compare the dimensionality of absolute position and word embeddings in BERT and ALBERT. We find that, unlike word embeddings, position embeddings occupy a low-dimensional subspace, typically utilizing under 10% of the dimensions available. Additionally, the principal vectors are dominated by a few low-frequency rotational components, a structure arising independently across models.

## introduction transformers, as introduced by @xcite , have significantly advanced the field of natural language processing, excelling in tasks like machine translation @xcite , question answering @xcite , information extraction @xcite , and text generation @xcite . the ability to encode positional information is vital in these models, since the transformer architecture otherwise does not take order into account. despite their widespread use, the structure of absolute position embeddings in nlp models like bert @xcite , roberta @xcite , albert @xcite , and elec-tra @xcite , as well as vision models like the vision transformer @xcite and beit @xcite , remains underexplored. our research aims to address this gap. this paper investigates the structure of learned absolute position embeddings in greater detail than before. specifically, we apply principal components analysis to the learned position embeddings across 12 different transformer-based language models. this yields several novel observations: ## background transformer-based language models, such as those by @xcite , have dramatically changed natural language processing by effectively integrating information across long distances in a sequence. central to the functionality of these models are position embeddings, which enable the encoding of sequence order-an essential aspect in otherwise order-agnostic transformer architectures. content embeddings z i in transformers are constructed as z @xmath0, where x i is the token at position i, e w represents word embeddings, and e p (i) is the position embedding vector of position i; e p will denote the matrix obtained by stacking all row-vectors e p . this setup allows the final representation of tokens in a sequence 1 237 to depend on token positions, which is crucial to adequately model contextual effects in text. while @xcite used a fixed, nonlearnable encoding scheme for position embeddings, subsequent work has aimed to enhance the expressiveness and efficiency of position encoding. bert @xcite and roberta @xcite used learnable, data-driven position embeddings to better capture positional dependencies. albert @xcite refined this approach by introducing factorized embedding parameterizations, reducing model size and complexity while maintaining performance. research has shown that varying word embedding sizes based on frequency can significantly improve computational efficiency and performance @xcite @xcite , but varying sizes between word and position embeddings has not been explored, with standard practice being to use the same dimensionality for both. most work on position embeddings examines their impact on model performance, not their intrinsic properties. existing results show that selfattention tends to localize in models using absolute position embeddings @xcite and that these embeddings have translationequivariant structure @xcite . ## dimensionality analysis of embeddings we now analyze the dimensionality of word and position embeddings in transformer-based language models. a deeper dive into into the structure of position embeddings is reserved for sec. 4. to understand the structural characteristics of position and word embeddings, we extracted both embedding types from various pre-trained transformer models, specifically twelve different versions of albert and bert provided by hugging face @xcite . we then applied principal component analysis (pca) to these embeddings, analyzing each type separately. pca computes a linear transformation that decomposes high-dimensional data into orthogonal vectors representing the primary axes of variation. dimensionality reduction is performed by keeping only the k leading principal components (pcs). from the tables, we see that position embeddings have a significantly lower-dimensional structure compared to word embeddings, suggesting that positional information is encoded more compactly. all albert and bert models considered have 50% of their variance in the first 1.4-5.5% of the principal components or less, while word embeddings require 19-30% of the pcs to achieve the same result. figure 1 graphs the variance explained by individual principal components in detail for the albert v2 models, finding that position-embedding vectors lie almost perfectly on a subspace of 10 to 20 dimensions, whereas word embeddings use the entire 128-dimensional space. ## analyzing the principal components having established the low dimensionality of position embeddings, we next explore what the uncovered principal components represent and how they contribute to the embedding structure. first, we plot (in figure 2 ) the leading principal components of a few models to visually interpret the dominant patterns in the position embeddings. this reveals intriguing patterns. in all cases, the first ten components take the form of smooth, periodic oscillations as a function of position, indicative of simple harmonic structure. although the specific ordering and frequencies change between models, components come in pairs that exhibit similar periodic structure, like sine and cosine representing cyclical motion. for all models except the largest, the highest components plotted appear flat and uniform red (i.e., close to zero), reflecting the limited dimensionality of the position embeddings. figure 5 in the appendix demonstrates, through fourier analysis, that the sequence of principal component scores contains only a few dominant frequencies, which accounts for their periodic appearance. the peak frequencies observed, such as those representing 1, 5, 15, and 49 revolutions as i runs through its full range from 0 to 511 in the case of albert-base-v2, are relatively low. this finding is distinct from the sinusoidal position embeddings described by @xcite , which utilize 512 sinusoids of equal magnitude. unlike the principal component scores, these sinusoids are not mutually orthogonal and are designed with different objectives for encoding position in a sequence. by computing the matrix e p e t p , which contains the inner products between all pairs of position embeddings, it has been found that learned position embeddings tend to exhibit translation equivariance @xcite . in contrast, classic sinusoidal position embeddings display weak inner products between off-diagonal elements, suggesting an absence of such patterns @xcite . by repeating this inner-product experiment, but approximating the position-embedding matrix e p by its top k principal components, we can see how translation-equivariant structure (where each row of the matrix is a translation of the one above it) is rapidly created using only a few principal components for the albert-v2-base model in figure 3 . finally, as pca is a dimensionality-reduction technique, we can visualize all 512 albert-base-v2 position-embedding vectors in two dimensions by means of a scatter plot of their two leading principal components, as shown in figure 4 . we observe a very clear rotational structure, where as the position i goes from 0 to 511, the 2d representation of e p (i) almost completes a full clockwise turn. other principal-component pairs show similar patterns, but complete multiple rotations as i runs through the full range of position indices. figure 4 exhibits two outliers from the circular pattern, namely vectors 0 and 511 (the first and last). this is likely due to how the model is trained: position embeddings are only ever used after being summed with a word embedding, and the the first and the last sequence positions are always assigned the specific tokens "cls" and "sep", respectively. this means that, unlike at all other positions, these two vectors are largely arbitrary, e.g., adding any vector v to e p (0) while subtracting the same vector from e w (cls) leads to the exact same z 0 . ## discussion and implications we have demonstrated that position embeddings operate within a significantly lower-dimensional space compared to word embeddings. this likely reflects their role in encoding less complex, but nonetheless essential structural information. opportunities for transformer models: intuitively, our findings present an opportunity to streamline embeddings and reduce computational demands without compromising the model's ability to interpret linguistic contexts. for example, one could utilize factorized embedding parametrizations of different dimensionalities for position embeddings versus word embeddings, similar to how albert mentions the possibility to use different embedding dimensionalities for different word tokens @xcite , although they opted not to do so. to further refine model inductive biases based on the patterns we observed, learned position embeddings could be initialized or otherwise incentivized to have rotational structure, e.g., being parameterized by sines and cosines with a learnable frequency. this would differ from the rotational position embeddings (rope) of by @xcite , whose rotational frequencies are not learnable. we observed a consistently low-dimensional structure of position embeddings among a wide class of transformer models. this supports the soundness of the heuristic approach for creating position embeddings used in longformer @xcite ) -where pre-trained roberta position embeddings were used as a starting point for training a new model -and further suggests that re-using learned position embeddings from older models may be useful as a general strategy. insights into embedding ordered sequences: transformers, particularly those like albert models which have undergone extensive training, exhibit an intriguing pattern in their position embeddings. these models often utilize approximately 10 principal components-closely aligning with 2 9 = 512, the typical maximum sequence length. this choice of dimensionality suggests that each dimension may function akin to a binary system, with each principal component potentially implementing a sine or cosine curve. such a structure effectively splits the data, allowing for a compact yet robust representation of sequence positions. this method of embedding sequences as concurrent rotations in low-dimensional spaces indicates a standardized approach to processing sequential data via embeddings. this geometric encoding strategy is echoed in findings across several recent studies. @xcite noted that transformer models trained on mathematical tasks often use a "clock algorithm" in their latent spaces, enabling modular arithmetic. similarly, @xcite and @xcite observed analogous rotational patterns in numerical embeddings, whether trained from scratch on mathematical tasks or using language-modeling techniques. these observations highlight the potential of using geometric transformations as a unified method to encode sequential information across diverse applications, like time-series analysis, where precision and optimized data representation are crucial. ## conclusions and future work we have found that learned position embeddings in a range of transformer language models differ from the behavior of word embeddings, in that position embeddings are confined to a low-dimensional linear subspace. we furthermore find evidence that this subspace takes the form of a few orthogonal rotational components at different frequencies. interesting future directions to explore include studying position embeddings in other domains, such as vision, and leveraging our findings to devise more efficient transformer variants with improved inductive biases for modeling sequence data.