Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings (2023)

Abstract: The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.

## introduction & related work transformer models have become the backbone of natural language processing applications @xcite @xcite . within the transformer architecture, there are two main categories: 1) bidirectional models, such as bert @xcite , that are trained using the masked language modeling objective, and 2) (causal) language models, such as gpt @xcite , that are trained using the traditional language modeling objective. both of these categories share the common feature of using positional embeddings for encoding token distance. whether positional embeddings are truly essential has been a subject of ongoing research. while they have been considered necessary for bidirectional transformer models @xcite @xcite , the situation is different for transformer language models @xcite ; tsai † correspondence to: tachungc@andrew.cmu.edu layer norm multi-head attention et @xcite @xcite . in transformer language models, the removal of positional embeddings results in only a marginal decline in performance, while enabling more efficient training @xcite . in addition to empirical evidence, it has been proven @xcite ) that transformer language models without positional embeddings are turingcomplete and able to model sequences akin to recurrent neural networks @xcite . despite this, it remains an open question where positional information is stored in the absence of positional embeddings. this motivates further investigation into individual operations within a transformer layer. the example architecture of a pre-ln (xiong work is shown in figure 1 . 1 we hereinafter refer to this configuration as tlm. our primary focus is on the multi-head attention (mha) module of a randomly initialized tlm, as it is the only module that allows inter-token information exchange. to gain a deeper understanding, we compute the mean and variance of mha outputs. to our surprise, we discover that the variance already encodes latent positional information, with later tokens in a sequence displaying smaller variance. this motivates us to quantify the variance by deriving the output distribution after mha operations. finally, through empirical validation using a fully pre-trained tlm, we confirm thatthe same variance shrinkage effect persists after extensive gradient updates. to the best of our knowledge, we are the first to identify and quantify the latent positional information in tlms. our results provide theoretical insights into the removal of positional embeddings, enabling more efficient pretraining of future tlms. ## probing experiments given bert and tlm (gpt) with positional embeddings removed, prior work @xcite shows that only tlm is able to maintain the same language modeling performance as its original version with positional embeddings. the discrepancy might be explained by the fact that only tlm encodes positional information within its layers, as shown by the position probing experiment in @xcite . since both bert and tlm have access to the same semantic input and the only difference is the use of causal attention masks in tlm, we hypothesize that the positional informa-1 post-ln places layer norm at different positions. it is the configuration used in bert @xcite . to further explore this hypothesis, we use a randomly initialized and frozen tlm to eliminate any semantic influence and focus solely on the architectural design. additionally, to prevent the model from memorizing the order of input sequences, we do not perform embedding lookups and feed the model with randomly sampled input vectors. a trainable two-layer linear classifier with relu activation in between was appended to the tlm to probe the position of each token (further details can be found in appendix b). we plot the mean absolute error (mae) w.r.t the number of transformer layers in figure 2 . the plot indicates a randomly initialized and frozen tlm with randomly sampled input vectors inherently provides positional information, with an increase in the number of layers resulting in higher probing performance. this surprising outcome prompts further investigation into the encoding of latent positional information inside the tlm architecture. ## theoretical analysis we dissect the inner workings of a tlm by deriving the distribution of tlm operations in the hope that they elucidate where the latent positional information is stored. the derivation is made possible thanks to the usage of a randomly initialized and frozen tlm. we adopt the initialization settings in accordance with those employed in gpt @xcite . wlog, our derivation is limited to the operations of the first layer in a tlm and the ffn component is omitted (justified in §3.4). the hyperparameters utilized in the simulations are: hidden dimension @xmath0, number of attention heads @xmath1, head dimension d/@xmath2, sequence length @xmath3, standard deviation for initialization σ = 0.02. all proofs of lemmas are deferred to appendix a. given a sequence of randomly sampled input embeddings {x m } l @xmath4, where each element of x @xmath5.i.d from n (0, σ 2 ), a tlm consists of the following operations: ## discussions why are the positions of later tokens in a sequence harder to be predicted in figure 3 of @xcite ? lemma 3 states the variance is inversely proportional to the position m, so the variance of later tokens (large m) plateaus, resulting in a harder numerical optimization problem. this also suggests a potential downside of removing positional embeddings: it might be challenging for the model to infer positional information of the later tokens in extremely long input sequences. why do lower layers (closer to input) give worse probing performances in both figure 2 and @xcite ? this can be explained by figure 4. most of the positions at the 0 th layer have tiny variance (exp(-10) = 4.5e -5 ), which again poses a difficult numerical optimization problem. why does bert fail to converge without positional embeddings? in a bert model @xcite , each token has access to all the other tokens, making the variance at all positions d 2 σ 4 l . therefore, a bert model cannot utilize variance differences as its positional indicator. ## post-training results our derivations only apply to the initial stage where the tlm and input embeddings are randomly initialized, which may not hold true after gradient updates. it is essential to verify the existence of variance properties and lemmas on a fully pre-trained tlm on openwebtext2 (details in appendix c). we expect that the properties of lower layers of a pre-trained tlm should align more closely with the theoretical results for two reasons: 1) there are more steps between the lower layers and the final language modeling loss, resulting in smaller gradients and thereby fewer parameter updates, and 2) lower layers typically encode more lowlevel information dependent on positional information @xcite . figures 6 and 7 demonstrate that the 0 th (lowest) layer exhibits highly similar cumulative attention probability and decay-with-position variance as the theoretical results. in contrast, higher layers deviate from the analyses in § 3. we posit that the model learns to rely more heavily on semantic rather than positional information. this also explains why 0 1 2 3 4 5 6 log positions 2 1 0 1 2 3 log variance layer 0 layer 6 layer 11 figure 7: we plot the log positions w.r.t their log variance for @xmath6. we average over 500 samples. ## conclusion we mathematically analyzed a randomly initialized transformer language model without positional embeddings. we showed that the variance of the selfattention output decreases as the position increases, which serves as an indicator for positional information. we validated that, after extensive gradient updates, the low layers of a pretrained language model still exhibit highly similar variance reduction behaviors. our results pave the way for the pretraining of more efficient and positional embedding-free transformer language models.