Transformer Language Models without Positional Encodings Still Learn Positional Information (2022)

Abstract: Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models and that this phenomenon is robust across different datasets, model sizes, and sequence lengths.Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information.We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position.Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism but also from the effects of the causal mask.

## introduction the attention mechanism @xcite of the transformer @xcite is agnostic to the position and order of tokens in the input sequence. it is therefore common practice to inject positional information via absolute positional embeddings @xcite or relative bias factors @xcite @xcite . here, we demonstrate that transformer language models without any explicit positional information can and do learn an implicit notion of absolute positions that is sufficient to achieve competitive performance. we compare the performance of language models trained with no explicit positional information (nopos language models) to those trained with three different position-aware mechanisms, namely: sinusoidal embeddings @xcite , learned embeddings @xcite , and alibi @xcite . results show that nopos models are competitive with position-aware nopos learned sinusoidal alibi 10 11 12 13 14 perplexity 13.10 13.05 12.93 12.51 figure 1 : transformer language models trained without explicitly encoding positional information (nopos) approach the performance of models trained with various positional encoding methods. all models have 1.3b parameters, and are trained on an excerpt of the pile. we hypothesize that this surprising behavior is tied to the causal attention mask, which implicitly injects positional information into the self-attention layer in order to preserve the autoregressive nature of language models. intuitively, a model that is able to count the predecessors of a given token can essentially infer its absolute position. to test our hypothesis, we run similar experiments for masked language models (mlm) @xcite , which use order-invariant attention (since no causal mask is applied). indeed, bidirectional models fail to converge when position information is absent, substantiating our hypothesis. to conclude, our main contributions are: ## positional encodings transformer models consist of interleaved selfattention and feed-forward layers, which are both order-invariant. therefore, to convey the order of the input tokens, some form of positional information is explicitly introduced into the model. absolute positions are commonly encoded as vectors (one for each position), which are then added to the input tokens' embeddings and fed to the first layer of the transformer. relative positions are typically encoded as biases (added to attention scores) within the self-attention layers. in this work, we consider three popular methods as baselines: learned. embeddings trained to represent absolute positions @xcite . learned positional embeddings are commonly used in mlms @xcite as well as in large autoregressive language models, such as gpt-3 @xcite . sinusoidal. constant vectors computed by a nonparametric function of the input token's absolute position. sine and cosine functions of different frequencies are used, such that each dimension of the positional encoding corresponds to a sinusoid. sinusoidal embeddings were introduced in @xcite for machine translation, and are also used in language modeling @xcite . alibi. attention with linear biases @xcite injects information about the relative distances between tokens by adding negative biases to attention scores, which grow linearly with the distance between each pair of tokens. ## experiment setup intuitively, encoding positional information explicitly is crucial for enabling transformer language models to predict the next token in a sequence. to test this intuition, we compared the validation set perplexity of models trained from scratch with no explicit positional information (denoted as nopos) to those trained with the various positional encoding methods discussed in section 2. we investigated the canonical wikitext-103 setting @xcite , as well as a newer, large-scale setting based on the pile corpus @xcite on model architectures inspired by @xcite , where we cover a spectrum of models sizes and sequence lengths. the canonical setting (wikitext-103). the wikitext-103 corpus @xcite consists of over 100 million words extracted from a set of high-quality wikipedia articles. the corpus is tokenized at the word level, resulting in a vocabulary of over 267k tokens. for this corpus, we used the adaptive embedding transformer model of @xcite , which contains 16 transformer layers with 1024 model dimensions, 4096 feed-forward dimensions, and 8 attention heads. overall, this model has 247m parameters in total. we trained with their exact optimization hyperparameters, as implemented in fairseq @xcite , with the exception of the input sequence length, which was shortened to 512 tokens (instead of 3072), as in @xcite . see app. c for detailed hyperparameters. the large-scale setting (the pile). the pile @xcite ) is an 800gb english text dataset composed of common crawl and 22 other diverse sources. for our experiments, we used 2 out of 30 shards; foot_0 of these, we filtered out the github and dm mathematics sources and removed the shortest 1% and longest 1% of examples from each source to reduce noise. we used gpt-2's tokenizer @xcite to convert the text into token sequences over a vocabulary of 50k tokens. we randomly sampled a validation set of 2000 documents (2.6m tokens) from the corpus, while the remaining 15m documents (21b tokens) comprised the training set. the baseline model in this setting follows the 1.3b parameter architecture of @xcite , also known as gpt-3 xl: 24 transformer layers with 2048 model dimensions, 8192 feed-forward dimensions, and 32 attention heads. the default input sequence length is 1024 tokens. we refer to app.c for detailed hyperparameters. to demonstrate the consistency of our results in different settings, we perform two scaling experiments. we first scale the model size by experimenting with the small (125m parameters), medium (350m parameters), large (760m parameters) and the xl (1.3b parameters) variants of the brown et al. (2020) architecture on the pile settings. in addition, we evaluate the effect of varying the sequence length using the xl (1.3b parameter) model. specifically, we experiment with sequences of lengths {256, 512, 1024, 2048}. last, to shed additional light on differences between the nopos model to other methods, we compare the model's performance on different parts of the sequence. details of this analysis and results are given in app. a. ## results as training models at the 1.3b parameter scale is resource-intensive, we publicly release our trained models for future research and analysis. 3 model size 125m 350m 760m 1.3b figure 2 : through probing, we find that the nopos model behaves similarly to models that use absolute learned position embeddings. we evaluated performance using mean absolute distance on 1.3b parameter models trained on the pile. ## analysis in this section, we examine whether the nopos model is able to encode positional information and show that such information is essential for its success. ## conjecture how do transformers without explicit positional encoding learn absolute positions? we conjecture that the causal attention in autoregressive transformer language models allows them to predict the number of attendable tokens at each position, i.e. the number of tokens in the sequence that precede the current one. such a mechanism could effectively encode the absolute position of each token into its vector representation. indeed, our analysis (section 5) reveals that some notion of absolute positions exists in the hidden layers of language models even when they are trained without explicit positional encoding, and that this information is acquired throughout the first few layers. on the other hand, bidirectional transformer encoders (which are used in masked language modeling, e.g. @xcite do not contain causal attention masks or any other limitation on the attention mechanism; thus, they should be unable to learn absolute positions without explicit positional encoding. we tested this corollary by training a masked language model based on roberta large @xcite on the pile (see app. c for hyperparameters). ## related work while there has been ample research on positional encoding variants, there has been relatively little prior work that investigate models' ability to infer @xcite trained with various positional encoding methods on an excerpt of the pile @xcite . the model architecture is based on roberta large @xcite , and processes 128 tokens per sequence. while position-aware models converge to very low perplexities, training without positional encodings (nopos) fails. ## conclusion we show that, contrary to popular belief, transformers language models do learn positional information even when are not provided with any explicit positional encoding. our experiments systematically demonstrate that this phenomenon is robust across different language modeling settings, and that one can approximate the absolute position of each token from the model's internal representations to a surprising degree. however, this phenomenon does not extend to transformer encoders trained on the mlm objective. we conjecture that the causal attention mechanism, which limits attention in one direction of the sequence, is responsible for implicitly imbuing the transformer with positional information.