What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020)

Abstract: In recent years, pre-trained Transformers have dominated the majority of NLP benchmark tasks. Many variants of pre-trained Transformers have kept breaking out, and most focus on designing different pre-training objectives or variants of self-attention. Embedding the position information in the self-attention mechanism is also an indispensable factor in Transformers however is often discussed at will. Hence, we carry out an empirical study on position embedding of mainstream pre-trained Transformers mainly focusing on two questions: 1) Do position embeddings really learn the meaning of positions? 2) How do these different learned position embeddings affect Transformers for NLP tasks? This paper focuses on providing a new insight of pre-trained position embeddings by feature-level analysis and empirical experiments on most of iconic NLP tasks. It is believed that our experimental results can guide the future works to choose the suitable positional encoding function for specific tasks given the application property.

## introduction word ordering often determines the meaning of a sentence; therefore how to utilize the position information of a word sequence has been an important topic in nlp and widely investigated recently. a common approach for modeling word ordering is to use recurrent neural networks (rnn), such as long short-term memory (lstm) @xcite or gated recurrent unit (gru) @xcite , which use a hidden state to represent the information of an ordered sequence and update model weights by backpropagation through time (bptt) @xcite ; thus the ordering information can be modeled by this structure. however, rnn and bptt are very inefficient in modern gpu computation due to the difficulty of parallelization with the time dependency. to solve this problem, recent work, such as convolutional seq2seq @xcite and transformers @xcite which apply convolutional neural network (cnn) @xcite and self-attention respectively, succeed to eliminate the time dependency to take the computational advantage of gpu. instead of storing the information of ordered sequences, these models utilize the position information by using a feature-level positional encoding. for example, convolutional seq2seq proposed learnable position embeddings to represent the positions in a sequence. recently, various pre-trained transformer language models keep breaking state-of-the-art results in numerous nlp tasks. there are many different ways to pre-train a transformer language model. for example, using an encoder, decoder, or the whole part of the transformer, adapting the selfattention masks, or training with different objectives @xcite @xcite @xcite @xcite . however, in terms of positional encoding, most work only used a learned position embedding which is originally proposed in convolutional seq2seq @xcite without any analysis, even different objectives may learn completely different position information. motivated by the above observations, our goal is to investigate what position information the pre-trained transformers could learn under different settings. we conduct a deep analysis of the learned position embeddings among three iconic pre-trained transformer language models: bert @xcite , roberta @xcite and gpt-2 @xcite . to examine the performance of different nlp types, we conduct the experiments on text classification, language modeling, and machine translation, and empirically analyze and explain the meaning and influence of position embeddings from different aspects. the contributions of this paper are 3-fold: ## related work the concept of using position embedding on position-insensitive models was first proposed by convolutional seq2seq @xcite , which built an encoder-decoder architecture on convolutional neural networks. @xcite proposed transformers that used the self-attention mechanism in the basic blocks. because the attention mechanism is position-insensitive, it proposed a pre-defined sinusoidal function as positional encoding. pre-trained language models became a trend among many nlp tasks after @xcite introduced elmo. affected by elmo, ope-nai gpt @xcite is the first pretrained language model using a transformer architecture, then many different variant of pre-trained transformer including bert @xcite , roberta @xcite and gpt-2 @xcite started evolving the researches of nlp tremendously. in transformers, the attention values are the same in each input position. thus, @xcite proposed a relative position representation in the attention level to address this issue. @xcite used a segment-level recurrence mechanism on transformers and also utilized an adaptive version of relative position embeddings inspired by @xcite . furthermore, @xcite extended the embedding space from real numbers to complex values , and also proposed a new learnable positional encoding function instead of a simple position embedding mapping. ## transformer transformer is an encoder-decoder sequence-tosequence model proposed by @xcite . in the architecture, transformer is composed of self-attention blocks that are position-insensitive modules. therefore, a positional embedding should be considered together with the nlp tasks. to elaborate on the experiments we conduct, this section briefly introduces transformers. input representation due to the property of position-insensitive in the attention module, the input representations should also contain the position information. in transformers @xcite , a word embedding is directly added with the positional encoding as the final representation: multi-head self-attention the attention mechanism is often used in an encoder-decoder architecture, and there are many variants of attention implementations @xcite . in transformers, the scaled dot-product attention is applied: transformer encoder a transformer encoder layer is composed of multi-head self-attention following a position-wise feed-forward network (ffn) with the residual connection @xcite and layer normalization @xcite : transformer decoder the transformer decoder is also stacked by self-attention blocks, and it only has two major differences from the encoder: therefore, the transformer decoder can decode tokens autoregressively like other conventional language models such as rnn. ## position embedding analysis in this section, we conduct feature-level analyses of the pre-trained position embeddings of two transformer encoders: bert @xcite and roberta @xcite , one transformer decoder: gpt-2 @xcite , and also the sinusoidal function proposed by @xcite is defined as ## performance effect in addition to the behavior analysis, we are interested in the performance difference of positional embeddings for different nlp tasks, where we conduct text classification (encoding), language modeling (decoding) and machine translation (encoding and decoding). note that each chosen task has its own important property where position information may cause different effects in transformers. ## conclusion this paper investigates the implicit meaning of pretrained transformer position embeddings. transformer encoders learn the local position information that can only be effective in masked language modeling. on the other hand, the transformer decoders for autoregressive language modeling actually learn about absolute positions. the empirical experiments on the pre-trained position embeddings validate our hypothesis. we also show that different nlp tasks with different model architectures and different training objectives may utilize the position information in different ways. as a result, it is believed that this study will benefit future work about choosing suitable positional encoding functions or designing other modeling methods for position information in the target nlp tasks based on their properties.