A Simple and Effective Positional Encoding for Transformers (2021)

Abstract: Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.

## introduction transformers are sequence-to-sequence models that achieve state of the art performance in many natural language processing (nlp) tasks, such as machine translation, language modeling and question answering @xcite @xcite . transformers have two major components: self-attention and a position-wise feed forward layer. both are permutation equivariant and are not sensitive to the order of input tokens. to make these models position-aware, the position information of the input words is typically added as an additional embedding to the input token embeddings @xcite . for example, input embedding (w ) of a sentence is added to the position embeddings (p ), resulting in input w + p to the transformer. these position embeddings only depend on the location the word appears. for multi-segment tasks, additional segment embeddings can be added just like the position embeddings @xcite . there have been multiple works exploring different ways to include position information in transformers @xcite @xcite . many of those note the advantages of using a relative position encoding scheme over absolute position encodings (see also fig 1 ). however what causes this difference is not clear. @xcite have shown that transformers with absolute position encodings are universal approximators of all sequence to sequence functions, proving that absolute position encodings can capture the position information. hence what causes the superiority of relative position encodings? a systematic study and understanding of the benefits and drawbacks of different position encoding methods is missing. @xcite hypothesised that the cross correlation between word and position embeddings while computing attention could be the cause of poor performance of absolute position encodings. however such cross terms are present in some of the relative position encoding methods @xcite , and these methods perform on par or better than the other position encoding schemes (see §4). in this paper we undertake a systematic study to understand different position encoding methods. we argue that absolute position embeddings mainly suffer from being added at the input. we show, with our experiments on classification, question answering and machine translation tasks, that absolute position encodings added to attention matrices with different parameters for each head improves significantly over absolute position encodings added to the input. this highlights that where the position information is included in the transformer is important, providing an explanation for the gap in performance between absolute and relative position encodings. we also compare different position encodings and the effect of sharing position encod- @xcite , xtreme @xcite and one neural machine translation dataset wmt 18 @xcite . absolute positional encoding (diet-abs) can achieve better performance than the relative counterpart (diet-rel), showing the importance of designing the right position encoding method. ## position encoding for transformers in this section, we briefly review the transformer models @xcite and discuss previous improvement of position encoding and analyze the limitation of the additive position embedding proposed in the initial and widely-adopted transformer model. ## proposed position and segment encodings in the previous section, we learned about the limitations of input additive positional embeddings and existing works. based on these observations, we propose two minimal/efficient ways to incorporate (absolute/relative) positional encodings along with a novel absolute segment encoding approach. by decoupling position and segment from token embeddings we match the sota performance while improving training/inference time (see §3.3). ## experiments in this section, we present our experimental results comparing different position and segment encoding approaches discussed in earlier sections. we conduct experiments in three different settings to cover a wide range of use cases. first, we examine the results of a popular transfer learning approach from masked-lm pretraining to the end tasks in glue @xcite . second, we study zero-shot cross-lingual transferability of the multilingual pretrained models @xcite to classification and question answering tasks in the xtreme benchmark @xcite . lastly, we consider training transformer models from scratch for machine translation. we compare the following positional encoding approaches -absolute positional embedding @xcite , relative positional embedding @xcite , combined absolute and relative positional encoding @xcite , relative scalar approach @xcite , our proposed diet-abs and diet-rel per-head positional encoding approaches. we denote the methods that add position/segment information directly to input token embeddings with input, and methods that add position/segment information directly in attention layer with per-head. for complete experimental setup, see appendix a. ## conclusion in this paper we theoretically and empirically examined the limitation of additive position embedding at input and showed that having per-head position embeddings results in better performance. we argued that the superior performance of some of the relative position encoding methods come from their per-head addition to attention matrix rather than the position information being relative vs absolute. indeed we show that using absolute position encodings per-head results in better performance. motivated by this we propose a simple per-head position and segment attention method that achieves the state-of-the-art performance on multiple nlp tasks and is more computationally efficient than existing approaches. choose d @xmath0. now chosing p with zeros in the first n -d p columns and identity in the last d p columns ( @xmath1,n-dp , i dp,dp ]) gives p @xmath2,n-dp 0 n-dp,dp 0 dp,n-dp i dp,dp . let @xmath3. we have trainable position embeddings @xmath4, which are added to the input sequence before feeding into the model g. for a given input x and label y, the objective for a loss function ℓ is as follows: theorem 2. let x and p be trainable embedding matrices in r n×d . then the gradients of the loss function in equation (5), at any point (x, y), and for any differentiable functions ℓ and g, are same for x and p. remarks. this theorem shows us that the gradients are same for the input token embeddings and position embeddings. while in standard nlp tasks the inputs x can be different in each step due to different input tokens being present in each mini batch, the result still suggests that additive position embedding can limit the model from learning the relative importance of position encodings with respect to token embeddings based on the training task at hand. proof of theorem 2. the above theorem follows by just computing the gradients and showing they are equal for each step. gradients of the above objective w.r.t x and p are as follows. the above computation of gradient follows from chain rule. this shows that the gradients of l w.r.t. x and p are the same.