Absolute Position Embedding Learns Sinusoid-like Waves for Attention Based on Relative Position (2023)

Abstract: Attention weight is a clue to interpret how a Transformer-based model makes an inference. In some attention heads, the attention focuses on the neighbors of each token. This allows the output vector of each token to depend on the surrounding tokens and contributes to make the inference context-dependent. We analyze the mechanism behind the concentration of attention on nearby tokens. We show that the phenomenon emerges as follows: (1) learned position embedding has sinusoid-like components, (2) such components are transmitted to the query and the key in the self-attention, (3) the attention head shifts the phases of the sinusoid-like components so that the attention concentrates on nearby tokens at specific relative positions. In other words, a certain type of Transformer-based model acquires the sinusoidal positional encoding to some extent on its own through Masked Language Modeling.

## introduction the architecture of transformer @xcite is symmetric with respect to the token position and it captures word order only through the position embedding included in the input. thanks to this design, transformer can flexibly learn relationships between tokens while allowing parallelization. to derive a representation of a context, previous language models have used, for instance, recurrent units to process tokens in the given token order, or convolution to aggregate tokens within a given range. in these models, the property of translation invariance of language has been captured through the architecture of the models. in contrast, transformer receives all tokens at the same time and does not restrict the positions of the tokens on which each token depends; in exchange for it, the translation invariance has to be learned rather than imposed by the architecture. position embedding is thus the key to make inferences in transformer context-dependent. while transformer uses deterministic sinusoidal position embedding, bert @xcite uses a learnable absolute position embedding. the latter learns the positional representation only through word cooccurrences. @xcite and @xcite investigated the attention weights of the self-attention in bert and found that the attention in some heads is largely determined by relative positions (figure 1 ). this implies that even when the absolute position embedding is used, self-attention can make an inference depending on relative position. @xcite compared various position embedding methods based on the performance in downstream tasks. their results showed that local translation invariance and asymmetric position bias with respect to the direction improved the performance. @xcite observed that some columns of absolute position embedding were periodic. @xcite showed that position representation was periodic even in hidden representations. however, it is not clear how the periodicity is used in the model. in this work, we analyze how attention depends on relative position. as a result, we show that the relative positional dependence of attention emerges due to the following factors. thus, it becomes partially clear how the selfattention equipped with learnable absolute position embeddings makes inferences based on context. however, it is suggested that, when the attention is strongly concentrated on the adjacent tokens, the word embedding is also a factor that enables inference based on relative position ( §5). ## background in this section, we review the multi-head selfattention mechanism and position embedding. ## relative position dependence of attention in this section, we demonstrate that the attention depends on the relative position. specifically, we analyze in how many heads the attention is dependent on the relative position, and the variation in the direction and concentration of the attention across the heads. the visualizations of the attention weight a lh in figure 1 shows that, in (a) and (d), each token strongly attends only to the adjacent token, while in (b) and (e), the attention is put on a few nearby tokens on the left or right side. we show that such a pattern is largely invariant with respect to the inputs, by clustering the patterns of the attention on different inputs. to focus on the relation between the attention and the relative positions, we summarized the attention weight for each input in each head into a vector and applied k-means clustering to them. for that, we defined t-offset trace tr t as follows: these vectors represent the trend of attention a lh with respect to relative position. we input 100 sentences of length 512 created from wikitext-2 @xcite into roberta and computed vectors a lh for every head. figure 2 shows the results of applying the k-means to a total of 100 × 12 × 12 vectors a lh when the number of clusters was set to 6. in the clusters named leftward and next-to-left (resp. rightward and next-to-right), the attention is concentrated on the nearby tokens on the left (resp. right). particularly, in the clusters named next-to-left and next-to-right, the attention is strongly concentrated on the adjacent tokens. we found that, for each head, the vectors a lh corresponding to the 100 inputs were grouped into one or two clusters. this means that there are some heads that always attended to the same direction for all the input. ## attention to nearby tokens in this section, we show that the attention depends on relative position due to periodic components in the position embeddings. first, we show that the learnable absolute position embeddings acquire several limited frequency components. second, we show that some attention heads extract the periodicity derived from position embeddings. finally, we show that the concentration of attention on the nearby tokens is due to the periodicity. ## attention to the adjacent tokens figure 9 shows that, in head(8, 9), there are not only sinusoid-like waves but also spikes at the relative position @xmath0. in this section, we dive deeper into the fact that, in some heads, the attention is focused only on the adjacent token. we analyze how the attention changes when we modify either the position embedding or the word embedding component. the following are the results on the modified inputs and insights from them. baseline the input is the sum of word embedding and position embedding as usual. this result is shown in figure 8 and 9 . only position embedding the word embeddings are replaced with zero vectors. figure 11a shows that spikes of cross-correlation do not appear for this case. this suggests that position embedding is not the only cause of the spikes. only word embedding the position embeddings are replaced with zero vectors. figure 11b shows that most of the cross-correlations are flat for this case. it suggests that word embedding contributes less to make the attention dependent on relative position. however, it is interesting that the cross-covariance at @xmath1.e., left neighbor) is relatively large even without position information. shuffling word embeddings the order of word embeddings is shuffled. figure 11c shows that spikes appear at the relative position @xmath2. it suggests that the contribution of position embedding is significant in determining the direction of attention, since the attention is focused on the preceding word regardless of what it actually is. as mentioned at the end of section 4.3 and also suggested by figure 11a , it is unlikely that position embedding is the only cause of the strong concentration of attention on the adjacent tokens. however, if the word embeddings identified the adjacent tokens (i.e., if the narrow peak of cross-correlation appeared due to an interaction of the word embeddings of a frequently occurring bigram), the attention would have been moved to non-neighboring positions by the shuffling of word embeddings, but this was not the case. it is thus suggested that the concept of adjacency in roberta is built upon both word meaning and positional information. ## remark on the learning process of position embeddings the training data of the masked language modeling is only word sequences without positional information. it suggests that the relative position dependence of attention is acquired by the combination of two factors: (1) the linguistic property that related words tend to appear nearby due to grammatical rules and collocations, and (2) the property that attention is focused on the word that is syntactically or semantically related and hence gives clue to fill the masked token. in appendix e, we demonstrate that the position representation can be acquired with a much smaller amount of training data than pre-training. recently, there are many variants of position embedding and @xcite surveyed them exhaustively. in particular, rotary position embedding (rope) @xcite , a type of relative position embedding (rpe) @xcite , relates to the property that the self attetnion acquires the rotation matrix through pre-training. to acquire relative positional dependence, rope widens the angle between query and key proportionally to the relative position, while pre-trained self-attention rotates the hidden states containing absolute positional bias by the same angle regardless of position. in other words, ape and selfattention, which are distant components, must acquire frequency and angle of rotation, respectively, to satisfy the relation eq. ( 27 ). if translation invariance is essential to language understanding, the cost of discovering this relation is a possible reason why apes are inefficient to learn compared to rpes. ## conclusion we analyzed the concentration of attention based on relative position in the self-attention using the learnable absolute position embedding. as a result, we showed that it is due to relative phase shifts of the periodic components in the query and the key derived from position embeddings. our results explain in part that absolute position embedding allows inference based on relative position.