O N P OSITION E MBEDDINGS IN BERT
Anonymous authors
Paper under double-blind review
A BSTRACT
Various Position Embeddings (PEs) have been proposed in Transformer based architectures (e.g. BERT) to model word order.
These are empirically-driven and perform well, but no formal framework exists to systematically study them.
To address this, we present three properties of PEs that capture word distance in vector space: translation invariance , monotonicity , and symmetry .
These properties formally capture the behaviour of PEs and allow us to reinterpret sinusoidal PEs in a principled way.
Moreover, we propose a new probing test (called ‘identical word probing’) and mathematical indicators to quantitatively detect the general attention patterns with respect to the above properties.
An empirical evaluation of seven PEs (and their combinations) for classiﬁcation (GLUE) and span prediction (SQuAD) shows that: (1) both classiﬁcation and span prediction beneﬁt from translation invariance and local monotonicity, while symmetry slightly decreases performance; (2) The fully-learnable absolute PE performs better in classiﬁcation, while relative PEs perform better in span prediction.
We contribute the ﬁrst formal and quantitative analysis of desiderata for PEs, and a principled discussion about their correlation to the performance of typical downstream tasks.
I NTRODUCTION
Position embeddings (PEs) are crucial in Transformer-based architectures for capturing word order; without them, the representation is bag-of-words.
Fully learnable absolute position embeddings (APEs) were ﬁrst proposed by Gehring et al.
(2017) to capture word position in Convolutional
Seq2seq architectures.
Sinusoidal functions were also used with Transformers to parameterize PEs in a ﬁxed ad hoc way (Vaswani et al., 2017).
Recently, Shaw et al.
(2018) used relative position embedding (RPEs) with Transformers for machine translation.
More recently, in Transformer pretrained language models, BERT (Devlin et al., 2018;
Liu et al., 2019) and GPT (Radford et al., 2018) used fully learnable PEs.
Yang et al.
(2019) modiﬁed RPEs and used them in the XLNet pre-trained language model.
Ke et al.
combined APEs and RPEs in pre-trained language models.
To our knowledge, the fundamental differences between the various PEs have not been studied in a principled way.
We posit that the aim of PEs is to capture the sequential nature of positions in vector space, or technically, to bridge the distances in N (for positions) and R D (for position vectors).
We therefore propose three expected properties for PEs: monotonicity , translation invariance , and symmetry 1 .
Using these properties, we formally reinterpret existing PEs and show the limitations of sinusoidal
PEs (Vaswani et al., 2017): they cannot adaptively meet the monotonicity property – thus we propose learnable sinusoidal PEs. We benchmark 13 PEs (including APEs, RPEs, and their combinations) in GLUE and SQuAD, in a total of 11 individual tasks.
Several indicators are devised to quantitatively measure translation invariance, monotonicity, and symmetry, which can be further used to calculate their statistical correlations with empirical performance in downstream tasks. We empirically ﬁnd that both text classiﬁcation tasks (in GLUE) and span prediction tasks (SQuAD V1.0 and V 2.0) can beneﬁt from monotonicity (in nearby offset) and translation invariance (in particular without considering special tokens like [CLS] ), but symmetry decreases performance since it can not deal with directions between query vectors and key vectors when calculating attentions. Plus, models with unbalanced attention regarding directions (generally attending more to preceding tokens than to succeeding tokens) slightly correlate with better performance (especially for span prediction tasks).
Experiments also show that the fully-learnable APE performs better in classiﬁcation, while RPEs perform better in span prediction tasks .
This is explained by our proposed properties as follows:
RPEs perform better in span prediction tasks since they meet better translation invariance, monotonicity (especially in a small offset), and asymmetry; the fully-learnable APE which does not strictly have the translation invariance and monotonicity properties during parameterizations (as it also performed worse in measuring translation invariance and local monotonicity than other APEs and all RPEs) still performs well because it can ﬂexibly deal with special tokens (especially, unshiftable [CLS] ).
Regarding the newly-proposed learnable sinusoidal PEs, the learnable sinusoidal APE satisﬁes the three properties to a greater extent than other APE variants, and the learnable sinusoidal RPE exhibits better direction awareness than other PE variants.
Experiments show that BERT with sinusoidal APEs slightly outperforms the fully-learnable APE in span prediction, but underperforms in classiﬁcation tasks.
Both for APEs and RPEs, learning frequencies in sinusoidal PEs appears to be beneﬁcial.
Lastly, sinusoidal PEs can be generalized to treat longer documents because they completely satisfy the translation invariance property, while the fully-learnable APE does not.
The contributions of this paper are summarised below: • We propose three principled properties for PEs.
These properties are either formally examined or empirically evaluated by an Identical Word Probing test. Moreover, we devise quantitative indicators to measure how much BERT models with individual PEs satisfy the above properties.
• We benchmark 13 PEs (including APEs, RPEs and their combinations) in GLUE, SQuAD V1.1 and SQuAD V2.0, in a total of 11 individual tasks. Furthermore, we experimentally evaluate how the performance in individual tasks beneﬁts from the above properties. • We propose two new PEs to extend ﬁxed sinusoidal PEs to learnable versions in both APEs and RPEs. Interestingly, the learnable sinusoidal APE well satisﬁes the above properties, whereas the learnable sinusoidal RPE has better direction awareness than other RPE variants.
P ROPERTIES OF P OSITION E MBEDDINGS
Gehring et al.
(2017); Vaswani et al.
(2017) use absolute word positions as additional features in neural networks.
Positions x ∈ N are distributively represented as an embedding of x as an element (cid:126)x ∈ R D in some Euclidean space.
By standard methods in representation learning, similarity between embedded objects (cid:126)x and (cid:126)y is typically expressed by an inner product (cid:104) (cid:126)x, (cid:126)y (cid:105) , for instance the dot product gives rise to the usual cosine similarity between (cid:126)x and (cid:126)y .
Generally, if words appear close to each other in a text (i.e., their positions are nearby), they are more likely to determine the (local) semantics together, than if they occurred far apart.
Hence, positional proximity of words x and y should result in proximity of their embedded representations (cid:126)x and (cid:126)y .
One common way of formalizing this is that an embedding should preserve the order of distances among positions 2 .
We denote φ ( · , · ) as a function to calculate closeness/proximity between embedded positions, and any inner product can be a special case of φ ( · , · ) with good properties.
We can express preservation of the order of distances as: For every x, y, z ∈ N ,
[Equation]
Note that on the underlying space, the property in Eq.
(1) has been studied for almost 60 years (Shepard, 1962), in both algorithmics (Bilu & Linial, 2005; Badoiu et al., 2008; Maehara, 2013), and machine learning (Terada & Luxburg, 2014; Jain et al., 2016) under the name ordinal embedding .
As we are interested in the simple case of positions from N , Eq.
(1) reduces to the following property:
Property 1.
Monotonicity : The proximity of embedded positions decreases when positions are further apart:
[Equation]
A priori, a position embedding might treat every element N individually.
However, considering pairs of positions based on their relative proximity (rather than the absolute value of the positions), can lead to simpliﬁed and efﬁcient position embeddings (Wang et al., 2020).
Such embeddings satisfy translation invariance :
Property 2.
Translation invariance : The proximity of embedded positions are translation invariant:
[Equation]
Finally, since the inner product is symmetric, we also consider whether φ ( · , · ) is symmetric:
Property 3.
Symmetry :
The proximity of embedded positions is symmetric,
[Equation]
There is no generally accepted standard set of properties for position embeddings; based on prior work as described above, we posit that the above properties are important, and now examine several existing PEs in relation to these properties, either formally (in Sec. 3) or empirically (in Sec. 4).
U NDERSTANDING PE S VIA THE PROPERTIES
PEs come in two variants: absolute PEs (APEs) where single positions are mapped to elements of the representation space, and relative PEs (RPEs) where the difference between positions (i.e., x − y for x, y ∈ N ) is mapped to elements of the embedding space.
For Transformer-based architectures, the difference between APEs and RPEs manifests itself in the attention mechanism, in particular how the matrices of query, key, and value weights W Q , W K , and W V are used to calculate attention in each attention head.
Consider two positions x, y ∈ N , let WE x be the word embedding of the word at position x , and let P x and P x − y be the embeddings of the position x and relative position x − y , respectively.
The query-key-value vector for the word at position x is typically calculated as below for APEs and RPEs 3 respectively:
[Equation]
Observe that while the APEs calculation is linear in ( W Q , W K , W V ) with the word and position embeddings merged into the coefﬁcient, the RPEs calculation is afﬁne, with the relative position embedding P x − y acting as an offset independent of the word embedding WE x .
In Transformers, the resulting representation is a sum of value vectors with weights depending on
A = QK T , that is, Attention ( Q, K, V ) =
softmax( QK T / √ d k ) V .
In the rest of the paper, we examine PEs in the above architecture with respect to the properties introduced in Section 2.
In particular, we study four well-known variants of PEs: (1) the fully learnable APE (Gehring et al., 2017), (2) the ﬁxed sinusoidal APE (Vaswani et al., 2017), (3) the fully learnable RPE (Shaw et al., 2018), and (4) the ﬁxed sinusoidal RPE (Wei et al., 2019).
3. U NDERSTANDING S INUSOIDAL PE S
With a sinusoidal parameterization in PEs, we may use a speciﬁc proximity, i
[Equation]
e., an efﬁcient inner product like a dot product, to check if the sinusoidal form of PEs meets the above properties.
The dot product between any two position vectors is
[Equation]
[Equation] [Table] Note that sinusoidal PEs satisfy both Property 2 ( translation invariance ) because the inner product is only associated with its position difference x − y , and Property 3 ( symmetry ), because the dot product itself is symmetric: (cid:104) (cid:126)x, (cid:126)y (cid:105)
= (cid:104) (cid:126)y, (cid:126)x (cid:105) .
Note also that checking Property 1 is equivalent to checking monotonicity of the map ψ ( m ) =
(cid:80) D/
i
[-=
-]=1 cos( ω i m ) .
ψ ( m ) is monotone on intervals where its ﬁrst order derivative ψ (cid:48) ( m ) =
(cid:80) D/ i
=1 − ω i sin( ω i m ) does not change sign, and these intervals depend on the choice of ω i .
With ﬁxed frequencies ω i
=
(1 / 10000) 2 i/D , it is monotonous when m is roughly between 0 and 50, indicating that it can only strictly perceive a maximum distance ofand it is insensitive to faraway distances (e.g. longer than 50).
Although sinusoidal PEs with ﬁxed frequencies (i.e., ω i =
(1 / 10000) 2 i/D ) are common in APEs and RPEs, we argue that learning these frequencies is useful because it can adaptively adjust intervals of monotonicity (they do not have to be 0-50 as in the ﬁxed sinusoidal APE) 4 .
With trainable frequencies, we can adaptively allocate a number of frequencies in a data-driven way.
App. A. explains the expressive power of sinusoidal PEs with trainable frequencies from the perspective of the Fourier series.
Extending existing ﬁxed sinusoidal PEs to a learnable version with learnable frequencies gives two variants: a learnable sinusoidal APE and a learnable sinusoidal RPE .
3. U NDERSTANDING RPE S
RPEs ignore the absolute position of words and directly encode their relative distance.
The RPEs expression adheres to the translation invariance property during parameterization, since relative distance with the same offset will be embedded as the same embedding, namely, P x 1 − y
= P x 2 − y
if x 1 − y
= x 2 − y 2 .
Plus, RPEs that separately embed forward and backward relative embeddings, i.e., P i − j (cid:54) =
P j − i , do not meet symmetry during parameterization.
Sinusoidal RPEs can also embed neighboring relative position in close vectors with a local monotonicity , similarly to sinusoidal APEs.
In sinusoidal RPEs, the dimension is the same as the dimension of each head (typically 64 in BERT).
Intervals of monotonicity are slightly different from APEs which have a dimension of 768.
Note that the dot products between two sinusoidal relative position vectors with the same offset, without distinguishing positive negative relative position vectors, should be identical 5 .
This makes it hardly perceive of the border between preceding and succeeding relative position vectors.  E XAMINING PE P ROPERTIES IN PRE - TRAINED LANGUAGE MODEL
We train BERT with six basic PEs as in Tab. 1 and their combination variants, and conduct a probing test to check to which degree they satisfy the properties.
Pre-training The pre-trained “BERT-base-uncased” checkpoint (Devlin et al., 2018) is used to train by replacing the original absolute PE module with a new PE variant (including APEs and
RPEs).
We train the new models with a sequence length of 128 for 5 epochs
and then 512 for another 2 epochs.
The training is the same as in the original BERT, i.e., BooksCorpus and Wikipedia (16G
[Figure] raw documents) with whole word masking.
To be fair, the BERT with the original fully-learnable APE is also further trained in the same way.
All models have about 110M parameters corresponding to a typical base setting, with minor differences solely depending on the parameterization in Tab. 1. 4. D OT PRODUCT BETWEEN POSITION VECTORS APEs We calculate dot products between two arbitrary position vectors for APEs and RPEs (see Fig. 1).
For APEs, neighboring position vectors are generally closer compared to faraway ones. This trend is clearer in the learnable sinusoidal APE , which imposes a strict sinusoidal regularization for PEs. Note that additionally adopting RPEs does not affect too much PE patterns, as can be seen by comparing Fig. 1(a) and 1(b), or Fig.
1(c) and 1(d).
RPEs In the fully-learnable RPE setting, the vertical and horizontal bright bands in 1(e) and 1(f) show that the relative position vectors for small offsets (e.g., { P − 5 , · · · , P 0 , · · · P 5 } ) are notably different to other relative position vectors; it indicates that the relative position vectors with small offsets are more distinguishable than faraway relative position vectors.
The four dark corners in 1(e) and 1(f) means that relative position vectors with longer offset than 20, i.e., from -64 to -20 and from 20 to 64, are very close, showing that the fully-learnable RPE does not signiﬁcantly distinguish far-distant RPEs.
This suggests that truncating RPEs into a ﬁxed distance (e.g. 64 in (Shaw et al., 2018)), is reasonable.
This effect is further explained in App.
D.
I DENTICAL WORD PROBING
In APEs, the attention matrix ( A = softmax ( QK T ) ) is related to individual words and their positions, an element of (inactivated)
A in the ﬁrst layer is given by:
[Equation]
Identical word probing for PEs To study the effect of only PEs in A without considering individual words, we use identical word probing : feed many repeated identical words (can be arbitrary,
[Figure]
denoted as ¯ w ) as a sentence to BERT to check the attention values ¯ A (1) , with each element
[Equation]
As we take an average of ¯ A (1) over many randomly-selected words ¯ w , the general patterns of ¯ A (1) will not be affected by any particular word.
Namely, ¯ A (1) is word-free and only related to learned
PEs.
Thus, ¯ A (1) can be treated as a general attention bias and can also implicitly convey positionwise proximity in Transformers.
Note that the probing test could also be applied to RPEs.
4.2. Q UALITATIVE ANALYSIS
Fig.
2 shows the average attention weights among all heads in the ﬁrst layer.
BERT without PE nearly treats all words uniformly (bag-of-words).
Almost all APEs and RPEs have a clear pattern of translation invariance, local monotonicity in a neighboring window, and symmetry.
Note that this is nontrivial since no speciﬁc constraints or priors were imposed on fully-learnable APEs/RPEs 7 .
BERT with APEs does not show any direction awareness since Fig. 2(b) and 2(c) are nearly symmetrical.
As seen from Fig.
2(f,h), BERT with learnable sinusoidal RPE generally attends more on forward tokens than backward tokens, which cannot be clearly found in fully-learnable RPE and ﬁxed sinusoidal RPE .
Interestingly, the white bands along the diagonal in Fig.
2 (d, f, g) suggest that some words generally do not attend to themselves, as previously observed in (Clark et al., 2019) 8 .
4.2. Q UANTITATIVE ANALYSIS
Using the activated attention values ¯ A (1) in Eq.
8 9 we adopt three quantitative indicators to measure to which extent BERT models with individual PEs satisfy the three properties and their derivative indicators (see App.
B for details of calculating these indicators) in Tab. 2.
Basically, all APEs and
RPEs satisfy monotonicity in small offsets and translation invariance compared to BERT without
PE ; All PEs nearly satisfy symmetry except for the learnable sinusoidal RPE and its combinations.
APEs and RPEs The learnable sinusoidal APE better satisﬁes all three properties than fully learnable APE and ﬁxed sinusoidal APE; this is due to its sinusoidal parameterization and ﬂexible frequencies.
RPEs satisfy translation invariance to a higher degree than APEs, because they directly
[Table]
satisfy translation invariance during parameterization.
In the last column, direction balance values of all PEs except for the ﬁxed sin.
APE are larger than one, which indicates that BERT models with all PEs generally attend more to preceding tokens than succeeding tokens, and this phenomenon appears to be stronger in learnable sinusoidal RPEs than others.
The fully learnable APE and [CLS] Fully learnable APE generally performs worse in translation invariance (see the 4-th column) as it has to deal with the unshiftable [CLS] which is always in the ﬁrst position.
Without considering [CLS] and [SEP] (see the 5-th column), the fully learnable
APE satisﬁes translation invariance better than other APEs, showing that the fully learnable APE can ﬂexibly deal with both special tokens and normal positions.
The fully learnable APE also could handle the mismatch between special tokens and normal positions in the monotonicity property.
PE S IN DOWNSTREAM TASKS
We empirically compare the performance of PEs in classiﬁcation and span prediction tasks.
Fine-tuning The ﬁne-tuning on GLUE and SQuAD is the same as in the Huggingface website as per Wolf et al.
(2019), see App.
E for details.
We report the average values of ﬁve runs per dataset.
For classiﬁcation, we use the GLUE (Wang et al., 2018) benchmark, which includes datasets for both single document classiﬁcation and sentence pair classiﬁcation. For span prediction, we use the SQuAD V1.1 and V2.0 datasets consisting of 100k crowdsourced question/answer pairs (Rajpurkar et al., 2016). Given a question and a passage from Wikipedia containing the answer, the task is to predict the answer text span in the passage.
In V2.0, it is possible that no short answer exists in the passage since it additionally has 50,000 unanswerable questions written adversarially by crowdworkers (Rajpurkar et al., 2018).
5. E XPERIMENTAL RESULTS FOR DOWNSTREAM TASKS
GLUE Tab.
3 shows that the fully-learnable APE (a.k.a, BERT-style APE) performs well in
GLUE. No PE variants, especially BERT with solely APEs or RPEs, notably outperform the fullylearnable APE.
BRRT models with a combination of an APE and an RPE do not always boost the performance of the model with solely the APE or RPE.
SQuAD Tab. 4 shows that nearly all BERT models with RPEs signiﬁcantly outperform the fully learnable APE .
The learnable sinusoidal APE is slightly better than the fully learnable APE in most cases.
Both the best-performed models in SQuAD V1.1 and V2.0 adopt the fully-learnable RPE. [Table] [Table] As demonstrated in Tab.
[Table] the fully learnable APE can ﬂexibly deal with [CLS] and translation invariance in normal positions, thus it performs well in classiﬁcation tasks (GLUE) which heavily relies on the unshiftable [CLS] token for inference. Span prediction tasks which do not infer from [CLS] can beneﬁt from strict translation invariance during parameterization (e.g., sinusoidal APEs and RPEs), see Tab. 5 in Sec.
6.1 for the correlations between performance of SQuAD and the translation invariance property..
Removing PEs ( BERT without PE ) dramatically decreases performance in SQuAD V1.1 and V2.0, and slightly harms performance on GLUE, showing that PEs are more important in SQuAD than GLUE.
Learnable sinusoidal PEs The sinusoidal APEs outperform fully-learnable APE in span prediction but underperform it in classiﬁcation tasks.
The learnable sinusoidal APE/RPE outperforms ﬁxed sinusoidal APE/RPE in GLUE and SQuADs, showing the expressive power of ﬂexible frequencies.
Complementarity of APEs and RPEs In SQuAD, jointly adopting APEs and RPEs can slightly boost performance in some cases.
For instance, BERT with learnable sinusoidal APE + APE + fully
RPE achieves the best EM score in both SQuADs.
However, this complementary effect is relatively weaker in GLUE, where the fully-learnable APE performs strongly.
D ISCUSSIONS ON PE S 6. H OW DO THE PROPERTIES CORRELATE TO INDIVIDUAL TASKS ?
We conduct a correlation analysis between the properties and the performance on individual tasks, as shown in Tab. 5.
The results show that violating monotonicity in relatively-small offsets (e.g., 20) and translation invariance is harmful since it is negatively correlated to the performance on
[Table]
GLUE and SQuAD.
However, violating symmetry (and direction-balance) is slightly beneﬁcial. This shows that many tasks require BERT models to distinguish preceding and succeeding tokens, especially to attend more on preceding tokens.
See Fig.
5b in App.
C, the correlations between the direction balance indicators and the performance of downstream tasks will be much higher when only considering a few neighboring tokens for calculating the indicator.
6. M ORE D ISCUSSIONS ON THE PROPOSED PROPERTIES
Monotonicity Monotonicity holds locally in a small neighboring window (usually in 5-20 offsets) for all PE variants, see Fig.2.
This shows that BERT models generally are not sensitive to longerdistance attendance patterns, also evidenced by the fact that performance in downstream tasks correlates more highly with monotonicity in middle-distance offsets (e.g., 20 in the second row of Tab. 5) than longer offsets (see App. C).
To check monotonicity guided by learned frequencies of learnable sinusoidal APEs in individual tasks, see App. A.
Translation invariance In BERT, we argue that absolute positions of words are uninformative since (1) absolute positions of the second segment depend on the length of the ﬁrst sentence; (2) words are randomly truncated in the beginning or end if a sentence exceeds the expected maximum length, which may shift absolute positions of all tokens with an unexpected offset (Devlin et al., 2018).
That is, absolute positions of words in pre-trained language models are arbitrarily replaceable, and thus adopting translation invariance is generally reasonable.
Models with strict Translation invariance (all RPEs and sinusoidal APEs) naturally make PEs generalize to longer documents than the documents used in the pre-training phase, see App. F for some empirical evidence. Symmetry APEs (especially sinusoidal APEs) express symmetry patterns without distinguishing the direction as shown in Fig 2.
As seen from Eq.
7, it is nontrivial to model directions in two linearly-transformed query vectors and key vectors.
This limits its performance in direction-sensitive downstream tasks.
RPEs could behave better on direction perception, since forward and backward relative embeddings are separately embedded (see Tab. 1); Especially, learnable sinusoidal RPE or combination variants including it have more unbalanced attending patterns (see last column in Tab. 2), as shown in Fig. 2 (f) and (h),
C ONCLUSION
To theoretically and empirically understand position embeddings (PEs), we have deﬁned three properties (translation invariance, monotonicity, and symmetry) inspired by distance mappings between the original domain of positions in N and their PEs in R D .
A probing test has been proposed to quantitatively examine these properties using appropriate mathematical indicators.
Our probing test has shown that these PEs nearly satisfy most properties even when they are fully-learnable without constraints.
Experimental results have shown that violating local monotonicity and translation invariance decreases performance in downstream tasks (classiﬁcation and span prediction tasks), and that violating symmetry beneﬁts downstream tasks because of direction awareness. We also ﬁnd that the fully-learnable absolute PE in general results in better performance for classiﬁcation, and that relative PEs result in better performance for span prediction tasks, which can be explained by the connections between their properties and task characteristics.
A U NDERSTANDING FREQUENCIES A. U NDERSTAND INDIVIDUAL FREQUENCIES
We argue in this paper that a learning schema for such frequencies will be useful in a sense it could adaptively adjust frequencies to meet different functions, see Fig. [Figure]
A. E XPRESSIVE P OWER OF LEARNABLE SINUSOIDAL PE S
In Transformers, linear transformation is commonly-used, for example query , key , and value transformations on word representations.
Let r i be the word representation paramertezied by the sum of word embeddings and position embeddings (like the learnable sinusoidal APEs).
Then, each element in r i
[Equation]
After a linear transformation parameterized by w (e.g., the key transformation W K in the ﬁrst Transformer layer), r i is linearly transformed as h i ( t ) =
wr
i ( h i ( t ) can be one of query/key/value vectors Q x , K x , V x in t -th position) with each element
[Equation]
The RHS is a typical Fourier series with a base term (cid:80) w j,k e i,k and Fourier coefﬁcients { w j, 2 k , w j, 2 k +1 } .
It is customarily assumed in physics and signal processing (Arfken & Weber, 1999) that the RHS in Eq.
10 with inﬁnite D and appropriate frequencies could approximate any continuous function on a given interval.
As using an inﬁnite D is not practical, dynamic allocation of a limited number of frequencies in a data-driven way could be beneﬁcial for general approximation.
The predeﬁned frequencies ω i = [Figure] [Figure]
[Figure] [Figure] [Figure] (1 / 10000) 2 i/D in the Transformer (Vaswani et al., 2017) can be considered as a special case when it enumerates various frequencies ranging from 1 / 10000 to 1 under a speciﬁc distribution.
A. L EARNED FREQUENCIES OF L EARNABLE SINUSOIDAL APE
The learned frequencies are shown in Fig. 4a.
Observe that the learned frequencies are generally smaller than the pre-deﬁned ones from (Vaswani et al., 2017) (i.e., ω i
=
(1 / 10000) 2 i/D ).
The learned frequencies are close to the learned one since we use ω i =
(1 / 10000) 2 i/D as initialization.
As shown in Fig.
4b, the patterns of dot products between positions for learnable frequencies are quite different to the predeﬁned ones (denoted as ‘default’ in the ﬁgure); indeed, the former appears more predisposed to deeming remote positions similar.
Moreover, ﬁne-tuned models for span prediction tasks (including SQuAD and SQuAD2) satisfy strict monotonicity in larger windows than for classiﬁcation tasks.
Observe also that the patterns in pre-training language models seem more similar to those in classiﬁcation tasks than span prediction tasks.
B Q UANTITATIVELY MEASURING THE PROPERTIES .
To quantitatively measure the primary properties we treat in this paper, we propose multiple criteria, described below.
Assume a position-wise attention matrix ¯ A (1) (denoted as A since there is no risk for confusion), in which each element is the (softmax) activated attention value from the i -th query token to j -th key token (all elements are positive).
Average In-group Variance (AIV) for translation invariance Let l be an offset between two positions; we denote by τ ( l ) the set of l -offset attention values { A i,j , j − i = l } ; for example, τ (1) =
{ A 1 , 2 , A 2 , 3 , · · · , A L − 1 ,L } .
Translation invariance requires that all elements in each group τ ( l ) should be identical, the smaller variance each τ ( l ) has, it is closer to translation invariance.
The
Average In-group Variance (AIV) is deﬁned as a weighted average over in-group variances of all { τ ( l ) } L − 1 − L +1 , namely:
[Equation]
where | · | is the number of elements in the set.
For normalization, this metric is further divided by the overall variance (i.e., var ( A ) ).
Ordered Pair Ratio (OPR) for monotonicity For a word in i -th position, based on the increasing distance to the i -th position, there a forward attention sequence S i, + = { A i,i , A i,i +1 , · · · , A i,L } and a backward attention sequence S i, − =
{ A i,i , A i,i − 1 , · · · , A i, 1 } .
This results in 2 L sequences denoted as S =
{ S 1 , + , S 1 , − , S 2 , + , S 2 , − , . . .
, S L, + , S L, − } .
The ideal (decreasing) monotonicity requires that each S (an element in S ) is totally ordered as s 0 > s 1 > · · · > s L − 1 .
We deﬁne the
Ordered Ratio of S by:
[Equation]
We deﬁne sign( x )
{+= 
+}
if x > 0 , and sign( x ) = 0 otherwise.
Ideally, the OPR of a totally ordered decreasing (increasing) sequence S should be zero (one).
The expected OPR of a randomly-ordered sequence (average OPR of the set of all such sequences) should be 0.5.
Finally, we get a weighted sum of OPRs of all sequences in S .
[Equation]
In the paper, we also consider a version of monotonicity within a offset of k (e.g., ‘monotonicity (ﬁrst 20 offsets)’ in Tab. 2), which OPR is calculated in ﬁrst k elements of each S ∈ S .
Symmetrical Discrepancy for symmetry We deﬁne the Symmetrical Discrepancy (SD) by:
[Equation]
Direction Balance We deﬁne the Direction Balance (DB) as the ratio between the sum of the lower (left) triangle and the upper (right) triangle of A .
Note that all elements are positive in A ,
DB(A) in l -offset range is always positive.
[Equation]
In Tab. 5 and 2, we report DB for a offset range of 20, see Fig.
5b for the performance correlations with other offset ranges.
C M EASURING CORRELATIONS BETWEEN PROPERTIES AND DOWNSTREAM TASKS .
In Tab. 5, monotonicity in 20 offsets and the correlations with performance in downstream tasks was reported; here we show how different ranges of the monotonicity correlate to performance in downstream tasks.
Among all tasks, we choose all single sentence classiﬁcation tasks (CoLA and SST-2), two biggest sentence pair classiﬁcation tasks (MNLI and QQP tasks have more training samples than others), average performance in GLUE, and in SQuAd (F1 metrics nearly have identical trends with EM metrics).
As shown in Fig.
5a, the monotonicity indicators in nearly 20-55 offsets are highest correlated to the performance of span prediction tasks (with Pearson correlation larger than 90% ).
Note that some classiﬁcation tasks (especially SST-2) also show opposite correlations comparing to span prediction tasks, probably due to the unshiftable [CLS] on which classiﬁcation tasks rely for interference does not need monotonicity.
In Fig.
5b, the performance correlates more to the direction balance indicators when considering neighboring tokens.
For instance, the direction balance indicators within a small offset has correlation bigger than 0.5, this tends to be smaller with increasing offsets.
D R ELATIVE POSITION EMBEDDING WITH LONG OFFSETS
The dot product between two position embeddings are shown in Fig.
1(e) and (f).
To analyze the behaviour, we replace the raw dot product with the cosine similarity (as the latter is normalized and
[Figure] [Figure] thus easier to interpret). When the cosine similarity is one, the two vectors are perfectly colinear and share the same direction.
For the purposes of this investigation, we arbitrarily pick 0 . 95 as a threshold for the cosine similarity, denote that two vectors are not signiﬁcantly different. From Fig. 6 we observe the following for all PE variants with fully-learnable RPE: (1) There is no signiﬁcant difference between relative position vectors with longer than 20-25 offsets; (2) forward relative position vectors are slightly more similar to forward relative position vectors instead of backward relative position vectors, and vice versa (see the central left-lower/right-upper white parts).
We train BERT base and BERT medium with both masked language prediction and next sentence prediction tasks; most parameters are listed in Tab. 6, with the remaining parameters set as in the original paper.
Note that we share RPE in different heads and layers.
Like (Shaw et al., 2018) RPE are truncated from − 64 to 64 . [Table]
We perform ﬁve runs for SQuAD and GLUE benchmark.
The results in GLUE are for the last checkpoint during ﬁne-tuning while SQuAD takes the best one for every 1000 steps.
Finally, we calculate the average over 5 runs.
All these settings are the same for all PEs.
We use Mismatched MNLI.
In GLUE (Wang et al., 2018), the train and dev are somewhat adversarial: training samples (in train and dev) containing the same sentence usually have opposite labels.
Models may get worse when it overﬁts in the train set, resulting in unexpected results.
Therefore, we exclude WNLI to calculate average in the last column in Tab. 3.
The ﬁne-tuning parameters are using default values in Huggingface project Wolf et al. F GENERALIZATION TO LONGER SENTENCES IN DOWNSTREAM TASKS
To fairly compare all models, we train a medium setting (8-layer transformer) on 128-length input in the ﬁrst 10 epochs and 512-length input in the last 2 epochs from scratch. Fig.
7 shows that before 512-length pre-trained (like the 10-th epoch 128-length pre-trained) learnable sinusoidal APEs and RPEs perform better than BERT-style (without sinusoidal parameterization) in both SQuADs. This happens because PEs with translation invariance ( learnable sinusoidal APEs and RPEs) generalize into longer positions 12 , while position vectors between 128-512 positions are not trained in fullylearnable PEs and they are randomly initialized and ﬁnetuned in the downstream.
[Table] [Figure]
G T HE EVOLUTION OF DOT PRODUCTS BETWEEN POSITION VECTORS
We exhibit dot products between position vectors during training a BERT-medium, as shown in Fig.
There is seemingly no pattern in the beginning, but as the number of training steps increase, a regular pattern with translation invariance and local monotonicity emerges.
H D ISCUSSIONS ON RELATED WORKS
Complementary effect between APE and RPE The complementary effect between APE and
RPE was demonstrated to be effective in (Wang et al., 2019) for machine translation. In the pretrained language model, Ke et al. (2020) propose that combining APE and RPE could be beneﬁcial for classiﬁcation tasks (GLUE), which in this paper, this complementary effect is not signiﬁcant since most PE combinations (APE and RPE) do not outperform the BERT-style fully-learnable APE on classiﬁcation.
Instead, we empirically conclude that most PE combinations boost the performance in span prediction tasks.
The beneﬁt in classiﬁcation tasks in (Ke et al., 2020) may come from other modiﬁcations, for example, it unties the [CLS] symbol from other positions.
Moreover, in the paper, it adopts a special relative position embedding like (Raffel et al., 2019) (as this paper also suggests to do so): a simpliﬁed form of PE that each “embedding” is simply a scalar bias added to the corresponding logits when computing the attention weights.
The fundamental difference between the ‘position bias’ and position embedding is unknown from now.
Study on attention visualization.
Many works are focusing on understanding attention patterns in individual heads.
For example Vig (2019) introduced a tool for visualizing attention in the Transformer at multiple scales; Rogers et al.
(2020) suggest attention mechanisms like Vertical, Diagonal,
Vertical + diagonal, Block, and Heterogeneous.
Clark et al.
(2019) found some attention mechanisms like attending broadly, to next, to [CLS] or [SEP] , attend to punctuation.
While our paper focuses on the general attention introduced by PEs from an average point of view, without considering any speciﬁc attention head.
Asymmetry in sequential labeling Yan et al.
(2019) suggested asymmetry of position embedding in named-entity recognition task (without involving pre-trained language models) which is a kind of sequential labeling tasks like span prediction (SQuAD) in this paper.
Their conclusion is generally compatible with ours, but we question its assumption that ‘the property of distance-awareness disappears when query and key projection are conducted’.
As shown in Fig.
9, we could slightly see some distance-awareness by directly taking the average position-position correspondence in the ﬁrst layer among many heads (i.e., PW Q, 1 ( W K, 1 ) T P T ).
Functional parameterization of PEs Xu et al.
(2019) proposes various variants of sinusoidal positional encodings inspired by functional analysis.
Wang et al.
(2020) proposed a sinusoid-like complex word embedding to encode word order.
Both (Xu et al., 2019) and (Wang et al., 2020) assume that PEs should satisfy the translation invariance property, but they induce different types of sinusoidal PE parameterization either in real or complex vector space.
Moreover, Liu et al.
(2020) use a neural ODE component to parameterize position encoding as a continuous dynamical model, which could learn suitable PEs in neural networks.
All of these PEs are inspiring.
Since selecting [Figure] the suitable parameterization type is not the main concern in this paper, we adopted the typical ones, namely, the fully-learnable, (learnable or ﬁxed) sinusoidal APEs/RPEs. The fundamental difference between these PE parameterizations needs further investigation.
More recently, (Wang &
Chen, 2020) empirically study the behaviour of many position embeddings and their performance in Transformers for various NLP tasks.
I T HE THREE PROPERTIES IN OTHER MODELS
By using the proposed identical probing test, we also check the properties of other trained Transformer models with decoder components in Tab. 7 and Fig.
The machine translation modelis a typical encoder-decoder architecture using multiple-layer Transformers.
GPT2 (Radford et al., 2019) adopts a purely decoder architecture; 12-layer base setting is used in this work.
Monotonicity Compared to BERT and the machine translation, GPT2 satisﬁes monotonicity (especially in the ﬁrst 20 offsets) better than other models, showing capturing distance between neighboring tokens matters in the language model.
Translation invariance As seen from the translation invariance indicators in Tab. 7, GPT2 satisﬁes translation invariance poorer than other models, since tokens in it also additionally attend to a few beginning tokens no matter how far the attended tokens are.
Symmetry GPT2 shows the biggest symmetrical discrepancy, since GPT2, which aims to predict the next word, adopts an attention mask of succeeding tokens to avoid information leakage.
Plus, the machine translation encoder slightly attends more to the succeeding tokens while BERT attends more on the preceding tokens than succeeding tokens.
[Table]
J W HITE BAND EFFECTS ALONG THE DIAGONAL
In order to analyze the white band effects along the diagonal, we show all results of identical word probing (average attention values in the ﬁrst layer of identical word probing with respect to
[Figure]
randomly-selected words).
This effect is more clear for fully-learnable RPE, learnable sinusoidal
RPE and any combination variants including them (see.
Fig.
11 (d,f,g,i,j,l)).
To show the obvious differences between these PEs, in this paper, we use average unnormalized attention weights matrix for probing, but all indicators are calculated using normalized attention values for better quantitative comparison.
[Figure]
generally attend more