R ETHINKING P OSITIONAL E NCODING IN L ANGUAGE P RE - TRAINING
A BSTRACT
In this work, we investigate the positional encoding methods used in language pretraining (e.g., BERT) and identify several problems in the existing formulations.
First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources.
It may bring unnecessary randomness in the attention and further limit the expressiveness of the model.
Second, we question whether treating the position of the symbol [CLS] the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks.
Motivated from above analysis, we propose a new positional encoding method called T ransformer with U ntied
P ositional E ncoding (TUPE).
In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together.
This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices.
Furthermore, TUPE unties the [CLS] symbol from other positions, making it easier to capture information from all positions.
Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method.
Codes and models are released at https://github.com/guolinke/TUPE .
I NTRODUCTION
The Transformer model (Vaswani et al., 2017) is the most widely used architecture in language representation learning (Liu et al., 2019;
Devlin et al., 2018;
Radford et al., 2019; Bao et al., 2020).
In Transformer, positional encoding is an essential component since other main components of the model are entirely invariant to sequence order.
The original Transformer uses the absolute positional encoding, which provides each position an embedding vector.
The positional embedding is added to the word embedding, which is found signiﬁcantly helpful at learning the contextual representations of words at different positions.
Besides using the absolute positional encoding, Shaw et al.
Raffel et al.
(2019) further propose the relative positional encoding, which incorporates some carefully designed bias term inside the self-attention module to encode the distance between any two positions.
In this work, we revisit and study the formulation of the widely used absolute/relative positional encoding.
First, we question the rationality of adding the word embedding with the absolute positional embedding in the input layer.
Since the two kinds of embeddings are apparently heterogeneous, this addition operation brings mixed correlations 1 between the positional information and word semantics.
For example, by expanding the dot-production function of keys and values in the self-attention module of the ﬁrst layer, we ﬁnd that there are explicit terms that use “word” to query “positions” and vice versa.
However, words may only have weak correlations to where they appear in the sentence.
Our empirical analysis also supports this by showing that in a well-trained model, such correlation is noisy.
Second, we notice that the BERT model does not only handle natural language words.
A special symbol [CLS] is usually attached to the sentence.
It is widely acknowledged that this symbol
[Figure]
Add & Layer Norm
FFN
FFN
Add & Layer Norm … …
𝑤 𝑗 𝑤 𝑖
Untie [CLS] 𝑏 0 𝑏 1 𝑏 2 𝑏𝑏 −1 𝑏 0 𝑏 1 𝑏𝑏 −2 𝑏 −1 𝑏 0 𝑏𝑏 −3 𝑏 −2 𝑏 −1 𝑏
Relative positional correlation 𝑏 𝑗−𝑖 𝑝
𝑗 𝑈 𝐾 𝑝 𝑖 𝑈 𝑄 ×
Self-Attention
Add & Layer Norm
FFN FFN
Add & Layer Norm
[Figure]
receives and summarizes useful information from all the positions, and the contextual representation of [CLS] will be used as the representation of the sentence in the downstream tasks.
As the role of the [CLS] symbol is different from regular words that naturally contain semantics, we argue that it will be ineffective if we treat its position the same as word positions in the sentence.
For example, if we apply the relative positional encoding to this symbol, the attention distribution of some heads will likely be biased to the ﬁrst several words, which hurts the understanding of the whole sentence.
Based on the investigation above, we propose several simple, yet effective modiﬁcations to the current methods, which lead to a new positional encoding called T ransformer with U ntied P ositional
E ncoding (TUPE) for language pre-training, see Figure 1.
In TUPE, the Transformer only uses the word embedding as input.
In the self-attention module, different types of correlations are separately computed to reﬂect different aspects of information, including word contextual correlation and absolute (and relative) positional correlation.
Each kind of correlation has its own parameters and will be added together to generate the attention distribution.
A specialized positional correlation is further set to the [CLS] symbol, aiming to capture the global representation of the sentence correctly.
First, we can see that in TUPE, the positional correlation and word contextual correlation are de-coupled and computed using different parameters.
This design successfully removes the randomness in word-to-position (or position-to-word) correlations and gives more expressiveness to characterize the relationship between a pair of words or positions.
Second, TUPE uses a different function to compute the correlations between the [CLS] symbol and other positions.
This ﬂexibility can help the model learn an accurate representation of the whole sentence.
We provide an efﬁcient implementation of TUPE.
To validate the method, we conduct extensive experiments and ablation studies on the GLUE benchmark dataset.
Empirical results conﬁrm that our proposed TUPE consistently improves the model performance on almost all tasks.
In particular, we observe that by imposing this inductive bias to encode the positional information, the model can be trained more effectively, and the training time of the pre-training stage can be largely improved.
P RELIMINARY 2. A TTENTION MODULE
The attention module (Vaswani et al., 2017) is formulated as querying a dictionary with key-value pairs, e.g., Attention ( Q, K, V ) =
softmax ( QK T √ d ) V , where d is the dimensionality of the hidden representations, and Q (Query), K (Key), V (Value) are speciﬁed as the hidden representations of the previous layer.
The multi-head variant of the attention module is popularly used which allows the model to jointly attend to the information from different representation sub-spaces, and is deﬁned as Multi-
[Equation]
[Equation]
where W Qk ∈ R d × d K , W Kk ∈ R d × d K ,
W Vk ∈ R d × d V , and W O ∈ R Hd V
× d are learnable project matrices, H is the number of heads.
d K and d V are the dimensionalities of Key and Value.
The self-attention module is one of the key components in Transformer and BERT encoder (Devlin et al., 2018).
For simplicity, we use the single-head self-attention module and set d K = d V = d for a demonstration.
We denote x l = ( x l 1 , x l 2 · · · , x ln ) as the input to the self-attention module in the l -th layer, where n is the length of the sequence and
each vector x li ∈ R
d is the contextual representation of the token at position i .
z l = ( z l 1 , z l 2 · · · , z ln ) is the output of the attention module.
Then the self-attention module can be written as
[Equation]
, where
[Equation]
As we can see, the self-attention module does not make use of the order of the sequence, i.e., is permutation-invariant.
However, natural language is well-structured and word order is important for language understanding (Sutskever et al., 2014).
In the next section, we show several previous works that proposed different ways of incorporating positional information into the Transformer model.
2. P OSITIONAL E NCODING
Generally, there are two categories of methods that encode positional information in the Transformer model, absolute positional encoding and relative positional encoding.
Absolute Positional Encoding.
The original Transformer (Vaswani et al., 2017) proposes to use absolute positional encoding to represent positions.
In particular, a (learnable) real-valued vector p i ∈
R
d is assigned to each position i .
Given a sentence, p i will be added to the word embedding w i at position i , and w i + p i will be used as the input to the model, e.g, x 1 i =
w i
+ p i .
In such a way, the Transformer can differentiate the words coming from different positions and assign each token position-dependent attention.
For example, in the self-attention module in the ﬁrst layer, we have
[Equation]
Relative Positional Encoding.
In absolute positional encoding, using different p i for different position i helps Transformer distinguish words at different positions.
However, as pointed out in
Shaw et al.
, the absolute positional encoding is not effective for the model to capture the relative word orders.
Therefore, besides using absolute positional encoding, Shaw et al. proposes a relative positional encoding as an inductive bias to help the learning of attention modules,
[Equation]
(4) where a lj −
i ∈
R
d is learnable parameter and can be viewed as the embedding of the relative position j − i in layer l .
In this way, embedding a lj − i explicitly models the relative word orders.
T5 (Raffel et al., 2019) further simpliﬁes it by eliminating a lj − 1 in Query-Key product.
[Equation]
For each j − i , b j −
i is a learnable scalar 2 and shared in all layers
T RANSFORMER WITH U NTIED P OSITIONAL E NCODING 3. U NTIE THE C ORRELATIONS BETWEEN P OSITIONS AND W ORDS
In absolute positional encoding, the positional embedding is added together with the word embedding to serves as the input to the neural networks.
However, these two kinds of information are heterogeneous.
The word embedding encodes the semantic meanings of words and word analogy tasks can be solved using simple linear arithmetic on word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Joulin et al., 2016).
On the other hand, the absolute positional embedding encodes the indices in a sequence, which is not semantic and far different from word meanings.
We question the rationality of the linear arithmetic between the word embedding and the positional embedding.
To check clearly, we take a look at the expansion of Eq.
[Equation]
The above expansion shows how the word embedding and the positional embedding are projected and queried in the attention module.
We can see that there are four terms after the expansion: word-to-word, word-to-position, position-to-word, and position-to-position correlations.
We have several concerns regarding this formulation.
First, it is easy to see that the ﬁrst and the last term characterize the word-word and position-position relationships respectively.
However, the projection matrices W Q,l and W K,l are shared in both terms.
As the positional embedding and the word embedding encode signiﬁcantly different concepts, it is not reasonable to apply the same projection to such different information.
Furthermore, we also notice that the second and the third term use the position (word) as the query to get keys composed of words (positions).
As far as we know, there is little evidence suggesting that the word and its location in a sentence have a strong correlation.
Furthermore, in BERT and recently developed advanced methods such as RoBERTa (Liu et al., 2010), sentences are patched in a random way.
For example, in BERT, each input contains multiple sentences and some of the sentences are negatively sampled from other documents to form the next sentence prediction task.
Due to the random process of batching, it is possible that a word can even appear at any positions and the correlations between words and positions could be weak.
To further investigate this, we visualize the four correlations in Eq.
(6) on a pre-trained BERT model.
We ﬁnd that the second and the third term looks uniform across positions, as shown in Figure 2.
This phenomenon suggests that there are no strong correlations 3 between the word and the absolute position and using such noisy correlation may be inefﬁcient for model training.
[Figure]
Our modiﬁcation.
To overcome these problems, we propose to directly model the relationships between a pair of words or positions by using different projection matrices and remove the two terms in the middle.
That is, we use
[Equation]
where U Q , U K ∈ R d × d are the projection matrice for the positional embedding, and scaling term√ 2 d is used to retain the magnitude of α ij (Vaswani et al., 2017) .
A visualization is put in Figure 3.
Our proposed method can be well combined with the relative positional encoding in Raffel et al.
(2019) by simply changing Eq.
(5) to
[Equation]
3. U NTIE THE [CLS] S YMBOL FROM P OSITIONS
Note that in language representation learning, the input sequence to the Transformer model is not always a natural sentence.
In BERT, a special symbol [CLS] is attached to the beginning of the input sentence.
This symbol is designed to capture the global information of the whole sentence.
Its contextual representation will be used to make predictions in the sentence-level downstream tasks after pre-training (Devlin et al., 2018;
Liu et al., 2019).
We argue that there will be some disadvantages if we treat this token the same as other natural words in the attention module.
For example, the regular words usually have strong local dependencies in the sentence.
Many visualizations (Clark et al., 2019a; Gong et al., 2019) show that the attention distributions of some heads concentrate locally.
If we process the position of [CLS] the same as the position of natural language words, according to the aforementioned local concentration phenomenon, [CLS] will be likely biased to focus on the ﬁrst several words instead of the whole sentence.
It will potentially hurt the performance of the downstream tasks.
Our modiﬁcation.
We give a speciﬁc design in the attention module to untie the [CLS] symbol from other positions.
In particular, we reset the positional correlations related to [CLS] .
For better demonstration, we denote v ij as the content-free (position-only) correlation between position i and j .
For example, when using the absolute positional encoding in Eq.
(7), v ij = 1 √ 2 d ( p i U Q )( p j U K ) T ; when using relative positional encoding in Eq.
(8), v ij = 1 √ 2 d ( p i U Q )( p j U K ) T + b j − i .
We reset the values of v ij by the following equation:
[Equation]
[Equation]
others
[Equation]
where θ = { θ 1 , θ 2 } is a learnable parameter.
A visualization is put in Figure 4.
𝑣 11 𝑣 12 𝑣 13 𝑣𝑣 21 𝑣 22 𝑣 23 𝑣𝑣 31 𝑣 32 𝑣 33 𝑣𝑣 41 𝑣 42 𝑣 43 𝑣𝜃𝜃𝜃𝜃𝜃𝑣 22 𝑣 23 𝑣𝜃𝑣 32 𝑣 33 𝑣𝜃𝑣 42 𝑣 43 𝑣
Untie [CLS]
3. I MPLEMENTATION D ETAILS AND D ISCUSSIONS
In the two subsections above, we propose several modiﬁcations to untie the correlations between positions and words (Eq.
(7) and Eq.
(8)), and
untie the [CLS] symbol from other positions (Eq.
By combining them, we obtain a new positional encoding method and call it TUPE (Transformer with Untied Positional Encoding).
There are two versions of TUPE.
The ﬁrst version is to use the untied absolute positional encoding with the untied [CLS] symbol (Eq.
(7) + Eq.
(9)), and the second version is to use an additional relative positional encoding (Eq.
(8) + Eq.
We call them TUPE-A and TUPE-R respectively and list the mathematical formulations as below.
[Equation]
The multi-head version, parameter sharing, and efﬁciency.
TUPE can be easily extended to the multi-head version.
In our implementation, the absolute positional embedding p i for position i is shared across different heads, while for each head, the projection matrices U Q and U K are different.
For the relative positional encoding, b j − i is different for different heads.
The reset parameter θ is also not shared across heads.
For efﬁciency, we share the (multi-head) projection matrices U Q and U K in different layers.
Therefore, in TUPE, the number of total parameters does not change much.
Taking
BERT-Base as an example, we introduce about 1.18M ( 2 × 768 × 768 ) new parameters, which is only about 1% of the 110M parameters in BERT-Base.
Besides, TUPE introduces little additional computational costs.
As the positional correlation term 1 √ 2 d ( p i U Q )( p j U K ) T is shared in all layers, we only need to compute it in the ﬁrst layer, and reuse its outputs in other layers.
Are absolute/relative positional encoding redundant to each other?
One may think that both the absolute/relative positional encoding in Eq.
(11) describe the content-free correlation, and thus one of them is redundant.
To formally study this, we denote B as an n × n matrix where each element B i,j = b j − i .
By deﬁnition, B is a Toeplitz matrix (Gray, 2006).
We also denote P as an n × n matrix where the i -th row is p i , and thus the absolute positional correlation in matrix form is√ 2 d ( PU Q )( PU K ) T .
We study the expressiveness of B and 1 √ 2 d ( PU Q )( PU K ) T by ﬁrst showing
B can be factorized similarly from the following proposition.
Proposition 1.
Any Toeplitz matrix B ∈ C n × n can be factorized into B =
GDG ∗ , where D is a 2 n × 2 n diagonal matrix.
G is a n × 2 n Vandermonde matrix in the complex space, where each element G j,k = 12 n e iπ ( j +1) k/n and G ∗ is the conjugate transpose of G .
The proof can be found in Appendix A.
The two terms B and 1 √ 2 d ( PU Q )( PU K ) T actually form different subspaces in R n × n .
In the multi-head version, the shape of the matrix U Q and U k are d × dH .
Therefore, ( PU Q )( PU K ) T can characterize low-rank matrices in R n × n .
But from the proposition, we can see that B forms a linear subspace in R n × n with only 2 n − 1 freedoms, which is quite different from the space of 1 √ 2 d ( PU Q )( PU K ) T .
There are also some practical reasons which make using both terms together essential.
As discussed previously, in Raffel et al.
(2019), the range of the relative distance j − i will be clipped up to an offset beyond which all relative positions will be assigned the same value.
In such a situation, the relative positional encoding may not be able to differentiate words faraway and 1 √ 2 d ( p i U Q )( p j U K ) T can be used to encode complementary information.
[Table]
E XPERIMENT
To verify the performance of the proposed TUPE
[Table]
we conduct extensive experiments and demonstrate the results in this section.
In the main body of the paper, we study TUPE under the BERT-Base setting (Devlin et al., 2018).
We provide all the experimental details and more results on applying TUPE under the BERT-Large setting and the ELECTRA setting (Clark et al., 2019b) in Appendix B and C.
E XPERIMENTAL D ESIGN
We use BERT-Base (110M parameters) architecture for all experiments.
Speciﬁcally, BERT-Base is consist of 12 Transformer layers.
For each layer, the hidden size is set to 768 and the number of attention head is set to 12.
To compare with TUPE-A and TUPE-R, we set up two baselines correspondingly: BERT-A, which is the standard BERT-Base with absolute positional encoding (Devlin et al., 2018); BERT-R, which uses both absolute positional encoding and relative positional encoding (Raffel et al., 2019)
(Eq.
Following Devlin et al.
(2018), we use the English Wikipedia corpus and BookCorpus (Zhu et al., 2015) for pre-training.
By concatenating these two datasets, we obtain a corpus with roughly 16GB in size.
We set the vocabulary size (sub-word tokens) as 32,768.
We use the GLUE ( G eneral
L anguage U nderstanding E valuation) dataset (Wang et al., 2018) as the downstream tasks to evaluate the performance of the pre-trained models.
All codes are implemented based on fairseq (Ott et al., 2019) in PyTorch (Paszke et al., 2017).
All models are run on 16 NVIDIA Tesla V100 GPUs with mixed-precision (Micikevicius et al., 2017).
4. O VERALL C OMPARISON
[Figure]
The overall comparison results are shown in Table 1.
Firstly, it is easy to ﬁnd that both TUPE-A and
TUPE-R outperform baselines signiﬁcantly.
In particular, TUPE-R outperforms the best baseline
BERT-R by 1.38 points in terms of GLUE average score and is consistently better on almost all tasks, especially on MNLI-m/mm, CoLA and MRPC.
We can also see that TUPE-R outperforms TUPE-A by 0.57 points.
As discussed in Sec.3.3, although using the absolute/relative positional encoding together seems to be redundant, they capture complement information to each other.
Besides the ﬁnal performance, we also examine the efﬁciency of different methods.
As shown in
Figure 5a, TUPE-A (TUPE-R) achieves smaller validation loss than the baselines during pre-training.
As shown in Table 1 and Figure 5c, TUPE-A (TUPE-R) can even achieve a better GLUE average score than the baselines while only using 30% pre-training steps.
Similar improvements can be found in BERT-Large and ELECTRA settings in Table.
3 and Table.
4 (in Appendix C).
Since the correlations between words and positions are removed in TUPE, we can easily visualize the attention patterns over positions, without considering the variability of input sentences.
In TUPE-A (see Figure 6), we ﬁnd there are mainly ﬁve patterns (from 12 heads): (1) attending globally; (2) attending locally; (3) attending broadly; (4) attending to the previous positions; (5) attending to the next positions.
Interestingly, the model can automatically extract these patterns from random initialization.
As there are some attention patterns indeed have strong local dependencies, our proposed method to untie [CLS] is necessary.
Similar patterns could be found in TUPE-R as well.
To summarize, the comparisons show the effectiveness and efﬁciency of the proposed TUPE.
As the only difference between TUPE and baselines is the positional encoding, these results indicate
TUPE can better utilize the positional information in sequence.
In the following subsection, we will examine each modiﬁcation in TUPE to check whether it is useful.
4. A BLATION S TUDY
Untie the [CLS] symbol from other positions.
To study the improvement brought by untying [CLS] , we evaluate a positional encoding method which removes the reset function in Eq.
We call it TUPE-A tie-cls and train this model using the same conﬁguration.
We also list the performance of TUPE-A tie-cls in Table 1.
From the table, we can see that TUPE-A works consistently better than
TUPE-A tie-cls , especially for low-resource tasks, such as CoLA and RTE.
Untie the correlations between positions and words.
Firstly, from Table 1, it is easy to ﬁnd that TUPE-A tie-cls outperforms BERT-A.
Since the only difference between TUPE-A tie-cls and BERT-A is the way of dealing with the absolution positional encoding, we can get a conclusion that untying the correlations between positions and words helps the model training.
To further investigate this, we design another encoding method, BERT-A d , which is based on BERT-A and uses different projection matrices for words and positions.
Formally, α ij =√
[Equation]
in
BERT-A d .
Therefore, we can check whether using different projection matrices (BERT-A vs.
BRET-A d ) can improve the model and whether removing word-to-position and position-to-word correlations (BRET-A d vs. TUPE-A tie-cls ) hurts the ﬁnal performance.
From the summarized results in Table 1, we ﬁnd that TUPE-A tie-cls is even slightly better (0.17) than BERT-A d and is more computationally efﬁcient 4 .
BERT-A is the worst one.
These results indicate that using different projection matrices improves the model, and removing correlations between words and positions does not affect the performance.
Summary.
From the above analysis, we ﬁnd that untying [CLS] helps a great deal for the lowresource tasks, such as CoLA and RTE.
Untying the positional correlation and word correlation helps high-resource tasks, like MNLI-m/-mm.
By combining them, TUPE can consistently perform better on all GLUE tasks.
We also have several failed attempts regarding modifying the positional encoding strategies, see Appendix D.
R ELATED W ORK
As introduced in Sec.2, Shaw et al.
(2018) was the ﬁrst work to leverage relative positional encoding to
Transformer.
Most of the other works are based on Shaw et al.
For example, Transformer-XL (Dai et al., 2019) re-parameterize the self-attention to integrate relative positional encoding directly.
T5 (Raffel et al., 2019) simpliﬁed the vector representation of relative positions in Shaw et al.
(2018) to a scalar.
He et al.
(2020) extended Shaw et al.
by adding the position-to-word correlation for relative position.
In (Kitaev & Klein, 2018), the authors show that separating positional and content information in the Transformer encoder can lead to an improved constituency parser. We mathematically show that such disentanglement also improves Transformer in general language pre-training.
There are some other parallel works to enhance the absolute positional encoding in Transformer, but not directly related to our work.
For example, Shiv & Quirk (2019) extended the sequence positional encoding to tree-based positional encoding in Transformer; Wang et al.
(2019) extended positional encoding to complex-valued domain; Liu et al.
(2020) modeled the positional encoding by dynamical systems.
C ONCLUSION
We propose TUPE (Transformer with Untied Positional Encoding), which improves existing methods by two folds: untying the correlations between words and positions, and untying [CLS] from sequence positions.
Speciﬁcally, we ﬁrst remove the absolute positional encoding from the input of the Transformer and compute the positional correlation and word correlation separately with different projection matrices in the self-attention module.
Then, we untie [CLS] by resetting the positional correlations related to [CLS] .
Extensive experiments demonstrate that TUPE achieves much better performance on GLUE benchmark.
Furthermore, with a better inductive bias over the positional information, TUPE can even outperform the baselines while only using 30% pre-training computational costs.
A P ROOF OF PROPOSITION
Proof.
We ﬁrst construct a circulant matrix ˆ B of shape 2 n × 2 n using B .
Note that B is a Toeplitz matrix consisting of 2 n − 1 values { b − ( n − 1) , b − ( n − 2) , · · · , b 0 , · · · , b ( n − 2) , b ( n − 1) } .
For ease of reference, we further deﬁne b − n = b n = b 0 .
Then ˆ B is constructed as
[Equation]
To avoid any confusion, we use k and j in the subscription for positions and use i as the imaginary unit.
It is easy to check that ˆ B is a circulant matrix by showing that ˆ B j,k
= ˆ B j +1 ,k +1 and ˆ B j, 2 n −
= ˆ B j +1 , 0 for any 0 ≤ j, k < 2 n − 1 .
Using Theorem 7 in Gray (2006), the matrix ˆ B can be factorized into ˆ B = QDQ ∗ .
Q is an 2 n × 2 n Vandermonde matrix, where Q k,j = 12 n e iπ ( j +1) k/n and D is a 2 n × 2 n diagonal matrix.
Since the top-left n × n submatrix of ˆ B is B , we can rewrite ˆ B = (cid:20) B
C 1 C
C(cid:21) .
We also rewrite Q = (cid:20) GC (cid:21) , where G is deﬁned in the theorem.
Then we have
QDQ ∗ = (cid:20) GC (cid:21)
(cid:20) D
D(cid:21)
[ G ∗
C ∗ 4 ] .
Considering that D is a diagonal matrix, we can obtain
B = GDG
B E XPERIMENTAL D ETAILS
Pre-training.
Following BERT (Devlin et al., 2018), we use the English Wikipedia corpus and
BookCorpus (Zhu et al., 2015) for pre-training.
By concatenating these two datasets, we obtain a corpus with roughly 16GB in size.
We follow a couple of consecutive pre-processing steps: segmenting documents into sentences by Spacy 5 , normalizing, lower-casing, and tokenizing the texts by Moses decoder (Koehn et al., 2007), and ﬁnally, applying byte pair encoding (BPE) (Sennrich et al., 2015) with setting the vocabulary size as 32,768.
We found the data cleaning is quite important for language pre-training.
Specially, we de-duplicate the documents, normalize the punctuations, concatenate the short sequences, replace the URL and other hyperlinks to special tokens, and ﬁlter the low-frequency tokens.
Therefore, our re-implemented baselines, like BERT, and achieve higher GLUE scores than the original papers.
We use masked language modeling as the objective of pre-training.
We remove the next sentence prediction task and use FULL-SENTENCES mode to pack sentences as suggested in RoBERTa (Liu et al., 2019).
We train the models for 1000 k steps where the batch size is 256 and the maximum sequence length is 512.
The masked probability is set to 0.15, with replacing 80% of the masked positions by [MASK] , 10% by randomly sampled words, and keep the remaining 10% unchanged.
We use Adam (Kingma & Ba, 2014) as the optimizer, and set the its hyperparameter (cid:15) to 1e-6 and ( β 1 , β 2) to (0.9, 0.999).
The peak learning rate is set to 1e-4 with a 10 k -step warm-up stage.
After the warm-up stage, the learning rate decays linearly to zero.
We set the dropout probability to 0.1, gradient clip norm to 1.0, and weight decay to 0.01.
Besides the ﬁnal checkpoint, we also save intermediate checkpoints ({100 k , 300 k , 600 k } steps) and ﬁne-tune them on downstream tasks, to check the efﬁciency of different methods.
Fine-tuning.
We use the GLUE ( G eneral L anguage U nderstanding E valuation) dataset (Wang et al., 2018) as the downstream tasks to evaluate the performance of the pre-trained models.
Particularly, we use nine tasks in GLUE, including CoLA, RTE, MRPC, STS, SST, QNLI, QQP, and MNLI-m/mm.
For the evaluation metrics, we report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks.
We use the same optimizer (Adam) with the same hyperparameters as in pre-training.
Following previous works, we search the learning rates during the ﬁne-tuning for each downstream task.
The setting details are listed in Table 2.
For a fair comparison, we do not apply any tricks for ﬁne-tuning.
Each conﬁguration will be run ﬁve times with different random seeds, and the median of these ﬁve results on the development set will be used as the performance of one conﬁguration.
We will ultimately report the best number over all conﬁgurations.
[Table]
[Table]
Normalization and rescaling.
Layer normalization (Ba et al., 2016; Xiong et al., 2020) is a key component in Transformer.
In TUPE, we also apply layer normalization on p i whenever it is used.
The 1 √ d in Eq.
(2) is used in the Transformer to rescale the dot product outputs into a standard range.
In a similarly way, we use 1 √ 2 d in the Eq.
(7) to both terms to keep the scale after the summation.
Furthermore, in order to directly obtain similar scales for every term, we parameterize θ 1 and θby using θ 1 = 1 √ 2 d ( p θ 1 U Q )( p θ 1 U K ) T and θ 2 = 1 √ 2 d ( p θ 2 U Q )( p θ 2 U K ) T , where p θ 1 , p θ 2 ∈ R d are learnable vectors.
C M ORE RESULTS IN THE BERT-L ARGE /ELECTRA-B ASE SETTINGS
In the main body of the paper, we provide empirical studies on TUPE under the BERT-Base setting.
Since TUPE only modiﬁes the positional encoding, one expects it to improve all language pre-training methods and deeper models.
To demonstrate this, we integrate TUPE-R (with relative position) in BERT-Large and ELECTRA-Base models, and conduct experiments for comparison.
For the
BERT-Large setting, we directly apply TUPE-R to the 24-layer Transformer model and obtain the
TUPE-Large model.
For ELECTRA-Base, we apply our positional encoding to both generator and discriminator and obtain the ELECTRA-TUPE model.
We use the same data introduced previously and use the same hyper-parameters as in Liu et al.
; Clark et al.
(2019b).
The experimental results are listed in the Table.
3 and Table.
4 respectively.
From the results, we can that TUPE is better than the corresponding baseline methods.
These additional experiments further demonstrate that TUPE is a better positional encoding solution.
D F AILED A TTEMPTS
We tried to replace the parametric form of the positional correlation ( 1 √ 2 d ( p i U Q )( p j U K ) T ) to the non-parametric form.
However, empirically we found that the training of this setting converges much slower than the baselines.
We also tried to parameterize relative position bias b j − i by ( r j − i F Q )( r j − i F K ) T .
But the improvement
[Table]