Pre-trained language models have achieved state-of-the-art results in various natural language processing tasks. Most of them are based on the Transformer architecture , which distinguishes tokens with the token position index of the input sequence. However, sentence index and paragraph index are also important to indicate the token position in a document. We hypothesize that better contextual representations can be generated from the text encoder with richer positional information. To verify this, we propose a segment-aware BERT , by replacing the token position embedding of Transformer with a combination of paragraphindex, sentenceindex ,  and token index embeddings. We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks  .
Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture . However, it distinguishes sequential tokens only with the token position index . We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer (Segatron) , by replacing the original token position encoding with a combined position encoding of paragraph , sentence, and token . We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task with Segatron . Experimental results show that BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks , and outperforms RoBERTa on zero-shot sentence representation learning .
R
Pre-trained language models have achieved state-of-the-art results in various natural language processing tasks. Most of them
Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models
R
, which distinguishes tokens
. However, it distinguishes sequential tokens only
R
of the input sequence. However, sentence index and paragraph index are also important to indicate the token position in a document.
.
R
text encoder
Transformer
R
BERT
Transformer (Segatron)
R
token position embedding of Transformer with a combination of paragraphindex, sentenceindex
original token position encoding with a combined position encoding of paragraph
A
sentence,
R
index embeddings. We pre-trained the SegaBERT on the
. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training
R
in BERT but without any affiliated tasks
with Segatron
R
our
BERT
R
model can outperform the original BERT model
with Segatron (SegaBERT) can outperform BERT with vanilla Transformer
A
, and outperforms RoBERTa on zero-shot sentence representation learning