Transformers are not suited for processing long document input due to its quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or  inferior modeling capability with comparable model size . In this paper, we propose ERNIE-DOC , a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism enable ERNIE-DOC with much longer effective context length  to capture the contextual information of a whole document. We pretrain ERNIE-DOC to explicitly learn the relationship among segments with an additional document-aware segment reordering objective. Various experiments  on both English and Chinese document-level tasks are conducted. ERNIE-DOC achieves SOTA language modeling result of  16.8 ppl on WikiText-103 and outperforms competitive pretraining models  on most language understanding tasks  such as text classification , question answeringby a large margin .
Transformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes . In this paper, we propose ERNIE-Doc , a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism , enable ERNIE-Doc, which has a much longer effective context length , to capture the contextual information of a complete document. We pretrain ERNIE-Doc to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various experiments were conducted on both English and Chinese document-level tasks . ERNIE-Doc improved the state-of-the-art language modeling result of perplexity to 16.8 on WikiText-103 . Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks , such as text classification and question answering .
R
document input due to its
documents, due to their
A
lead to an
R
with comparable model size
against comparable model sizes
R
ERNIE-DOC
ERNIE-Doc
R
enable ERNIE-DOC with
, enable ERNIE-Doc, which has a
A
,
R
whole
complete
R
ERNIE-DOC
ERNIE-Doc
R
relationship
relationships
R
segment reordering
segment-reordering
A
were conducted
R
are conducted. ERNIE-DOC achieves SOTA
. ERNIE-Doc improved the state-of-the-art
A
perplexity to
D
ppl
R
and outperforms
. Moreover, it outperformed
A
by a large margin
A
,
R
, question answeringby a large margin
and question answering