  Transformers are not suited for processing long document input due to its
quadratically increasing memory and time consumption. Simply truncating a long
document or applying the sparse attention mechanism will incur the context
fragmentation problem or inferior modeling capability with comparable model
size. In this paper, we propose ERNIE-DOC, a document-level language
pretraining model based on Recurrence Transformers. Two well-designed
techniques, namely the retrospective feed mechanism and the enhanced recurrence
mechanism enable ERNIE-DOC with much longer effective context length to capture
the contextual information of a whole document. We pretrain ERNIE-DOC to
explicitly learn the relationship among segments with an additional
document-aware segment reordering objective. Various experiments on both
English and Chinese document-level tasks are conducted. ERNIE-DOC achieves SOTA
language modeling result of 16.8 ppl on WikiText-103 and outperforms
competitive pretraining models on most language understanding tasks such as
text classification, question answering by a large margin.
