ERNIE - D oc: A Retrospective Long-Document Modeling Transformer (2021)

Abstract: Transformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes. In this paper, we propose ERNIE-Doc, a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism, enable ERNIE-Doc, which has a much longer effective context length, to capture the contextual information of a complete document. We pretrain ERNIE-Doc to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various experiments were conducted on both English and Chinese document-level tasks. ERNIE-Doc improved the state-of-the-art language modeling result of perplexity to 16.8 on WikiText-103. Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks, such as text classification and question answering.

## introduction transformers @xcite have achieved remarkable improvements in a wide range of natural language tasks, including language modeling @xcite ), text classification @xcite , and question answering @xcite . this success is largely due to the self-attention mechanism, which enables the network to capture contextual information from the *indicates equal contribution. 1 source code and pre-trained checkpoints can be found at https://github.com/paddlepaddle/ernie/ tree/repro/ernie-doc . entire input sequence. nevertheless, the memory usage and computation complexity caused by the self-attention mechanism grows quadratically with the sequence length, incurring excessive cost when processing a long document on existing hardware. currently, the most prominent pretrained models, such as bert @xcite , are used on fixed-length input segments of a maximum of 512 tokens owing to the aforementioned limitation. thus, a long document input must be partitioned into smaller segments of manageable sizes. however, this leads to the loss of important crosssegment information, that is, the context fragmentation problem @xcite , as shown in fig. 1(a ). to mitigate the problem of insufficient interactions among the partitioned segments of long documents, recurrence transformers @xcite permit the use of contextual information from previous segments in computing the hidden states for a new segment by maintaining a memory component from the previous activation; this enables the modeling of long documents. in addition, sparse attention transformers @xcite @xcite focus on reducing the complexity of self-attention operations to explicitly improve the modeling length, but only up to a restricted context length (4,096) due to resource limitations. we argue that existing strategies are not sufficiently effective or reliable, because the contextual information of a complete document is still not available for each segment during the training phase. as depicted in fig. 1 , when training on segment s 2 , the model is ideally optimized by maximizing p (y | (s 1 , s 2 , s 3 )) conditioned on the contextual information of the entire document @xmath0, s 2 , s 3 }, in contrast to the following suboptimal solutions: p (y | s 2 ) for vanilla/sparse transformers 2 and p (y | (s 1 , s 2 )) for recurrence transformers. to address this limitation, we propose ernie-doc (a retrospective long-document modeling transformer) based on the recurrence transformer paradigm. inspired by the human reading behavior of skimming a document first and then looking back upon it attentively, we design a retrospective feed mechanism in which segments from a document are fed twice as input. as a result, each segment in the retrospective phase could explicitly fuse the semantic information of the entire document learned in the skimming phase, which prevents context fragmentation. however, simply incorporating the retrospective feed mechanism into recurrence transformers is infeasible because the maximum effective context length is limited by the number of layers @xcite , as shown in fig. 1 (b ). thus, we present an enhanced recurrence mechanism, a drop-in replacement for a recurrence transformer, by changing the shifting-one-layer-downwards recurrence to the same-layer recurrence. in this manner, the maximum effective context length can be expanded, and past higher-level representations can be exploited to enrich future lower-level representations. moreover, we introduce a segment-reordering objective to pretrain a document-level model. specifically, it is a document-aware task of predicting the correct order of the permuted set of segments of a document, to model the relationship among segments directly. this allows ernie-2 for sparse transformers, the length of segment s2 could be up to 4,096 in @xcite ; @xcite . doc to build full document representations for prediction. this is analogous to the sentencereordering task in ernie 2.0 @xcite but at a segment level of granularity, spanning (commonly) multiple training steps. we first evaluate ernie-doc on autoregressive word-level language modeling using the enhanced recurrence mechanism, which, in theory, allows the model to process a document with infinite words. ernie-doc achieves state-of-theart (sota) results on the wikitext-103 benchmark dataset, demonstrating its effectiveness in long-document modeling. then, to evaluate the potential of ernie-doc on document-level natural language understanding (nlu) tasks, we pretrained the english ernie-doc on the text corpora utilized in bigbird @xcite ) from the roberta-released checkpoint, and the chinese ernie-doc on the text corpora utilized in ernie 2.0 @xcite ) from scratch. after pretraining, we fine-tuned ernie-doc on a wide range of english and chinese downstream tasks, including text classification, question answering and keypharse extraction. empirically, ernie-doc consistently outperformed roberta on various benchmarks and showed significant improvements over other high-performance long-text pretraining models for most tasks. recurrence transformers @xcite have been successfully applied in generative language modeling. they employ the transformer decoder as a parametric model for each conditional distribution in p(x) = l @xmath1, where x denotes a text sequence. to capture long dependencies, they process the text in segments from left to right based on the segment recurrence mechanism @xcite . this mechanism maintains a memory bank of past activations at each layer to preserve a history of context. compressive transformer @xcite adds a compressive memory bank to sufficiently store old activations instead of discarding them, which facilitates long-range sequence learning. however, these methods operate from left to right, which limits their capacity for discriminative language understanding tasks that require bidirectional information. xlnet @xcite proposed a permutation language modeling objective to construct bidirectional information and achieve superior performance in multiple nlp tasks; however, its application to long-document modeling tasks remains largely unexplored. ernie-doc builds on the ideas of the recurrence transformers to 1) tackle the limitation of recurrence transformers for utilizing bidirectional contextual information and 2) improve the behavior of the segment recurrence mechanism to capture longer dependencies. hierarchical transformers @xcite have enabled significant progress on numerous document-level tasks, such as document summarization @xcite and document ranking @xcite . similar to vanilla transformers, hierarchical transformers also split long documents into shorter segments with manageable lengths and then feed them independently to produce corresponding segment-level semantic representations. unlike in vanilla transformers, however, separate transformer layers are used in hierarchical transformers to process the concatenation of these representations. hierarchical transformers ignore the contextual information from the remaining segments when processing each segment of a long document, thus suffering from the context fragmentation problem. ## proposed method in this section, we first describe the background (sec. 3.1) that ernie-doc builds on. then, we present the implementation of ernie-doc, including the retrospective feed mechanism in sec. 3.2, the enhanced recurrence mechanism in sec. 3.3, and the segment-reordering objective in sec. 3.4. ## conclusion in this paper, we proposed ernie-doc, a document-level language pretraining model based on the recurrence transformers paradigm. two well-designed mechanisms, namely the retrospective feed mechanism and the enhanced recurrent mechanism, enable ernie-doc, which theoretically has the longest possible dependency, to model bidirectional contextual information of a complete document. additionally, ernie-doc is pretrained with a document-aware segment-reordering objective to explicitly learn the relationship among segments of a long context. experiments on various downstream tasks demonstrate that ernie-doc outperforms existing strong pretraining models such as roberta, longformer, and bigbird and achieves sota results on several language modeling and language understanding benchmarks. in future studies, we will evaluate ernie-doc on language generation tasks, such as generative question answering and text summarization. we will also investigate its potential applicability in other areas, such as computational biology. another possibility is to incorporate graph neural networks into ernie-doc to enhance its modeling capability