Globalizing BERT -based Transformer Architectures for Long Document Summarization (2021)

Abstract: Fine-tuning a large language model on downstream tasks has become a commonly adopted process in the Natural Language Processing (NLP) (CITATION). However, such a process, when associated with the current transformer-based (CITATION) architectures, shows several limitations when the target task requires to reason with long documents. In this work, we introduce a novel hierarchical propagation layer that spreads information between multiple transformer windows. We adopt a hierarchical approach where the input is divided in multiple blocks independently processed by the scaled dot-attentions and combined between the successive layers. We validate the effectiveness of our approach on three extractive summarization corpora of long scientific papers and news articles. We compare our approach to standard and pre-trained language-model-based summarizers and report state-of-the-art results for long document summarization and comparable results for smaller document summarization.

## introduction language model pre-training has become a key component to improve performances on a majority of natural language processing (nlp) tasks @xcite . most of the recent competitive architectures @xcite @xcite are based on the efficient transformer layer introduced in @xcite . bert @xcite is one of these architectures that has been widely adopted for comprehension and generation tasks. it is a multi-layer transformer network, pretrained with different self-supervised objectives. numerous variations of transformer architectures have been proposed to improve this approach @xcite @xcite . however, this type of process is only evaluated on tasks composed of relatively short input text, glue @xcite , squad @xcite , swag @xcite . indeed, for the tasks that require reasoning with longer documents, this approach exhibits several limitations. the transformer self-attention memory quadratically increases with the number of input tokens, making it technically impossible to compute on document-scale sequences. in addition, they usually require to define a fixed maximum input length, typically of 512 tokens, at the pre-training stage. one solution is to pre-train the entire model on longer sequences. however, this will still require a massive computation power and will only push the length limitation further. other alternatives have been proposed to extend multi-layer transformers architectures to longer sequences without modifying this maximum length limitation. the first one is to limit the input sequence to its first tokens by removing the text beyond the length limit. obviously, it cannot be a reasonable solution to treat long documents that are consistently longer than this limit. the second alternative is to apply the model on a window that slides all over the document. it has been used in @xcite to deal with squad documents that are longer than the 512 token limitation and in @xcite for a co-reference resolution task on long documents. this approach can only work if the tokens need to be contextualized only in their surroundings because there is no interaction between the different windows. it seems to be a solution for co-reference resolution @xcite as they usually can be solved with a reasonably sized window. another approach adopted to deal with long documents or multi-document is to select a sub-sample of the input that is small enough for the transformer model. most of the state-of-the-art pipelines on the multihop question answering dataset hotpotqa @xcite use a first model to retrieve the relevant pieces of text before feeding them to a transformerbased architecture @xcite . we argue that these solutions are not feasible to deal with tasks that require a global understanding of long documents. an example is extractive summarization, where the decision for each sentence should be based on the information of the complete document. to address these challenges, we propose a simple adaptation of the multi-layer transformer architecture that can scale to long documents and benefit from pre-trained parameters with a relatively small length limitation. the general idea is to independently apply a transformer network on small blocks of a text, instead of a long sequence, and to share information among the blocks between two successive layers. to the best of our knowledge, this is the first attempt to introduce hierarchical components directly between the layers of a pre-trained model and not only on top of it @xcite @xcite . between each of the transformer layers, we use a bidirectional gated recurrent unit (bigru) network @xcite to spread global information across the blocks. adding these propagation layers between the transformer layers preserves the original structure of the pre-trained model and makes it possible to transfer parameter weights from a large pre-trained language model with only few additional parameters to propagate information between blocks. the contributions of this paper can be summarized as follows: (i) we propose a novel architecture dedicated to long documents which interweaves recurrent hierarchical modules with transformer layers and which exploits pre-trained language models like bert, and (ii) we demonstrate that this architecture is able to build informative representations in the context of extractive summarization. ## global bert-based transformer architecture in this part, we briefly recall the transformer layer from @xcite and its integration in the bert model @xcite . then we describe our modifications of this architecture that allow the model to read longer documents. transformers: the transformer architecture, based on a sequence of transformer layers, has been initially introduced in @xcite . the key idea of this layer is to produce a contextualized representation of an input sequence of tokens. it is composed of the succession of a multi-head selfattention, a first normalizer, a feed-forward neural network, and a second normalizer. this model, which has originally been introduced for machine translation, has then been adopted for most natural language comprehension tasks. most of the successful approaches @xcite @xcite are composed of multiple stacked transformer layers. in the remainder, we denote by t ℓ the transformation corresponding to the ℓ th , 1 ≤ ℓ ≤ l, transformer layer (t ℓ is a function from r n ×h to r n ×h , where n denotes the length of the sequence and h the hidden dimension). bert @xcite ) is a multi-layer transformer encoder pre-trained on large text corpora. two bert architectures have been proposed in @xcite : bert base composed of 12 stacked transformer layers with hidden dimension of 768 (@xmath0, @xmath1, @xmath2. for both architectures, the input length is limited to 512 wordpiece tokens and the pre-training includes two self-supervised tasks, namely masked language modeling and next sentence prediction. for masked language modeling, 15% of all the wordpiece tokens of the input sequence are masked or corrupted, and the model is used to predict the original token with a crossentropy loss. for next sentence prediction, the model is trained as a classifier to predict if two sentences are contiguous or not. the pre-training procedure uses the bookscorpus @xcite and documents from english wikipedia. it requires 4 days of optimization on 16 tpu chips for bert base and 64 tpu chips for bert large . ## experiments we evaluate our approach, which we refer to as gbt-extsum (for 'global bert-based transformer for extractive summarization'), in the context of extractive summarization, the goal of which being to identify and extract from a document the pieces of text that are the most important @xcite . we view this task as a sentencelevel classification problem where each sentence has to be labeled according to its belonging to the summary or not. to validate the effectiveness of our approach, we propose to test it on three summarization datasets, namely arxiv, pubmed and cnn/dailymail: ## conclusion in this paper, we have introduced a novel transformer-based model for long document summarization based on propagation layers that spread information between multiple transformer windows. this model preserves the architecture of commonly used pre-trained language models, thus allowing the transfer of parameters. an evaluation, conducted on top of the bert model in the context of an extractive summarization task, further revealed its effectiveness in dealing with long documents compared to other adaptations of bert and previously proposed models. in the future, we plan to adapt our model to other tasks that require understanding long documents, as question-answering and document-scale machine translation.