Hie- BART : Document Summarization with Hierarchical BART (2021)

Abstract: This paper proposes a new abstractive document summarization model, hierarchical BART (Hie-BART), which captures hierarchical structures of a document (i.e., sentence-word structures) in the BART model. Although the existing BART model has achieved a state-of-the-art performance on document summarization tasks, the model does not have the interactions between sentence-level information and word-level information. In machine translation tasks, the performance of neural machine translation models has been improved by incorporating multi-granularity self-attention (MG-SA), which captures the relationships between words and phrases. Inspired by the previous work, the proposed Hie-BART model incorporates MG-SA into the encoder of the BART model for capturing sentence-word structures. Evaluations on the CNN/Daily Mail dataset show that the proposed Hie-BART model outperforms some strong baselines and improves the performance of a non-hierarchical BART model (+0.23 ROUGE-L).

## introduction in recent years, improvements to abstractive document summarization models have been developed through the incorporation of pre-training. the bertsum model @xcite has been proposed as a pre-training model for document summarization tasks. for sequence-tosequence tasks, the t5 model @xcite and the bart model @xcite have been proposed as part of generalized pre-training models. among the existing pre-training models, the bart model achieves state-of-the-art performance on document summarization tasks. however, the bart model does not capture the hierarchical structures of documents when generating a summary. neural machine translation has been improved by the capture of multiple granularities of information in input texts such as "phrases and words" and "words and characters". in particular, transformer-based machine translation model has been improved by incorporating multi-granularity self-attention (mg-sa) @xcite , which considers the relationships between words and phrases by decomposing an input text into its elements using multiple granularity (i.e., words and phrases) and assigning each granular element (i.e., a word or a phrase) to a head in multi-head self-attention networks (sans). this method enables interactions not only between words but also between phrases and words, through self-attentions. inspired by previous work, this paper proposes a new abstractive document summarization model, hierarchical bart (hie-bart), which captures a document's hierarchical structures (i.e., sentenceword structures) through the sans of the bart model. here, a document is divided into elements with word-level and sentence-level granularity, where each element is assigned to a head of the sans layers of the bart encoder. then, information with multi-granularity is captured by combining the output of the sans layers, where the ratio of combining word-level and sentencelevel information is controlled by a hyperparameter. we evaluated the proposed model in an abstractive summarization task with the cnn/daily mail dataset. our evaluation shows that our hie-bart model improves the f-score of rouge-l by 0.23 points relative to the non-hierarchical bart model, and the proposed model is better than the strong baselines, bertsum and t5 models. ## conclusion in this study, we proposed hie-bart to can take into account the relationship between words and sentences in bart by dividing the self-attention layer of encoder into word and sentence levels. in the experiments, we confirmed that hie-bart improved the f-score of rouge-l by 0.23 points relative to the non-hierarchical bart model, and the proposed model was better than the strong baselines, bertsum and t5 models for the cnn/daily mail dataset. as future work, we intend to investigate methods to incorporate information between sentences in addition to word-to-word and word-to-sentence information.