H i S truct+: Improving Extractive Text Summarization with Hierarchical Structure Information (2022)

Abstract: Transformer-based language models usually treat texts as linear sequences. However, most texts also have an inherent hierarchical structure, i.e., parts of a text can be identified using their position in this hierarchy. In addition, section titles usually indicate the common topic of their respective sentences. We propose a novel approach to formulate, extract, encode and inject hierarchical structure information explicitly into an extractive summarization model based on a pre-trained, encoder-only Transformer language model (HiStruct+ model), which improves SOTA ROUGEs for extractive summarization on PubMed and arXiv substantially. Using various experimental settings on three datasets (i.e., CNN/DailyMail, PubMed and arXiv), our HiStruct+ model outperforms a strong baseline collectively, which differs from our model only in that the hierarchical structure information is not injected. It is also observed that the more conspicuous hierarchical structure the dataset has, the larger improvements our method gains. The ablation study demonstrates that the hierarchical position information is the main contributor to our model’s SOTA performance.

## introduction texts, especially long documents, contain internal hierarchical structure like sections, paragraphs, sentences, and tokens. when we manually summarize a text, the hierarchical text structure usually plays a key role. taking a scientific paper as an example, we might focus more on the sections with the titles of "methodology", "discussion", and "conclusion" while paying less attention to the sections like "background". furthermore, the sentences within one section could have closer relationship with each other, than the ones outside this section. understanding not only the sequential relations between the sentences but also the internal hierarchical text structure helps us better determine the important sentences within a document. similarly, a neural summarization model could benefit from these hierarchical structure information. in this paper, we focus on extractive text summarization of single documents, which is the task of binary sentence classification with labels indicating whether a sentence should be included in a summary. recently, pre-trained language models based on transformer @xcite , such as bert @xcite , have been widely used to extract contextual representations from texts. the pre-trained transformer language models (tlms) can be easily reused for finetuning on the downstream tasks, so that the representations already learned from the large pretraining corpora are preserved. @xcite have achieved the state-of-the-art (sota) performance by fine-tuning bert for extractive summarization on short document datasets including cnn/dailymail. however, the tlms consider merely the sequential-context-dependency by adding a linear positional encoding to each input token embeddings. the hierarchical text structure information is not taken into account explicitly. we propose a novel approach to formulate, extract, encode and inject the hierarchical structure (histruct) information explicitly into an extractive summarization model (histruct+ model), which consists of a tlm for sentence encoding and two stacked inter-sentence transformer layers for hierarchical learning and extractive summarization (see figure 1 ). we experiment with bert @xcite , roberta @xcite , and longformer @xcite as underlying tlms. the histruct information utilized in our work includes the section titles and the hierarchical positions of sentences, which are encoded using our proposed novel methods. the resulting embeddings can be injected into the tlm sentence representations to provide the histruct information for the summarization task. ## results and discussion we evaluate the performance of our summarization models automatically using rouge metrics @xcite including f1 rouge-1 (r1), rouge-2 (r2) and rouge-l (rl). tables 1, 2 and 3 summarize the performance of our models in comparison to the baselines and the previously reported sota results on cnn/dailymail, pubmed and arxiv respectively. on all three datasets, ablation studies are systematically conducted to investigate the contributions of different experimental settings. to analyze the output summaries from an overall perspective, we plot the distribution of the extracted sentences on each dataset and compare it to the oracle summaries and those outputted by the comparison baseline (see figure 2 ). appendix a.6 demonstrates human evaluation of extracted summaries for a more intuitive understanding about the superiority of the proposed system. ## conclusions this work addresses hierarchical modeling for extractive text summarization by explicitly leveraging hierarchical structure information, including section titles, as well as hierarchical position information of the sentences. we propose an intuitive and interpretable approach to formulate, extract, encode and inject the hierarchical structure information into an extractive summarization model. the proposed histruct+ models are systematically evaluated on cnn/dailymail, pubmed, and arxiv. on pubmed, our model increases the former sota rouges for extractive summarization by 1.2/0.02/1.12. on arxiv, the new sota results for extractive summarization are set to 45.22/17.67/40.16. our ablation studies suggest that the sota performance are mostly gained by providing the hierarchical position information of sentences to the summarization model. when comparing the histruct+ models with the baselines that are unaware of the histruct information, improvements are consistently observed on all three datasets under various experimental settings, indicating the effectiveness of the proposed method. moreover, our experiments show that the more conspicuous hierarchical structure the dataset has, the larger the improvements of our method are. the proposed metrics of hi-depth and hi-width determine whether it is worth using our method by comparing the metrics of any dataset to those of the three involved datasets. utilizing the histruct information also for abstractive summarization is subject of future work. similarly, we see great potential in an encoderdecoder architecture with the proposed histruct injection components.