Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer (2021)

Abstract: Sentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents. However, constructing a coherent and informative summary is difficult using a pre-trained BERT-based encoder since it is not explicitly trained for representing the information of sentences in a document. We propose a nested tree-based extractive summarization model on RoBERTa (NeRoBERTa), where nested tree structures consist of syntactic and discourse trees in a given document. Experimental results on the CNN/DailyMail dataset showed that NeRoBERTa outperforms baseline models in ROUGE. Human evaluation results also showed that NeRoBERTa achieves significantly better scores than the baselines in terms of coherence and yields comparable scores to the state-of-the-art models.

## introduction document summarization is a task of creating a concise summary from a given document while keeping the original content. in general, sentence extraction methods, which select sentences in a document to create its summary, have the advantages of truthfulness compared with abstractive methods @xcite and of fluency compared with word extraction methods @xcite . neural networks have achieved great success in sentence extraction-based document summarization @xcite . recently, @xcite proposed bert-sum, which utilizes bert @xcite for sentence representations to create a summary. although the use of bert resulted in significant performance improvement, this method decides the selection for each sentence independently. @xcite proposed discobert by considering inter-sentence information through discourse graphs to construct a coherent summary. although they achieved remarkable scores in rouge, it was ## nested tree structure in this section, we describe how we construct two different types of graphs for a nested tree structure: a discourse graph and a syntactic graph. we obtain discourse dependency relationships between sentences in a document through an rst parser. a given document can be parsed into a tree format with the rst parser, where each leaf node is an edu, a text span in the document. each text span has two types, nucleus and satellite. while the nucleus spans contain semantically salient information, the satellite spans support and modify the nucleus ones. we use the recent state-of-the-art rst parser foot_1 @xcite to build an rst discourse tree (rst-dt) for all documents and convert it to an inter-sentential rst-dt (isrst-dt). the isrst-dt is first converted into a dependency-based discourse tree (isdep-dt) using the method described in @xcite . then, parent-child dependency relationships for each sentence can be formed. we construct a directed graph for the discourse dependencies @xcite . a dependency parser is used to build up the syntactic dependency relationships between words @xcite . we construct an undirected graph for the syntactic dependencies by following the previous settings @xcite . let w i be the i-th token in a document @xmath0, w 2 , ..., w n }. our model predicts p(1|d, k), the probability of the k-th sentence in d being kept in a summary through the following modules. ## conclusion in this paper, we proposed neroberta, which incorporates syntactic and discourse information as a nested tree structure to create an informative and coherent summary. the experimental results on the cnn/dailymail dataset showed that our method improves the performance over the baseline methods both in the automatic and human evaluations.