EDU -level Extractive Summarization with Varying Summary Lengths (2023)

Abstract: Extractive models usually formulate text summarization as extracting fixed top-k salient sentences from the document as a summary. Few works exploited extracting finer-grained Elementary Discourse Unit (EDU) with little analysis and justification for the extractive unit selection. Further, the selection strategy of the fixed top-k salient sentences fits the summarization need poorly, as the number of salient sentences in different documents varies and therefore a common or best k does not exist in reality. To fill these gaps, this paper first conducts the comparison analysis of oracle summaries based on EDUs and sentences, which provides evidence from both theoretical and experimental perspectives to justify and quantify that EDUs make summaries with higher automatic evaluation scores than sentences. Then, considering this merit of EDUs, this paper further proposes an EDU-level extractive model with Varying summary Lengths (EDU-VL) and develops the corresponding learning algorithm. EDU-VL learns to encode and predict probabilities of EDUs in the document, generate multiple candidate summaries with varying lengths based on various k values, and encode and score candidate summaries, in an end-to-end training manner. Finally, EDU-VL is experimented on single and multi-document benchmark datasets and shows improved performances on ROUGE scores in comparison with state-of-the-art extractive models, and further human evaluation suggests that EDU-constituent summaries maintain good grammaticality and readability.

## introduction automatic text summarization aims at aggregating information in long document(s) into a shorter piece of text while keeping important information. extractive summarization and abstractive summarization are two categories of it. this paper focuses * corresponding author. 1 https://github.com/yuping-wu/edu-vl document: (...) [the second audio,] [taken from dash cam video from inside a patrol car,] [captures a phone call between slager and someone] [cnn believes] [is his wife.] (...) reference summary: the second audio captures a phone call between slager and someone cnn believes is his wife. most existing works extract sentences from the document and some works further @xcite propose post-processing steps to prune the generated summary. the only exception is the few works @xcite , which extract finer-grained textual segments, i.e., discourse-level text or edu, with little justification. the intuition is that a sentence consisting of multiple clauses is inevitable to contain less important information. as demonstrated in when selecting textual segments, the top-k strategy with k fixed for all documents is dominant in deciding the length of the generated summary. some works @xcite manage to output summaries with different lengths, i.e., various numbers of extracted segments, via formulating the problem as deriving a subset of sentences from the combination of top-k sentences. due to the foreseeing explosion of the combination of sentences to form subsets, these approaches are limited to generating summaries with relatively small values of k. to overcome such a weakness, the second main motivation of this paper is to propose and develop an approach allowing varying lengths for extractive summarization without explicit limitation on the maximum value of k, i.e., the maximum length. following the above motivations, the comparison analysis between edus and sentences ascertains that edu is a better text unit for the extractive task because edu-level summaries achieve higher automatic evaluation scores than sentencelevel summaries. this conclusion is justified from two perspectives. theoretically, a formal theorem about this conclusion could be derived from the property that edu is essentially part of a sentence. experimentally, results of comprehensive analysis about oracle summaries of five datasets further quantify this conclusion, i.e., how much the rouge scores of edu-level oracle summary are higher than sentence-level oracle summary. based on the aforementioned conclusion and foundation, this paper further proposes and develops an edu-level extractive model and algorithm, which generates summaries with varying lengths, i.e., edu-vl. we extend transformer-based pretrained language model with an extra classification layer to encode edus in a document and predict the corresponding probabilities. multiple k values are provided to the model to generate a set of candidate summaries under the flexible top-k strategy for the document. multiple transformer encoder layers encode the full document and candidate summaries individually. finally, a similarity score with the encoded document is calculated for each candidate summary and the one with the highest score is the final output of edu-vl. experiments are conducted on five benchmark datasets from different domains and with various writing styles. the experimental results suggest that edu-vl achieves better performance than all state-of-the-art extractive baselines on singledocument summarization datasets cnn/dailymail, xsum, reddit, and wikihow, in terms of three rouge metrics. with direct comparison to the multi-document model, edu-vl still achieves comparable performance on the multi-document summarization dataset multi-news. human evaluation is further carried for the summaries generated by edu-vl to assess the syntax structure of edu-constituent summaries. the results provide evidence for the good grammaticality and readability of edu-constituent summaries and therefore justify the applicability. the contributions of this paper are threefold: ## oracle analysis of edus and sentences oracle analysis refers to the analysis of oracle summary whose definition is stated in section 3.1. we conducted oracle analysis from both theoretical and experimental perspectives to justify and quantify that discourse-level summary achieves higher scores on automatic evaluation metrics than sentence-level summary. ## edu-level extractive model with candidate summary generator given a predefined extraction lengths set @xmath0, . . . , k c ], the s-th candidate summary, cand s , consists of edus whose probabilities are in top-k s (p), i.e., [edu i 1 , . . . , edu i j , . . . , edu i k ] where i @xmath1, @xmath2, 2, ..., k s . the initial representation vector, cand c s , for cand s is the concatenation of representation vectors of edus in it. the initial document representation vector, d c , is aggregated from the representation vectors of all edus. document-level block multiple transformer encoder layers (mtl) are stacked to encode document-level information for document d c , and all candidate summaries, e.g., cand c s , separately, and generate d d and cand d s in equation (3). then cosine similarity, i.e., sim s in equation ( 4 ), is computed between the encoded document representation and the encoded s-th candidate summary representation. the candidate summary with the highest similarity with the document is taken as the final model-generated summary. experiments 5.1 datasets cnn/dailymail (hermann et al., 2015) is the most commonly used news dataset for the extractive task with human-written highlights as reference summary. the non-anonymized version was used in our experiments. xsum (narayan et al., 2018) is another news dataset with the first introductory sentence in the article as the reference summary. reddit (kim et al., 2019) is a dataset crawled from the social media forum with the content in the section tl;dr as the reference summary. experiments were conducted on the tifu-long version. wikihow (koupaee and wang, 2018) is a dataset crawled from the question-answering website with the first sentence in each paragraph as the reference summary. multi-news (fabbri et al., 2019) is a multi-document dataset with one summary for a cluster of documents. we follow zhong et al.'s (2020) setting to split reddit and multi-news datasets and concatenate multiple documents into one single document. the detailed statistics of the five datasets in our experiments can be found in appendix c. 5.2 baselines various extractive models are selected as baselines. hetformer (liu et al., 2021) modifies longformer with longer input lengths to implement multi-granularity attention and selects sentences. among models generating summaries with varying lengths, matchsum (zhong et al., 2020) selects among a set of candidate summaries derived from a trained sentence-level extractive model; hahsum (jia et al., 2020) transforms a document into a heterogeneous hierarchical graph and flexibly selects sentences based on a threshold. among models with sub-sentential segments as input, the proposed model by huang and kurohashi (2021) is another longformer-based model but extracts edus based on the constructed heterogeneous graph; discobert (xu et al., 2020) and d-sum (liu and chen, 2019) are models extracting discourse-level textual segments but they differ in whether integrating gnn into the model. sgsum @xcite is a multi-document model by encoding all documents within one cluster individually and selecting the best sub-graph. far @xcite is an unsupervised ranking model considering facet-specific information. ## conclusion in this paper, we verify and quantify the argument that the edu-level summary achieves higher automatic evaluation scores than sentence-level summary from both theoretical and experimental perspectives. we further propose an edu-level extrac-