AREDSUM : Adaptive Redundancy-Aware Iterative Sentence Ranking for Extractive Document Summarization (2021)

Abstract: Redundancy-aware extractive summarization systems score the redundancy of the sentences to be included in a summary either jointly with their salience information or separately as an additional sentence scoring step. Previous work shows the efficacy of jointly scoring and selecting sentences with neural sequence generation models. It is, however, not well-understood if the gain is due to better encoding techniques or better redundancy reduction approaches. Similarly, the contribution of salience versus diversity components on the created summary is not studied well. Building on the state-of-the-art encoding methods for summarization, we present two adaptive learning models: AREDSUM-SEQ that jointly considers salience and novelty during sentence selection; and a two-step AREDSUM-CTX that scores salience first, then learns to balance salience and redundancy, enabling the measurement of the impact of each aspect. Empirical results on CNN/DailyMail and NYT50 datasets show that by modeling diversity explicitly in a separate step, AREDSUM-CTX achieves significantly better performance than AREDSUM-SEQ as well as state-of-the-art extractive summarization baselines.

## introduction extractive summarization is the task of creating a summary by identifying and concatenating the most important sentences in a document @xcite @xcite . given a partial summary, the decision to include another sentence in the summary depends on two aspects: salience, which represents how much information the sentence carries; and redundancy, which represents how much information in the sentence is already included in the previously selected sentences. although there have been a few studies on redundancy a long time ago, most recent research on extractive summarization focuses on salience alone. they usually model sentence salience as a sequence labeling task @xcite or classification task @xcite and do not conduct redundancy removal. previous methods that consider redundancy usually use a separate step after salience scoring to handle redundancy, denoted as sentence selection @xcite @xcite . sentence selection often follows a greedy iterative ranking process that outputs one sentence at a time by taking into account the redundancy of candidate sentences with previously selected sentences. several approaches for modeling redundancy in sentence selection have been explored: heuristicsbased methods such as maximal marginal relevance (mmr) @xcite , trigram blocking (triblk) @xcite , or model based approaches @xcite , etc. heuristic-based methods are not adaptive since they usually apply the same rule to all the documents. model-based approaches depend heavily on feature engineering and learn to score sentences via regression with point-wise loss, which has been shown to be inferior to pairwise loss or list-wise loss in ranking problems @xcite . redundancy has also been handled jointly with salience during the scoring process using neural sequence models @xcite . neusum @xcite scores sentences considering their salience as well as previous sentences in the output sequence and learns to predict the sentence with maximum relative gain given the partial output summary. despite the improved efficacy, it is not well-understood if the gain is due to better encoding or better redundancy-aware iterative ranking approaches (i.e., the sequence generation). in this work, we propose to study different types of redundancy-aware iterative ranking techniques for extractive summarization that handle redundancy separately or jointly with salience. extending bertsumext @xcite , a stateof-the-art extractive summarization model, which uses heuristic-based trigram blocking (triblk) for redundancy elimination, we propose two supervised redundancy-aware iterative sentence ranking methods for summary prediction. our first model, aredsum-seq, introduces a transformerbased conditional sentence order generator network to score and select sentences by jointly considering their salience and diversity within the selected summary sentences. our second model, aredsum-ctx, uses an additional sentence selection model to learn to balance the salience and redundancy of constructed summaries. it incorporates surface features (such as n-gram overlap ratio and semantic match scores) to instrument the diversity aspect. we compare the performance of our redundancy-aware sentence ranking methods with trigram-blocking @xcite , as well as summarization baselines with or without considering redundancy on two commonly used datasets, cnn/dailymail and new york times (nyt50). experimental results show that our proposed aredsum-ctx can achieve better performance by reducing redundancy and outperform all the baselines on these two datasets. the model's advantage can be attributed to its adaptiveness to scenarios in which redundancy removal has different potential gains. in summary, our contributions are: 1) we propose two redundancy-aware iterative ranking methods for extractive summarization extending bert-sumext; 2) we conduct comparative studies between our redundancy-aware models as well as the heuristic-based method that bertsumext uses; 3) our proposed aredsum-ctx significantly outperforms bertsumext and other competitive baselines on cnn/dailymail and nyt50. ## related work extractive summarization methods are usually decomposed into two subtasks, i.e., sentence scoring and sentence selection, which deal with salience and redundancy, respectively. salience scoring. graph-based models are widely used methods to score sentence salience in summarization @xcite @xcite . there are also extensions to such methods, e.g., with clus-tering @xcite or leveraging graph neural networks @xcite . classical supervised extractive summarization uses classification or sequence labeling methods such as naive bayes @xcite , maximum entropy @xcite , conditional random fields @xcite or hidden markov model @xcite . human engineered features are heavily used in these methods such as word frequency and sentence length @xcite . in recent years, neural models have replaced older models to score the salience of sentences. hierarchical lstms and cnns have replaced manually engineered features. lstm decoders are employed to do sequence labeling @xcite @xcite . these architectures are widely used and also extended with reinforcement learning @xcite . more recently, summarization methods based on bert @xcite have been shown to achieve state-of-the-art performance @xcite @xcite on salience for extractive summarization. sentence selection. there are relatively fewer methods that study sentence selection to avoid redundancy. integer linear programming based methods @xcite formulate sentence selection as an optimizing problem under the summary length constraint. @xcite propose to find the optimal subset of sentences with submodular functions. greedy strategies such as maximal marginal relevance (mmr) @xcite select the sentence that has maximal salience score and is minimally redundant iteratively. trigram blocking @xcite follows the intuition of mmr and filters out sentences that have trigram overlap with previously extracted sentences. @xcite leverage two groups of handcrafted features to capture informativeness and redundancy respectively during sentence selection. in contrast to learning a separate model for sentence selection, @xcite propose to jointly learn to score and select sentences with a sequence generation model. however, it is not compared with other redundancy-aware techniques, and it is not clear whether its improvement upon other methods is from the sequence generation method or the encoding technique. in this paper, we compare the efficacy of different sentence selection techniques grounded on the same bert-based encoder. we propose two models that either conduct redundancy removal with a separate model or jointly with salience scoring and compare them with a heuristic-based method. as far as we know, our work is the first to conduct comparative studies on different types of redundancyaware extractive summarization methods. ## iterative sentence ranking we formulate single document extractive summarization as a task of iterative sentence ranking. given a document @xmath0, s 2 , • • • , s l } of l sentences, the goal is to extract t sentences, i.e., ŝt = {ŝ k |1 ≤ @xmath1, ŝk ∈ d}, from d that can best summarize it. with a limit of selected sentence count l, the process of extracting sentences can be considered as a l-step iterative ranking problem. at each k-th step (1 ≤ @xmath2, given the current summary ŝk-1 , a new sentence s k is selected from the remaining sentences d \ ŝk-1 and added to the summary. function m ( ŝk ; s * ) foot_0 measures the similarity between the extracted summary ŝk and the ground truth summary s * . the objective is to learn a scoring function f (•) so that the best sentence ŝk selected according to f (•) can maximize the gain of the output summary: since ground truth summaries s * of existing summarization corpora are usually abstractive summaries written by experts, previous studies on extractive summarization usually extract a group of pseudo ground truth sentences ŝ * from d based on their similarities to the ground truth summaries s * for training purposes. then labels 1 and 0 are assigned to sentences in ŝ * and the other sentences in d. in this case, m ( ŝt ; ŝ * ) is used to guide training instead of m ( ŝt ; s * ). ## redundancy-aware summarization most recent redundancy-aware extractive summarization systems use heuristics to select diverse sentences after salience scoring @xcite @xcite . among them, bertsumext @xcite is a state-ofthe-art model with trigram-blocking (triblk) that reduces redundancy by filtering out sentences that have trigram overlap with previously selected ones at each time step. as we empirically show later in § 6, heuristics can be effective on some datasets yet harmful on others since it applies the same rule to all the documents. in contrast, we present an adaptive learning process for redundancy-aware extractive summarization, aredsum, and introduce two methods, aredsum-seq and aredsum-ctx, extending bertsumext by either consider redundancy jointly with salience during sentence scoring or separately with an additional selection model. ## conclusions extending a state-of-the-art extractive summarization model, we propose aredsum-seq that jointly scores and selects sentences with a sequence generation model and aredsum-ctx that learns to balance salience and redundancy with a separate model. experimental results show that aredsum-ctx outperforms aredsum-seq and all other strong baselines, which yields that redundancy reduction helps improve summary quality, and it is better to model the effect of redundancy explicitly than jointly with salience during sentence scoring.