Jointly Learning Salience and Redundancy by Adaptive Sentence Reranking for Extractive Summarization (2021)

Abstract: Extractive text summarization seeks to extract indicative sentences from a source document andassemble them to form a summary. Selecting salient but not redundant sentences has alwaysbeen the main challenge. Unlike the previous two-stage strategies this paper presents a unifiedend-to-end model learning to rerank the sentences by modeling salience and redundancy simul-taneously. Through this ranking mechanism our method can improve the quality of the overall candidate summary by giving higher scores to sentences that can bring more novel informa-tion. We first design a summary-level measure to evaluate the cumulating gain of each candidate summaries. Then we propose an adaptive training objective to rerank the sentences aiming atobtaining a summary with a high summary-level score. The experimental results and evalua-tion show that our method outperforms the strong baselines on three datasets and further booststhe quality of candidate summaries which intensely indicate the effectiveness of the proposed framework.

## introduction extractive summarization aims to create a summary by identifying and concatenating the most important sentences in a document @xcite @xcite @xcite . to choose an appropriate sentence from the document, we need to consider two aspects: salience, which represents how much information the sentence carries; and redundancy, which represents how much information in the sentence is already included in the previously selected sentences @xcite . the former focuses on sentence-level importance, while the latter considers the relationship between sentences at the summary-level. the main challenge of extractive summarization is how to combine salience and redundancy simultaneously. most previous methods @xcite only consider salience at the sentence-level, where they usually model sentence-selecting as a sequence labeling task. several approaches for modeling redundancy between selected sentences are generally classified into two types: heuristics-based and model-based approaches @xcite . the former such as trigram blocking @xcite is not adaptive since they usually apply the same rule to all the documents, which results in limited effects on a few specific datasets. the latter depends heavily on feature engineering or a neural post-processing module to model redundancy. @xcite extracts ngram-matching and semantic-matching features to indicate the redundancy of a candidate sentence. @xcite proposes a two-step pipeline that first scores salience, then learns to balance salience and redundancy as a text-matching task. the critical drawbacks are error propagation and high computation cost. compared to these models, we aim to propose an efficient and unified end-to-end model to jointly learn salience and redundancy without extra post-processing modules. b. c. d. e. rank a 0.74 b 0.63 c 0.44 d 0.37 e 0.34 re-rank a 0.73 b 0.67 e 0.40 d 0.36 c 0.33 summary-level score: 0.45 0.68 figure 1: a summary sample whose sentences are scored by the baseline model(labeled as rank) and our model(labeled as re-rank) respectively. both models only select sentences with top-3 scores as candidate summaries. sentences on blue background constitute the gold summary. phrases painted in the same color indicate n-gram overlap. in this paper, we present a unified end-to-end ranking-based method for extractive summarization. unlike previous methods, we train a single unified model that is both salience-aware and redundancy-aware to extract high-quality sentences at the summary level. the principle idea is that the best summary should consist of candidate sentences that can make the largest cumulating metric gain. as shown in figure 1 , only modeling salience when scoring sentences leads to give a high score to a salient but redundant sentence. due to the redundancy between the sentences labeled as b and c in figure 1 , the readers can not get the most overall information from the candidate summary consisting of the top 3 sentences. our method aims to give higher scores to the sentences containing novel but important information by globally considering its contribution to the summary-level metric gain, and generates a candidate summary with a high summary-level score. specifically, similar to the intuition in @xcite , we first define a new summarization evaluation measure -normalized cumulating gain based on the overlap of @xcite between candidate sentences and gold summary, which can better evaluate the overall quality of candidate summaries by considering the distances of multiple ngram matching scores between the candidate summary and golden summary. then we design a novel redundancy-aware ranking loss -adaptive summary ranking loss (adpsr-loss) modified by ncgon. we penalize the deviation of the predicted ranking position probabilities of sentence pairs from the desired probabilities, which leads our model to rerank sentences adaptively according to the difference of ncgon between candidate summaries, and finally find the best candidate summary end-to-end. we conduct experiments on a range of benchmark datasets. the experimental results demonstrate that the proposed ranking framework achieves improvements over previous strong methods on a large range of benchmark datasets. comprehensive analyses are provided to further illustrate the performance of our method on different summary length, matching n-gram and so on. our contributions are as follows: ## method in this section, we first introduce the overall architecture including the sentence scoring model and our training mechanism in section 3.1. then we introduce our designed reranking training objective in section 3.2. ## conclusions in this paper, we propose a novel end-to-end summary-level method jointly learning salience and redundancy in the extractive summarization task. experiments on three benchmark datasets confirm the effectiveness of our one-stage mechanism based on the adaptive ranking objective. moreover, we find that our method delivers more benefits to short-sentence summaries. we believe the power of this ranking-based summarization framework has not been fully exploited especially on the design of cumulating gain and fine-grained learning, and we hope to provide new guidance for future summarization work.