Systematically Exploring Redundancy Reduction in Summarizing Long Documents (2020)

Abstract: Our analysis of large summarization datasets indicates that redundancy is a very serious problem when summarizing long documents. Yet, redundancy reduction has not been thoroughly investigated in neural summarization. In this work, we systematically explore and compare different ways to deal with redundancy when summarizing long documents. Specifically, we organize existing methods into categories based on when and how the redundancy is considered. Then, in the context of these categories, we propose three additional methods balancing non-redundancy and importance in a general and flexible way. In a series of experiments, we show that our proposed methods achieve the state-of-the-art with respect to ROUGE scores on two scientific paper datasets, Pubmed and arXiv, while reducing redundancy significantly.

## introduction summarization is the task of shortening a given document(s) while maintaining the most important information. in general, a good summarizer should generate a summary that is syntactically accurate, semantically correct, coherent, and non-redundant @xcite . while extractive methods tend to have better performance on the first two aspects, they are typically less coherent and more redundant than abstractive ones, where new sentences are often generated by sentence fusion and compression, which helps detecting and removing redundancy @xcite . although eliminating redundancy has been initially and more intensely studied in the field of multidocument summarization @xcite , because important sentences selected from multiple documents (about the same topic) are more likely to be redundant than sentences from the same document, generating a non-redundant summary should still be one of the goals for single document summarization @xcite . generally speaking, there is a trade-off between importance and diversity (non-redundancy) @xcite , which is reflected in the two phases, sentence scoring and sentence selection @xcite in which extractive summarization task can be naturally decomposed. the former typically scores sentences based on importance, while the latter selects sentences based on their scores, but also possibly taking other factors (including redundancy) into account. traditionally, in non-neural approaches the tradeoff between importance and redundancy has been carefully considered, with sentence selection picking sentences by optimizing an objective function that balances the two aspects @xcite . in contrast, more recent works on neural extractive summarization models has so far over-emphasized sentence importance and the corresponding scoring phase, while paying little attention to how to reduce redundancy in the selection phase, where they simply apply a greedy algorithm to select sentences (e.g., @xcite ; @xcite ). notice that this is especially problematic for long documents, where redundancy tends to be a more serious problem, as we have observed in key datasets. improving redundancy reduction in neural extractive summarization for long documents is a major goal of this paper. indeed, some recently proposed neural methods aim to reduce redundancy, but they either do that implicitly or inflexibly and only focusing on short documents (e.g., news). for instance, some models learn to reduce redundancy when predicting the scores @xcite , or jointly learn to score and select sentences @xcite in an implicit way. however, whether these strategies actually help reducing redundancy is still an open empirical question. the only neural attempt of explicitly reduce redundancy in the sentence selection phase is the trigram blocking technique, used in recent extractive summarization models on news datasets (e.g., @xcite ). however, the effectiveness of such strategy on the summarization of long documents has not been tested. finally, a very recent work by @xcite attempts to reduce redundancy in more sophisticated ways, but still focusing on news. furthermore, since it relies on bert, such model is unsuitable to deal with long documents (with over 3,000 words). to address this rather confusing situation, characterized by unclear connections between all the proposed neural models, by their limited focus on short documents, and by spotty evaluations, in this paper we systematically organize existing redundancy reduction methods into three categories, and compare them with respect to the informativeness and redundancy of the generated summary for long documents. in particular, to perform a fair comparison we re-implement all methods by modifying a common basic model (xiao and carenini, 2019), which is a top performer on long documents without considering redundancy. additionally, we propose three new methods that we argue will reduce redundancy more explicitly and flexibly in the sentence scoring and sentence selection phase by deploying more suitable decoders, loss functions and/or sentence selection algorithms, again building for a fair comparison on the common basic model @xcite . to summarize, our main contributions in this paper are: we first examine popular datasets, and show that redundancy is a more serious problem when summarizing long documents (e.g., scientific papers) than short ones (e.g. news). secondly, we not only reorganize and re-implement existing neural methods for redundancy reduction, but we also propose three new general and flexible methods. finally, in a series of experiments, we compare existing and proposed methods on long documents (i.e., the pubmed and arxiv datasets), with respect to rouge scores @xcite and redundancy scores @xcite . as a preview, empirical results reveal that the proposed methods achieve state-of-the-art performance on rouge scores, on the two scientific paper datasets, while also reducing the redundancy significantly. ## related work in traditional extractive summarization, the process is treated as a discrete optimization problem balancing between importance scores and redundancy scores, with techniques like maximal marginal relevance(mmr) @xcite , redundancy-aware feature-based sentence classifiers @xcite and graph-based submodular selection @xcite . in recent years, researchers have explored neural extractive summarization solutions, which score sentences by training the neural models on a large corpus, and simply apply a greedy algorithm for sentence selection @xcite . although a model with a sequence decoder might plausibly encode redundancy information implicitly, @xcite empirically show that this is not the case, since non auto-regressive models (the ones scoring each sentence independently), perform on par with models with a sequence decoder. in one of our new methods, to effectively capture redundancy information, we specify a new loss that explicitly consider redundancy when training the neural model. beyond a greedy algorithm, the trigram blocking is frequently used to explicitly reduce redundancy in the sentence selection phase @xcite . in essence, a new sentence is not added to the summary if it shares a 3-gram with the previously added one. @xcite first adopt the strategy for abstractive summarization, which forces the model not to produce the same trigram twice in the generated summaries, as a simplified version of mmr @xcite . arguably, this method is too crude for documents with relatively long sentences or specific concentrations (e.g. scientific papers), where some technical terms, possibly longer than 2-grams, are repeated frequently in the 'important sentences' (even in the reference summaries). to address this limitation, we propose a neural version of mmr to deal with redundancy within the sentence selection phase in a more flexible way, that can be tuned to balance importance and non-redundancy as needed. the idea of mmr has also inspired zhou et al. @xcite , who propose a model jointly learning to score and select the sentences. yet, this work not only focuses on summarizing short documents (i.e., news), but also uses mmr implicitly, and arguably sub-optimally, by learning a score that only indirectly captures the trade-off between relevance and redundancy. to improve on this approach, in this paper we propose a third new method, in which importance and redundancy are explicitly weighted, while still making the sentence scoring and selection benefit from each other by fine tuning the trained neural model through a reinforcement learning (rl) mechanism. finally, @xcite is the most recent (still unpublished) work on reducing redundancy in neural single document summarization. however, their goal is very different form ours, since they focus on relatively short documents in the news domain. ## measuring redundancy: metrics and comparing long vs. short documents we use the following two relatively new metrics to measure redundancy in the source documents and in the generated summaries. unique n-gram ratio foot_1 : proposed in @xcite , it measures n-grams uniqueness; the lower it is, the more redundant the document is. normalized inverse of diversity (nid): captures redundancy, as the inverse of a diversity metric with length normalization. diversity is defined as the entropy of unigrams in the document @xcite . since longer documents are more likely to have a higher entropy, we normalize the diversity with the maximum possible entropy for the document log(|d|). thus, we have: note that higher nid indicates more redundancy. when we compare the redundancy of long vs. short documents with respect to these two metrics on four popular datasets for summarization (cn-ndm @xcite , xsum @xcite , pubmed and arxiv @xcite )), we observe that long documents are substantially more redundant than short ones (as it was already pointed out in the past @xcite ). ## redundancy reduction methods we systematically organize neural redundancy reduction methods into three categories, and compare prototypical methods from each category. a the decoder is designed to implicitly take redundancy into account. ## experiments in this section, we describe the settings, results and analysis of the experiments of different methods on the pubmed and arxiv datasets. ## conclusion and future work balancing sentence importance and redundancy is a key problem in extractive summarization. by examining large summarization datasets, we find that longer documents tend to be more redundant. therefore in this paper, we systematically explore and compare existing and newly proposed methods figure 5: comparing the average rouge scores and average unique n-gram ratios of different models with different word length limits on the pubmed dataset. see appendices for similar results on arxiv. for redundancy reduction in summarizing long documents. experiments indicate that our novel methods achieve sota on the rouge scores, while significantly reducing redundancy on two scientific paper datasets (pubmed and arxiv). interestingly, we show that redundancy reduction in sentence selection is more effective than in the sentence scoring phase, a finding to be further investigated . additional venues for future work include experimenting with generating summaries at finer granularity than sentences, as suggested by our analysis of the λ parameter. we also intend to explore other ways to assess redundancy, moving from computing the cosine similarity between sentence embeddings, to a pre-trained neural model for sentence similarity. finally, we plan to run human evaluations to assess the quality of the generated summaries. this is quite challenging for scientific papers, as it requires participants to possess sophisticated domain-specific background knowledge. the association for computational linguistics: student research workshop, pages 223-228, florence, italy. association for computational linguistics. pengjie ren, furu wei, zhumin chen, jun ma, and ming zhou. 2016. a redundancy-aware sentence regression framework for extractive summarization. in proceedings of coling 2016, the 26th international conference on computational linguistics: technical papers, pages 33-43, osaka, japan. the coling 2016 organizing committee. horacio saggion and thierry poibeau. 2013. automatic text summarization: past, present and future, pages 3-21. springer berlin heidelberg, berlin, heidelberg. jade stewart and jaime carbonell. 1998. summarization: (1) using mmr for diversity -based reranking and (2) evaluating summaries. pages 181-195. oriol vinyals, meire fortunato, and navdeep jaitly. 2015. pointer networks. in c. cortes, n. d. lawrence, d. d. lee, m. sugiyama, and r. garnett, editors, advances in neural information processing systems 28, pages 2692-2700. curran associates, inc. wen xiao and giuseppe carenini. 2019. extractive summarization of long documents by combining global and local context. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3002-3012, hong kong, china. association for computational linguistics. qingyu zhou, nan yang, furu wei, shaohan huang, ming zhou, and tiejun zhao. 2018. neural document summarization by jointly learning to score and select sentences. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 654-663, melbourne, australia. association for computational linguistics.