Abstractive Summarizers are Excellent Extractive Summarizers (2023)

Abstract: Extractive and abstractive summarization designs have historically been fragmented, limiting the benefits that often arise from compatible model architectures. In this paper, we explore the potential synergies of modeling extractive summarization with an abstractive summarization system and propose three novel inference algorithms using the sequence-to-sequence architecture. We evaluate them on the CNN & Dailymail dataset and show that recent advancements in abstractive system designs enable abstractive systems to not only compete, but even surpass the performance of extractive systems with custom architectures. To our surprise, abstractive systems achieve this without being exposed to extractive oracle summaries and, therefore, for the first time allow a single model to produce both abstractive and extractive summaries. This evidence questions our fundamental understanding of extractive system design, and the necessity for extractive labels while pathing the way for promising research directions in hybrid models.

## introduction extractive summarization selects a set of salient sentences from the original document(s) and composes them into a summary. compared to abstractive summaries, made up of words or phrases that do not appear in the input document, extractive summaries are less flexible but avoid inconsistencies and hallucinations. the pipeline for building an extractive summarizer typically consists of two separate stages: sentence labeling and extractive modeling. since few summarization datasets come with gold labels indicating which document sentences are summary-worthy, the first step is to create oracle sentence labels @xcite . the task is commonly modeled with a sequence labeling architecture @xcite where a salience score is estimated for each document sentence, and top-ranked sentences are selected for summary inclusion. recent work has expanded extractive modeling to higher-order sentence selection to account for complex label dependencies, via extracting sentences stepwise @xcite , or reranking a small set of summary candidates @xcite . in this work, we revisit these fundamental concepts in extractive summarization. specifically, we highlight that heuristically-derived sentence labels can be highly suboptimal @xcite , and that customized neural architectures for extractive modeling prevent taking advantage of independent improvements. we recognize that generative modeling with a neural encoder-decoder architecture @xcite , the de facto choice for abstractive summarization @xcite @xcite , constitutes a promising direction for extractive summarization. in particular, such models learn directly from abstractive references and do, therefore, not require sentence labeling, while also embodying the extractive capabilities previously enabled by specialized neural architectures. existing literature has established varied and many connections between abstractive and extractive modeling such as copy mechanism @xcite , content selection @xcite , and generation guidance @xcite . these connections, however, are mostly abstract-centric which are identified or constructed to improve abstractive summarization. in contrast, there are few studies from an extract-centric point of view. in this work, we propose a new summarization paradigm that unifies extractive and abstractive summarization with generative modeling, without compromising abstractive performance. to this end, we treat extractive summarization as an inference-time task and explore methods for adapting a pre-trained abstractive system for extractive summarization without further optimization. we hypothesize that an abstractive system can be used as a summary evaluator for not only abstracts but extracts as well. a model optimized on abstractve references should be able to provide an accurate quality estimation for an extractive candidate summary when conditioned on the input document. a straightforward approach to validate this assumption is to search for the best document extract with an abstractive model for candidate evaluation. however, performing an exhaustive search over a combinatorial space of all eligible summary candidates is computationally intractable. to tackle this challenge, we propose genx, generative extractive summarization, which introduces a set of inference algorithms (shown in figure 1 ) to reduce the search complexity via various approximations of the entire search space, at either sentence-or summary-level. experiments show that genx achieves competitive or superior performance compared to custom systems developed for extractive summarization on the cnn/dm benchmark without compromising its ability to generate abstracts. particularly, for one-stage summarization the proposed method shows superior results to custom extractive stateof-the-art systems. genx also exhibits high robustness in zero-shot transfer: on xsum, its zero-shot performance surprisingly surpasses its fully supervised counterpart. we further conduct an extensive analysis of genx's properties, providing potential directions for future research on generative modeling for extractive summarization. ## generative modeling for extracts given a generative model θ trained on summarization data comprising documents and abstractive references, at inference time, for an input document d and a summary sequence y , we estimate the length-normalized log probability of y , following the standard practice in neural text generation @xcite : as θ is optimized at the token level, we evaluate both complete and partial summaries with p θ (y |d). the candidate summary space for a document @xmath0, consisting of |c(d)| = c m n candidate summaries of length m. to sidestep the computational intractability, we introduce three inference algorithms that reduce the search complexity via approximations. the first two (ranking and reranking) construct a candidate summary set, using either a discriminative or generative model (see figure 1(a) ), while the last approach searches directly over the partial summary candidate space (see figure 1(b) ). ## experimental setup we perform supervised experiments on cnn/dm @xcite and zero-shot experiments on xsum @xcite . we evaluate summaries with rouge @xcite . details for our experimental settings and datasets can be found in appendix a. as there is no established baseline for extractive summarization with generative modeling, we construct posthoc rank, a posthoc method for direct comparison with genx. the baseline first generates an abstract using the abstractive model. then, the generated abstract is used to query document sentences and m sentences are retrieved with bm25 as the summary while applying tri-gram blocking. ## results supervised summarization the second block reports the performance of one-stage summarization systems. stepwise ect-sum @xcite ) is a state-of-the-art autoregressive system that learns to score partial summaries by selecting which sentence is a summary sentence iteratively. different from genx, it is a highly-customized extractive architecture optimized with extractive oracle summaries. as can be seen, genx performs on par with stepwise etc-sum, and outperforms bertsumext @xcite and robertasumext @xcite . two popular extractive systems based on sequence labeling. the third block presents the results of two-stage systems. trb denotes an additional stage for sentence selection with trigram blocking, an effective method for reducing redundancy. matchsum @xcite ) is a state-of-the-art extractive system that takes top-ranked sentences from bertsumext and then re-ranks the summary candidates composed by them with a model based on a siamese-bert architecture. as can be seen, genx models improve over the one-stage bertsumext and robertasumext, i.e., with or without bert-sumext as a sentence-level ranker. its reranking variant also outperforms bertsumext+trb and robertasumext+trb, showing that generative summary-level evaluation is more effective than heuristically-derived selection criteria. note, the performance of genx still falls short of state-ofthe-art matchsum. this is all achieved while the design allows the base generative model to retain its ability to produce abstractive summaries. this is not applicable to any existing extractive systems except posthoc rank, which shows significantly inferior performance. ## ablation study we further assessed genx with an ablation study. replacing brio (trained with mle and contrastive loss) with bart (trained with mle) leads to the largest performance drop. with the augmentation of contrastive learning, the abstractive system is competent in the dual role of both a generation and evaluation model, emphasizing the importance of calibrating a generative model on its summarylevel probability, even for its extractive inference. the dynamic stopping mechanism introduced in equation ( 4 ) performs on par with fixed-step search, showing that learning directly from abstracts is a promising way to teach models when to stop for summary extraction. genx is also shown to be able to search for extractive summaries of less redundancy as its performance can not be further improved by incorporating trigram blocking. ## efficiency we have shown that abstractive systems are capable extractive summarizers, however, it is important to highlight that the proposed method exhibits different computational requirements than that of contemporary extractive designs. unlike extractive designs that compute a single score for a candidate sentence or summary (via a classification token), abstractive systems produce scores for all individual tokens in a candidate summary foot_2 . computing these extra tokens causes approaches such as ranking and reranking with genx more computationally demanding. however, when combined with search genx stands as an efficient solution to searching through an otherwise intractable candidate summary space. this is enabled by an abstractive system's ability to sequentially score text (see equation 1 ) and boils down to the complexity of beam search. this is a clear improvement in computational efficiency over systems like match-sum which only supports scoring complete summaries and must exhaustively recompute different permutations in the candidate summary spaces. to make this strategy computationally tractable these models resort to heavy pruning which limits the expressiveness that high-order modeling otherwise enables. ## related work there is a plethora of work on controlling different aspects of summarization, from content @xcite to formats @xcite . in this work, we offer efficient and effective control over the summary type (extract versus abstract) during inference. recent work also investigates how to treat discriminative tasks such as information extraction and retrieval with generative modeling and its effectiveness for entities @xcite and string identifiers @xcite . others have suggested delegating extractive inference to the encoder of a generative model @xcite . despite the resemblances, extractive summarization with generative modeling remains under-explored and stands as a promising research direction with the surge of innovations in large language models. ## conclusion in this paper, we explored the possibility of modeling extractive summarization with an abstractive system. we proposed three novel inference algorithms which allow an abstractive model to perform the extractive task. our results showed that not only is extractive summarization feasible, but recent systems are directly competitive with contemporary extractive systems. this work shows that extractive and abstractive paradigms can be unified through a sequence-to-sequence design, removing the need for oracle summary labels and custom extractive model architectures.