D iffu S um: Generation Enhanced Extractive Summarization with Diffusion (2023)

Abstract: Extractive summarization aims to form a summary by directly extracting sentences from the source document. Existing works mostly formulate it as a sequence labeling problem by making individual sentence label predictions. This paper proposes DiffuSum, a novel paradigm for extractive summarization, by directly generating the desired summary sentence representations with diffusion models and extracting sentences based on sentence representation matching. In addition, DiffuSum jointly optimizes a contrastive sentence encoder with a matching loss for sentence representation alignment and a multi-class contrastive loss for representation diversity. Experimental results show that DiffuSum achieves the new state-of-the-art extractive results on CNN/DailyMail with ROUGE scores of 44.83/22.56/40.56. Experiments on the other two datasets with different summary lengths and cross-dataset evaluation also demonstrate the effectiveness of DiffuSum. The strong performance of our framework shows the great potential of adapting generative models for extractive summarization.

## introduction document summarization aims to compress text material while keeping its most salient information. it plays a critical role with the growing amount of publicly available text data. automatic text summarization approaches can be divided into two streams: abstractive and extractive summarization. although abstractive methods @xcite @xcite produce flexible and less redundant summaries, they suffer from problems of generating ungrammatical or even nonfactual contents @xcite . in contrast, extractive summarization forms a summary by directly extracting sentences from the source document. thus, the extracted summaries are grammatically accurate and faithful. we focus on extractive summarization in this work. extractive summarization is commonly formulated as a sequence labeling problem, which predicts a 0/1 label for each sentence, indicating whether the sentence should be included in summary @xcite @xcite . compared to individual sentence label prediction in the sequence labeling setting, generative models offer increased flexibility and attend to the entirety of input context. recent works have also successfully applied generative models to wide-ranging token-level sequence labeling tasks @xcite @xcite . nonetheless, how to apply generative models for sentence-level tasks like extractive summarization has not been explored. recently, continuous diffusion models have achieved great success in vision and audio domains @xcite @xcite @xcite . researchers have also attempted to apply diffusion models for text generation by converting the discrete token to continuous embeddings and mapping from embedding space to words with a rounding method @xcite @xcite . however, these approaches are not applicable for sentence-level tasks like summarization: (1) summarization has a relatively longer input context and larger generation length (around 3-6 sentences), while the above token-level diffusion-lm models are only applicable to short generation tasks like text simplification and question generation. their performance tends to drop by a large margin when generating longer sequences; to address the above issues, we propose a novel extractive summarization paradigm, diffusum, which generates the desired summary sentence representations with transformer-based diffusion models and extracts summaries based on sentence representation matching. instead of generating word by word, diffusum directly generates the desired continuous representations for each summary sentence and thus could process much longer text. diffusum is a summary-level framework since the transformer-based diffusion architecture generates all summary sentence representations simultaneously. moreover, diffusum incorporates a contrastive sentence encoding module with a matching loss for sentence representation alignment and a multi-class contrastive loss @xcite for representation diversity. diffusum jointly optimizes the sentence encoding module and the diffusion generation module, and extracts sentences by representation matching without any rounding step. we validate diffusum by extensive experiments on three benchmark datasets and experimental results demonstrate that diffusum achieves a comparable or even better performance than state-of-the-art systems that rely on pre-trained language models. diffusum also shows a strong adaptation ability based on cross-dataset evaluation results. we highlight our contributions in this paper as follows: ## method in this section, we introduce the detailed design of diffusum. diffusum consists of two major modules: a sentence encoding module and a diffusion module, which will be introduced in section 4.1 and section 4.2, respectively. after that, we explain how we optimize our model and conduct inference in section 4.3. the overall model architecture of diffusum is also illustarted in figure 2 . ## analysis 6.1 ablation study to understand the strong performance of diffusum, we perform an ablation study by removing model components of the sentence encoding module and show the results in ## conclusions this paper proposes a new paradigm for extractive summarization with generation augmentation. instead of sequentially labeling sentences, diffusum directly generates the desired summary sentence representations with diffusion models and extracts summary sentences based on representation matching. experimental results on three benchmark datasets prove the effectiveness of diffusum. this work is the first attempt to adapt diffusion models for summarization. future work could explore various ways of applying continuous diffusion models to both extractive and abstractive summarization.