A Set Prediction Network For Extractive Summarization (2023)

Abstract: Extractive summarization focuses on extracting salient sentences from the source document and incorporating them in the summary without changing their wording or structure. The naive approach for extractive summarization is sentence classification, which makes independent binary decisions for each sentence, resulting in the model cannot detect the dependencies between sentences in the summary. Recent approaches introduce an autoregressive decoder to detect redundancy relationship between sentences by step-by-step sentence selection, but bring train-inference gap. To address these issues, we formulate extractive summarization as a salient sentence set recognition task. To solve the sentence set recognition task, we propose a set prediction network ( SetSum ), which sets up a fixed set of learnable queries to extract the entire sentence set of the summary, while capturing the dependencies between them.Different from previous methods with an auto-regressive decoder, we employ a non-autoregressive decoder to predict the sentences within the summary in parallel during both the training and inference process, which eliminates the train-inference gap. Experimental results on both single-document and multi-document extracted summary datasets show that our approach outperforms previous state-of-the-art models.

## introduction extractive summarization is the process of extracting a brief set of sentences from the source document to cover the salient information of it. compared with abstractive summarization @xcite , extractive summarization is less likely to deviate from the source document, as well as more efficient in execution. due to these advantages, it has become widely utilized for automatic summarization tasks. recently, most of the approaches (nallapati et al., 2017; liu and lapata, 2019; xu et al., 2020) † corresponding author. formulate extractive summarization as a sequence labeling task, following the encoder-decoder framework. sentence classification @xcite is a naive solution for the task, which makes independent binary decisions for each sentence in the source document, leading to high redundancy of the summary, as shown in figure 1 (a) . to tackle the redundancy problem, @xcite ; @xcite introduce an autoregressive decoder, which select sentences step by step to construct the summary, until the length limit of the summary is reached, as shown in figure 1 (b) . trigram blocking @xcite , as a plugand-play method during inference, has the same motivation, which allows the model to consider the sentences already selected when making binary decisions for the current sentence. besides, some methods @xcite @xcite incorporate reinforcement learning to optimize the final evaluation metric with a step-by-step selection strategy. @xcite ; @xcite propose a method to construct a summary with two steps, where step one is for constructing candidates summary, step two is for selecting a summary from the candidates. however, these methods not only decrease the efficiency of training and inference but also suffer from the training-inference gap. to address the above issues, we formulate extractive summarization as a salient sentence set recognition task, which treats summary as the salient set of sentences, rather than a set of salient sentences. to solve the sentence set recognition task, we propose a set prediction network based on the encoderdecoder framework, which sets up a fixed set of learnable queries to extract the entire sentence set of the summary, while capturing the dependencies between them. as shown in figure 1 (c), different from previous approaches, we employ a nonautoregressive decoder to extract the sentence set of the summary in parallel during both training and inference. the non-autoregressive decoder receives the sentence representation and a set of learnable vectors called sentence queries to decode the final sentence set in the summary. to measure the difference between predictions and gold labels in an end-to-end manner, we employ a loss function based on bipartite matching, which can produce an optimal matching between predictions and gold labels with minimal assignment cost. compared with the autoregressive approach, our set prediction network is efficient and can eliminate the gap between training and inference. furthermore, the decoder is able to capture the dependencies between sentences through a self-attention mechanism between sentence queries and then make joint decisions on the entire sentence set. experimental results on four single-document and one multi-document extracted summary datasets show that our approach outperforms previous state-of-the-art models. our main contributions are as follows: traditional approaches @xcite @xcite formulate extractive summarization as a sequence labeling task following the encoder-decoder framework. to improve the performance of extractive summarization, some methods introduce autoregressive decoder @xcite or reinforcement learning @xcite . these methods construct summary step-by-step until the length limit of the summary is reached. in addition to the above paradigm of sequence labeling, zhong et al. (2020); an et al. (2022) formulate the extractive summarization task as a semantic text matching problem. tang et al. (2022) formulates extractive summarization as an optimal transport (ot) problem from document to summary. jia et al. (2021) selects sentences simultaneously from the source document when the predicted sentence probabilities exceed a threshold. although the above methods have made different degrees of progress in extractive summarization, they also encounter various challenges, such as insufficient decoding efficiency, training, inference gap, unstable results, etc. recently, the query-based approach is employed in the summarization task. for example, @xcite unifies that all summaries are a response to a query. @xcite use learnable queries as a control signal to control summary generation. these query-based methods focus on generating a highly relevant summary for a given query, in which queries can be observable or latent. besides, these methods restrict queries to a single sample and semantic space. in this paper, we propose a set prediction network for extractive summarization based on sample-independent queries, which uses a non-autoregressive decoder to improve the decoding efficiency and unify the training and inference processes. ument encoder §3.2, a non-autoregressive set decoder §3.3, and a bipartite matching loss §3.4. ## conclusion in this paper, we propose a set prediction network for extractive summarization task. compared with previous sequence labeling methods, our approach formulates extractive summarization as a sentence set prediction problem. in our approach, a set of sentence queries are fed into a non-autoregressive decoder, which then predicts all sentences within the summary in parallel. to measure the difference between the parallel prediction results and the gold labels, we apply a bipartite matching loss to train the model. to demonstrate the effectiveness of our approach, we conduct experiments on single-document and multi-document datasets. the experimental results demonstrate that our method outperforms the previous state-of-the-art models.