Source Identification in Abstractive Summarization (2024)

Abstract: Neural abstractive summarization models make summaries in an end-to-end manner, and little is known about how the source information is actually converted into summaries. In this paper, we define input sentences that contain essential information in the generated summary as source sentences and study how abstractive summaries are made by analyzing the source sentences. To this end, we annotate source sentences for reference summaries and system summaries generated by PEGASUS on document-summary pairs sampled from the CNN/DailyMail and XSum datasets. We also formulate automatic source sentence detection and compare multiple methods to establish a strong baseline for the task. Experimental results show that the perplexity-based method performs well in highly abstractive settings, while similarity-based methods perform robustly in relatively extractive settings.

## introduction text summarization research has enjoyed recent advances in neural networks and pre-trained language models, which make abstractive summarization the most common approach @xcite @xcite . while continuing efforts in improving factuality and faithfulness @xcite have been made, abstractive summarization models, when trained properly, can create concise and coherent summaries from source documents. different from extractive summaries, for which we know the source information, it is not clear how an abstractive summary gathers various pieces of information that spread over different sentences in the input document (or input documents for multidocument summarization). identifying source in-formation is essential for the explainability and interpretability of summaries. therefore, in this paper, we aim to disentangle the abstractive summarization mechanism by identifying sentences that contain essential source information described in the generated summary. existing studies use lexical similarity (e.g., rouge) and semantic similarity (e.g., bertscore) for detecting sentences in the input document @xcite to help understand what the key source information for a generated summary. another line of work analyzes cross-attention weights for abstractive summarization @xcite and data-to-text generation @xcite . however, the approach mostly focuses on lexical and semantic similarity between the generated summary and input sentences without considering which input sentences provide source information. to this end, we define input sentences that contain essential information for the generated summary as source sentences and aim to understand how abstractive summaries are composed by analyzing source sentences. we annotate source sentences for both reference summaries and system summaries generated by pegasus @xcite on the xsum and cnn/daily mail (cnn/dm) datasets, which are among the most popular summarization benchmarks in english. we also formulate the automatic source sentence detection task to verify the effectiveness of existing methods (i.e., attention-based and similarity-based) for detecting source sentences. we develop a simpleyet-effective method based on perplexity gain-the difference in perplexity between the original text and the text after a specific sentence has been removed. we show that it significantly outperforms the existing methods in abstractive settings. the contributions of the paper are as follows: ## sourcesum in this paper, we used xsum foot_1 @xcite and cnn/dm foot_2 @xcite as the source datasets, as (1) they are the most common summarization benchmarks and (2) they have different levels of abstractiveness @xcite , to make the benchmark comprehensive and robust. ## source sentence detection problem formulation given an input document x, which consists of n sentences (s 1 , . . . , s n ), and a system summary y generated by a summarization model θ, the task is to identify a proper subset of input sentences d ′ that are essential to creating y . the task can be cast as a sentencescoring problem, where the score of each input sentence r(s), assuming the threshold value d to be a hyperparameter (i.e., d ′ = {@xmath0. ## evaluation evaluation metrics to make the evaluation independent of the choice of threshold selection, we used ranking metrics for evaluation, namely ndcg and map @xcite . for ndcg, we used the total votes as the score to consider sentences with more votes more important. for map calculation, we binarized annotations and considered source sentences if two annotators agree it is relevant. @xcite .8475 .6740 .8523 .6756 .9110 .8484 .8984 .8087 simcse @xcite .8579 .7016 .8661 .7093 .9141 .8469 .9048 .8169 pmi @xcite .8193 .6316 .8329 .6480 .8069 .6919 .7353 .5592 gpt-3.5 @xcite . ## analysis are summaries reconstructable? as reference summaries for the xsum (cnn/dm) dataset were scraped from the introductory sentence (the summary bullets), it is not ensured that reference summaries can be created only from the input documents. the same thing can be said for summaries generated by abstractive summarization models, which may hallucinate content. to analyze this, we annotated document-summary pairs with respect to the reconstructability ( §2.1). are non-source sentences unnecessary? we have defined source sentences from which summaries can be made. a natural question is whether the other "non-source" sentences are necessary for generating the same abstractive summaries. to answer the question, we evaluated the quality of summaries by pegasus, bart @xcite , and lexrank under two settings: (1) all sentences and (2) source sentence only. do different methods detect different source sentences? to analyze this, we calculated correlation coefficients of scores calculated by the different methods. figure 3 shows that the scores of the similaritybased methods are highly correlated while perplexity gain and cross attention detect source sentences differently. it is hard to interpret how commonly used transformer-based summarization models generate abstractive summaries. @xcite developed an ablation-attribution framework that identifies the generation model by comparing behaviors of a language model and a summarization model. @xcite investigated the interpretability of multi-head attention in abstractive summarization and found that attention heads can be pruned without a significant performance drop. another line of work analyzes how multiple sentences are fused into summary sentences @xcite . @xcite created a dataset that contains fine-grained pointof-correspondence between a summary and two source sentences. our work covers beyond the scope of their work as sourcesum assigns source sentence labels to all source sentences on both generated and human summaries. one simple-yet-effective approach for explainability is to highlight sentences similar to the generated summary. @xcite and @xcite use rouge and bertscore to capture the lexical and semantic similarity to help the user understand the source information of the generated summary. juraska and walker (2021) use crossattention to understand the behavior of the data-totext model. @xcite develops a hybrid summarization model that takes into account sentence similarity to improve explainability and faithfulness. @xcite develops a framework that uses neural modules to construct a tree representation to understand the relationship between a human-written summary and the input document. this paper is aligned with the line of work but rather focuses on formulating the source sentence detection task and creating a benchmark, so we can evaluate and compare different methods quantitatively and qualitatively. ## conclusion in this paper, we formulate the source sentence detection task, which finds input sentences that are essential to generating the given abstract summary, to study how abstractive summaries are made. we annotated source sentences for reference summaries and system summaries generated by pegasus on xsum and cnn/dm and created a benchmark sourcesum. experimental results on sourcesum show that perplexity gain, which is based on the perplexity increase when the target sentence is removed, performs the best in highly abstractive settings (xsum), while similarity-based methods perform robustly in extractive settings (cnn/dm).