Are Abstractive Summarization Models truly ‘Abstractive’? An Empirical Study to Compare the two Forms of Summarization (2022)

Abstract: Automatic Text Summarization has seen a large paradigm shift from extractive methods to abstractive (or generation-based) methods in the last few years. This can be attributed to the availability of large autoregressive language models that have been shown to outperform extractive methods. In this work, we revisit extractive methods and study their performance against state of the art(SOTA) abstractive models. Through extensive studies, we notice that abstractive methods are not yet completely abstractive in their generated summaries. In addition to this finding, we propose an evaluation metric that could benefit the summarization research community to measure the degree of abstractiveness of a summary in comparison to their extractive counterparts. To confirm the generalizability of our findings, we conduct experiments on two summarization datasets using five powerful techniques in extractive and abstractive summarization and study their levels of abstraction.

## introduction the amount of data on the internet has been growing exponentially creating excessive information for users to consume. automatic text summarization alleviates this issue of information overload by producing shorter and concise summaries that capture the essence of the long source text. automatic text summarization can be broadly classified into kinds-extractive summarization and abstractive summarization. extractive summarization identifies important excerpts from the source document to produce summaries. these excerpts are composed of sentences and phrases which the model deems most appropriate for summarizing the source document. with the advent of sophisticated language models trained on large amounts of data, most of the recent work in summarization has drifted towards abstractive summarization. summarization tasks have now become one of the benchmarks to beat with many of the sota generation models. in abstractive summarization, natural language generation techniques are employed to generate a summary. humans write concise summaries by introducing novel words and only using information from the source text that they deem absolutely necessary. since abstractive summarization models generate words which are not necessarily present in the source text as is, the expectation is that the summaries generated would be truly abstractive and hence closer to human generated summaries. in theory, abstractive summarization models should outperform extractive methods as they have the ability to generate free form text like humans. in practice, however, we find that the abstractive summaries of sota models today are closer to extractive summaries than to the human generated summaries against which they are compared. in order to analyze the extent of these issues in abstractive summarization models, we chose three sota abstractive summarization models trained on cnn dailymail @xcite and the x-sum @xcite datasets (section 3). we also chose two sota extractive summarization ## related work both extractive and abstractive summarization techniques have been well studied in the natural language processing (nlp) community. earlier work in extractive summarization relied on clues such as position of sentences and frequency of words while extracting most important snippets for summaries @xcite . more recently, neural network-based extractive summarization have gained more popularity @xcite @xcite @xcite @xcite . there has also been significant work in abstractive summarization @xcite @xcite . most of the recent approaches use encoder-decoder architectures to generate summaries @xcite @xcite @xcite . these methods produce summaries using words that are not present in the source text and hence these methods have gained more popularity over the last few years. transformer models with selfsupervised training @xcite @xcite @xcite have shown to perform well on language learning when fine-tuned on various nlp tasks. more recently, bart @xcite , pegasus @xcite and t5 @xcite have shown state-of-the-art performance in abstractive summarization, so we analyze the generated summaries of these models in comparison with sota methods in extractive summarization @xcite . previous work on measuring the abstractiveness and extractiveness in summaries has been restricted to measuring diversity of n-grams @xcite but we move to a more semantic based metric to measure these aspects of a summary. ## datasets we use two standard summarization benchmarks: cnn-dailymail (2017) : a corpus of news articles and human generated summaries. we use the entire test set of size 11, 490. x-sum (2018) : corpus of news articles and the task is to predict the first sentence given the remaining article content. we again use the entire test set of 11, 334 samples from this dataset. ## models we study bart @xcite , pegasus and t-5 @xcite for abstractive summarization. for extractive summarization we use, bertsumext @xcite and matchsum @xcite . we use the versions fine-tuned on the datasets described in section 3 for experiments. these models are in the top 10 on the leaderboard at the time of writing this paper foot_0 . these approaches are described below: bart: an encoder decoder model that uses a bidirectional encoder and an autoregressive model as its decoder. the model is trained using de-noising objective. pegasus: encoder decoder model that is pretrained to encourage abstractive summarization. the model is trained to predict masked out sentences in the source text. t-5: text-based language problems are handled in a unified text to text framework to obtain significant improvements on several benchmark tasks. bertsumext: model learns to pick the top 3 sentences that seem most relevant from the source text using bert and inter-sentence transformer layers. matchsum: model uses semantic matching between the document and the candidate summaries to pick the summary that is closest to the document. they achieve this using a margin-based triplet loss. ## experiments and results the summarization community has looked at using text overlap metrics like, bleu or meteor. the widely used text overlap metric for summarization is rouge-l that measures the overlap of words using longest common subsequence between the generated summary and the ground truth. while these metrics attempt to compare the ground truth with the generated sentences, they certainly do not measure the abstractive component that one expects from abstractive summarization models. we describe experiments that show the model is merely copying words from the source text and not introducing novel words present in the ground truth. in order to allow future models to measure this abstractiveness we introduce a metric called absextscore that will allow us to measure the abstractive prowess of these summarization models. ## conclusion and future work a central tenet of abstractive models is to abstract relevant information from source text. we find that the abstractive models merely copy words from source text and are failing to insert novelty words. we show that the abstractive summaries are closer to the extractive summaries than they are to ground truth. we use models built for semantic understanding to introduce a new metric called absextscore which the summarization community can adopt to understand the level of abstraction introduced by their proposed abstractive models in comparison to their extractive counterparts. we conducted a human subject study to show the correlation between the automated metric and human judgements.