What Have We Achieved on Text Summarization? (2020)

Abstract: Deep learning has led to significant improvement in text summarization with various methods investigated and improved ROUGE scores reported over the years. However, gaps still exist between summaries produced by automatic summarizers and human professionals. Aiming to gain more understanding of summarization systems with respect to their strengths and limits on a fine-grained syntactic and semantic level, we consult the Multidimensional Quality Metric (MQM) and quantify 8 major sources of errors on 10 representative summarization models manually. Primarily, we find that 1) under similar settings, extractive summarizers are in general better than their abstractive counterparts thanks to strength in faithfulness and factual-consistency; 2) milestone techniques such as copy, coverage and hybrid extractive/abstractive methods do bring specific improvements but also demonstrate limitations; 3) pre-training techniques, and in particular sequence-to-sequence pre-training, are highly effective for improving text summarization, with BART giving the best results.

## introduction automatic text summarization has received constant research attention due to its practical importance. existing methods can be categorized into extractive @xcite @xcite and abstractive @xcite @xcite methods, with the former directly selecting phrases and sentences from the original text as summaries, and the latter synthesizing an abridgment by using vocabulary words. thanks to the resurgence of deep learning, neural architectures have been investigated for both extractive @xcite and abstractive @xcite @xcite summarization systems. although improved rouge scores have been reported on standard benchmarks such as gigaword @xcite , nyt @xcite and cnn/dm @xcite over the years, it is commonly accepted that the quality of machine-generated summaries still falls far behind human written ones. as a part of the reason, rouge has been shown insufficient as a precise indicator on summarization quality evaluation @xcite . in the research literature, human evaluation has been conducted as a complement @xcite . however, human evaluation reports that accompany rouge scores are limited in scope and coverage. on a fine-grained level, it still remains uncertain what we have achieved overall and what fundamental changes each milestone technique has brought. we aim to address the above issues by quantifying the primary sources of errors over representative models. in particular, following mqm (mariana, 2014), we design 8 metrics on the accuracy and fluency aspects. models are analyzed by the overall error counts on a test set according to each metric, and therefore our evaluation can be more informative and objective compared with existing manual evaluation reports. we call this set of metrics polytope. using polytope, we manually evaluate 10 text summarizers including lead-3, textrank @xcite , sequenceto-sequence with attention @xcite , summarunner @xcite , point-generator @xcite , point-generator-with-coverage @xcite , bottom-up @xcite , bertsumext @xcite , bertsumextabs @xcite and bart @xcite , through which we compare neural structures with traditional preneural ones, and abstractive models with their extractive counterparts, discussing the effectiveness of frequently-used techniques in summarization systems. empirically, we find that: we release the test set, which includes 10 system outputs and their manually-labeled errors based on polytope, and a user-friendly evaluation toolkit to help future research both on evaluation methods and automatic summarization systems foot_0 . abstractive summarization @xcite presented a cut-paste based abstractive summarizer, which edited and merged extracted snippets into coherent sentences. @xcite proposed a sequence-to-sequence architecture for abstractive summarization. subsequently, transformer was used and outperformed traditional abstractive summarizer by rouge scores @xcite . techniques such as amr parsing @xcite , copy @xcite , coverage @xcite , smoothing @xcite and pre-training @xcite were also examined to enhance summarization. hybrid abstractive and extractive methods adopt a two-step approach including content selection and text generation @xcite @xcite , achieving higher performance than end-to-end models in rouge. ## models we re-implement and evaluate 10 representative and influential methods. their publicly reported rouge f1 scores are illustrated in ## evaluation method we analyze system performance by using rouge @xcite for automatic scoring and polytope for human scoring. rouge has been adopted by most work on summarization. it is a recall-based metric calculating lexical overlap between system output and human summaries. particularly, rouge-1 is based on unigram overlaps, rouge-2 on bigrams and rouge-l on longest common subsequences. polytope is an error-oriented fine-grained human evaluation method based on multidimensional quality metric (mqm) @xcite . in particular, it consists of 8 issue types (section 4.1), 8 syntactic labels (section 4.2) and a set of severity rules (section 4.3) to locate errors and to automatically calculate an overall score for the tested document. as illustrated in figure 3 , compared with rouge, polytope is more fine-grained in offering detailed and diagnostic aspects of overall quality. we develop an operating interface for annotation, which is shown in appendix a.1. particularly, a human annotator is presented the original text and an output summary in juxtaposition, and is asked to select segments that are deemed incorrect after reading. upon a preliminary selection, he is asked to make a further selection among 8 issue types and 8 syntactic labels, respectively. an embedded severity score is then generated automatically for every incorrect segment, and the quality score is calculated for the annotated summary as: @xmath0, where @xmath1,major,critical}, indicating the error count for each severity. severity scores are deducted for errors of different severity, with the deduction ratio being set as 1:5:10 for minor, major and critical, respectively. word count is the total number of words in samples. for a skilled annotator, it takes 2.5-4 minutes averagely to complete annotation of one sample, of which 2-3 minutes are used for extensive reading and 0.5-1 minutes for annotation. after polytope evaluation, 3-dimensional error points show the overall quality of the tested model (figure 1 ). the inter-annotator agreement over 20 documents is 0.8621 in terms of pearson correlation coefficient, which shows that polytope can significantly reduce subjective bias of annotators. more human annotation details are illustrated in appendix b. ## evaluating model performance we evaluate the aforementioned 10 models using the above two metrics, focusing on comparisons between pre-neural and neural methods, extractive and abstractive methods, and better understanding the effects of milestone techniques such as copy, coverage, pre-training and hybrid abstractive/extractive models. we randomly sample 150 trials from the non-anonymized cnn/dm dataset @xcite . cases supporting observations below are illustrated in appendix c. ## analysis of evaluation methods the main goal of this paper is to investigate the differences between summarization systems, rather than to promote a human evaluation metric. nonetheless, our dataset gives us a testbed to calculate the correlation between automatic and human evaluation methods. in this section, we report a contrast between rouge and polytope quantitatively, and between polytope and other human evaluation metrics qualitatively to demonstrate why we used polytope for our research goal. first, research has shown that rouge is inconsistent with human evaluation for summary quality @xcite @xcite . we evaluate rouge using polytope from the perspective of both instance-level and system-level performances. on the instance level, the individual 1500 outputs from the 10 models are adopted to calculate the pearson correlation coefficients between rouge and polytope. additionally, we select test instances that only make accuracy or fluency errors to better understand the correlation between rouge and accuracy/fluency aspects. on the system level, the overall scores of each model are adopted to calculate the pearson correlation coefficients between rouge and polytope. second, figure 3 shows results of two models on one test document by rouge, pyramid, ranking, scaling, qa and polytope evaluation metrics. as can be seen from the figure, polytope offers more fine-grained information in quality evaluation. @xcite warned that human evaluation prefers to give higher scores to longer and more informative summaries. under the setting of polytope, there was relatively little influence from the sentence length. taking bertsumext and bertsumextabs models as examples, the pearson correlation coefficients between length of their out-puts and the corresponding scores is 0.25 and 0.27, respectively, suggesting that polytope is more objective and meaningful for current models that produce summaries without pre-specified length. finally, we also evaluate the reference summaries of our 150 test trials by means of polytope, obtaining a general score of 96.41, with 63 errors in the accuracy aspect and 0 errors in the fluency aspect. gold summaries did not receive full marks in the polytope evaluation, mainly because of hallucinating content. for example, a news article describes an event as happening "on wednesday" in a summary although the original document has "on april 1". the human summary requires external knowledge beyond the document and thus suffers penalization. another common hallucination involves rhetorical but irrelevant sentences, e.g., "click here for more news". in addition, there are a few grammatical issues that affect the accuracy. for example, in "piglet was born in china with only two front legs has learned to walk.", there is a missing conjunction between two verb phrases. ## conclusion we empirically compared 10 representative text summarizers using a fine-grained set of human evaluation metrics designed according to mqm for human writing, aiming to achieve a better understanding on neural text summarization systems and the effect of milestone techniques investigated recently. our observations suggest that extractive summarizers generally outperform abstractive summarizers by human evaluation, and more details are also found about the unique advantages gained by copy, coverage, hybrid and especially pre-training technologies. the overall conclusions are largely in line with existing research, while we provide more details in an error diagnostics aspect.