Hiding in Plain Sight: Insights into Abstractive Text Summarization (2023)

Abstract: In recent years, there has been growing interest in the field of abstractive text summarization with focused contributions in relevant model architectures, datasets, and evaluation metrics. Despite notable research advances, previous works have identified certain limitations concerning the quality of datasets and the effectiveness of evaluation techniques for generated summaries. In this context, we examine these limitations further with the help of three quality measures, namely, Information Coverage, Entity Hallucination, and Summarization Complexity. As a part of this work, we investigate two widely used datasets (XSUM and CNNDM) and three existing models (BART, PEGASUS, and BRIO) and report our findings. Some key insights are: 1) Cumulative ROUGE score is an inappropriate evaluation measure since few high-scoring samples dominate the overall performance, 2) Existing summarization models have limited capability for information coverage and hallucinate to generate factual information, and 3) Compared to the model generated summaries, the reference summaries have lowest information coverage and highest entity hallucinations reiterating the need of new and better reference summaries.

## introduction abstractive text summarization (ats) is the process of compressing given textual content into short and concise form by paraphrasing or rewriting the most important information from the source. considering the high-level language understanding, reasoning, and generation capabilities required for ats, considerable improvements are reported in this field with contributions such as large-scale datasets @xcite , use of innovative techniques/architectures @xcite , and novel evaluation metrics for effective validation. recently, significant interest has been observed in examining the quality of summarization datasets @xcite , reliability article: andros townsend enjoyed silencing the critics with his wonder strike for england, saying naysayers like paul merson provided the perfect motivation for him in italy. this has been a topsy-turvy season for the 23-year-old, who has yet to reach the heights he scaled when he first burst onto the international scene. three lions manager roy hodgson has, however, kept faith with the tottenham winger -belief he paid back in quite exceptional fashion at the juventus stadium. andros townsend scores england's equaliser in their 1-1 friendly draw with italy in turin on tuesday night . townsend celebrates his strike with tottenham hotspur team-mates ryan mason (left) and kyle walker . reference summary: andros townsend scored the equaliser in england's 1-1 draw with italy . townsend tweeted to hit back at paul merson for his previous comments . townsend has been been 'desperate' to silence his critics . merson had slammed townsend for his display against man united . of evaluation metrics @xcite , architectural choices, and overall impact of these on model performance. in this paper, we re-evaluate the quality of textual content from summarization datasets and generated summaries with information coverage, entity hallucination, and summarization complexity as primary dimensions of evaluation. popular summarization datasets, xsum @xcite and cnn-dm @xcite (see among all reference-free @xcite ; gao et al., 2020) and reference-dependant @xcite metrics proposed to date, rouge @xcite is preferred owing to its ease of interpretation, usage, and comparison with other baselines even though it misses out several quality evaluation dimensions such as factuality and informativeness @xcite @xcite @xcite . in this paper, we examine two widely used datasets (xsum and cnn-dm) and analyze the performance of three ats models (bart @xcite , pegasus @xcite , and brio @xcite ) on three interpretable quality evaluation dimensions. in contrast to the similar existing works where the human-based evaluation with a very small subset of datasets is considered @xcite , we present a computational framework for these dimensions. we believe that the framework is especially useful in reducing the dependence on humanbased evaluation for the quality of the datasets and ats models. ## quality evaluation dimensions in this section, we define three dimensions for quality evaluation. we examine the performance of ats models over these dimensions. we report the rouge-based performance of these models for comparison (see ## analysis in this section, we discuss the insights from each of these dimensions. in all our analyses, we divide the samples in the test set of both datasets into four groups. each group contains 25% samples from the original test set sorted based on the rouge-l score. group 1 (g1) contains samples with the lowest rouge-l score whereas group 4 (g4) contains samples with the highest rouge-l score. while reporting the results for the reference summary, we use the groups identified using the rouge-l ranking of samples with the brio model. we report the average scores for each group (see tables 3 and 4 for information coverage, bart gives tough competition to brio. although brio gets the highest t c and kic score on the cnn-dm dataset, bart performs competitively. on the xsum dataset, both models perform equally well. pegasus has the worst t c among all three models suggesting that the generated summaries with pegasus are limited in their capability to cover the overall source document. we need new reference summaries! it is interesting to note that the reference summaries show worst kic than all three models suggesting that the ats model's capability to cover key information is limited due to training on these poor-quality reference summaries. also, the topical coverage of reference summaries is significantly lower in g1 compared to other groups in both datasets, denoting the need for targeted analysis for this group. models trained on the xsum dataset tend to show higher entity hallucination. eh is more prominent in the models trained on the xsum dataset due to the inherent nature of the dataset (i.e., very high eh score of reference summaries), which calls for the need to look beyond word overlap-based metrics like rouge while training and evaluating the ats models. also, the high eh of reference summaries in both datasets is concerning since it directly limits the capability of proposed techniques for ats. the rouge-based bench-marking of ats models is inadequate. in addition to giving tough competition to brio for information coverage, bart consistently shows the least eh than the other two models. pegasus and brio have a similar degree of eh on both datasets (see classwise eh distribution in appendix). low information coverage and high eh of pegasus compared to bart contradicts pegasus's superior behavior based on the rouge score (see the tendency to be more abstractive is costly for brio! brio-generated summaries are more abstractive in nature, especially in the low rougescoring group g1. the significantly lower p o ref score in this group compared to other groups results in a lower rouge score suggesting that the abstractiveness proves costly for brio. rouge score is dominated by a few samples. for both the datasets, all models show a sharp increase in the p o ref score as we move from g1 to g4 (see xsum and cnn-dm datasets are not the benchmark datasets for the ats task. as discussed earlier, the reference summaries in the cnn-dm dataset are more extractive in nature. it is inter- ## conclusion in this paper, we document our experiments on two widely used ats datasets and three models trained on these datasets. we evaluate these on three dimensions of quality and demonstrate how the reported progress made in terms of the rouge metric is inconclusive. our analysis shows that bart still shows competing behavior with current stateof-the-art models on various quality dimensions. we also highlight the need to carefully analyze the reference summaries in both datasets. alternate evaluation metrics are required to account for different quality dimensions such as summarization complexity. topic 1: the security forces are reported to have used tear gas against stone-throwing protesters. topic 2: they also surrounded the hometown of burhan wani, 22, who was killed fighting indian troops last year. separately seven people are reported to have been killed in shelling across the line of control that divides indian and pakistani-administered kashmir. officials on the pakistani side told reuters that five people died in indian shelling, while indian officials say two people were killed by pakistani fire. topic 3: there has been an armed revolt in the muslim-majority region against rule by india since 1989, although violence has waned in recent years. the disputed region is claimed by both india and pakistan in its entirety. india blames pakistan for fuelling the unrest, a claim denied by islamabad. topic 4: burhan wani is credited with reviving the image of militancy in muslim-majority indian-administered kashmir, becoming a figurehead for young people. saturday's violence started as people tried to walk to his home in tral -where he died in a shootout with the army last july. his death led to a wave of protests during which dozens of people were killed. topic 5: the indian authorities imposed heavy restrictions in the kashmir valley for the anniversary, stopping internet access and sealing off tral. there have also been reports of army personnel being injured in a militant attack overnight on friday.