Event-based evaluation of abstractive news summarization (2025)

Abstract: An abstractive summary of a news article contains its most important information in a condensed version. The evaluation of automatically generated summaries by generative language models relies heavily on human-authored summaries as gold references, by calculating overlapping units or similarity scores. News articles report events, and ideally so should the summaries. In this work, we propose to evaluate the quality of abstractive summaries by calculating overlapping events between generated summaries, reference summaries, and the original news articles. We experiment on a richly annotated Norwegian dataset comprising both events annotations and summaries authored by expert human annotators. Our approach provides more insight into the event information contained in the summaries.

## introduction a summary of a news article provides a condensed version of its main content @xcite . one of the primary practical applications of large language models (llms) is generating concise text summaries, and many news publishers in norway have already integrated llm-generated summaries into their articles. however, assessing the quality and accuracy of these summaries remains a challenge. current evaluation metrics compare generated summaries to ideal summaries created by humans, in terms of overlapping words/units, such as rouge-l @xcite , or semantic similarity, such as bertscore @xcite . however, these metrics provide limited information on the semantic content of the summaries themselves. with the increasing usage of llms for text generation, there has been a growing number of studies on evaluating the factuality of these texts from the perspective of contained information, such as factscore @xcite . for summarization, @xcite propose to use se-mantic triplet units as a judgment of the semantic content units in generated texts, and @xcite also propose a similar protocol based on semantic units, named atomic content units. inspired by event extraction (ee), a nlp task that extracts event information from unstructured texts into structured forms @xcite , we propose to analyze the quality of news article summaries by comparing the overlapping events between generated summaries, reference summaries, and the source articles. by using structured event information, we provide more insight into both the generated summaries and humanauthored summaries. we experiment on a norwegian dataset with rich annotations both for events (eden @xcite ), and summaries (nor-summ @xcite ), and demonstrate the usefulness of the proposed event-based evaluation metric which is grounded in the overlap of identified events. ## event-overlap our proposed metric calculates the degree of overlapping events between summaries (generated and human-authored) and the source texts. first, an event extraction system is used to extract events from summaries and source articles. second, standard event extraction evaluation metrics are adapted and applied to calculate the actual event overlaps. ## experimental setup datasets we use two recently released datasets: the norwegian event detection dataset eden @xcite and the human-authored summaries of norwegian news articles dataset nor-summ @xcite . the source articles of norsumm are a subset of eden. these parallel annotations of events and summaries make it possible to evaluate our approach and contrast gold vs predicted event information on gold vs generated summaries. more concretely, we use the test set of norsumm, which contains 33 news articles, each coupled with three unique human-authored summaries. llms for automatic summarization, we evaluate a range of norwegian and nordic open-source pretrained and instruction-finetuned decoder-only llms: llama-3-8b-instruct, 2 llama-3-8b, 3 meta-llama-3-8b-instruct 4 , mistral-nemo-instruct-2407, 5 normistral-11b-warm foot_5 , and normistral-7b-warm-instruct. 7 all the llms are available via huggingface. 8 we use the same prompts as in the norsumm evaluation @xcite to generate summaries, and keep only one summary that has highest average score of rouge-l and bertscore values for each model. ## results and discussion we here present the analysis of our event-overlap metric on the test set of norsumm. we first present the event-overlap between summaries and the original articles; we then present the event-overlap between generated summaries and human-authored summaries. finally, we discuss the overall picture summarizing event-overlap scores. ## conclusion in this article, we introduce a new approach for evaluating abstractive summaries using event identification information. our proposed event-overlap metric quantifies shared events between generated summaries, human-authored summaries, and the original news articles, offering more insight into the event information of the summaries. in conjunction with standard summarization evaluation metrics, our event-overlap metric adds a valuable dimension to assessing the quality of llm generated summaries. experiments conducted on norsumm, a richly annotated norwegian dataset, demonstrate the effectiveness and practicality of our method. our approach is also easily adaptable to other datasets and languages.