Cross-Document Event-Keyed Summarization (2025)

Abstract: Event-keyed summarization (EKS) requires summarizing a specific event described in a document given the document text and an event representation extracted from it. In this work, we extend EKS to the cross-document setting (CDEKS), in which summaries must synthesize information from accounts of the same event as given by multiple sources. We introduce **SEAMuS** (**S**ummaries of **E**vents **A**cross **Mu**ltiple **S**ources), a high-quality dataset for CDEKS based on an expert reannotation of the FAMuS dataset for cross-document argument extraction. We present a suite of baselines on SEAMuS—covering both smaller, fine-tuned models, as well as zero- and few-shot prompted LLMs—along with detailed ablations and a human evaluation study, showing SEAMuS to be a valuable benchmark for this new task.

## introduction providing useful information about events requires the ability not only to extract relevant, userspecified information from documents, but also to present that information in a readable form. drawing on this observation, @xcite recently proposed event-keyed summarization (eks), a task that entails summarizing a particular event, given a document and an event representation extracted from it. eks thus seeks to satisfy both requirements-reconciling the specific information needs of ie end users with the more generic outputs of traditional summarization models-in order to communicate precise information about a single event in a contextualized and readable form. eks can thus be viewed as event-centric controllable summarization @xcite , where the controlled attributes are the event and roles of interest. however, adequately understanding a particular event often requires synthesizing information across multiple sources-evidenced in part by the rapidly growing interest in retrieval augmented generation (rag; @xcite . accordingly, this work extends eks to the cross-document setting (cdeks), drawing on-and enhancing-the famus dataset for cross-document argument extraction (cdae) to do so @xcite . we summarize our contributions as follows: event-centric summarization in introducing eks, @xcite released mucsum, an eks dataset based on the classic muc-4 template filling dataset @xcite . mucsum contains abstractive event-keyed summaries for each event template in muc-4, written so as to faithfully express the role of each template argument, plus any minimal additional context required for the summary to act as a standalone account of the event. @xcite baselines on mucsum, and also conduct a human evaluation of model outputs, which inspires our own ( §5). other event-centric summarization research has focused on timeline summarization (tls), which constructs chronological lists of events, often with timestamps and usually based on multiple documents @xcite @xcite . @xcite use extracted event-related keywords to condition singledocument summarization, and integrate an eventoriented attention mechanism into bart to encourage models to cover all events discussed. additionally, @xcite introduce the poque dataset, which has annotations that characterize the subevent structure of complex events in stories and the changes undergone by their participants. among these annotations are process summaries, which give high-level descriptions of a complex event, and change summaries, which describe the changes experienced by a participant as a result. multi-document summarization cdeks is an event-centric multi-document summarization (mds) task. work on mds has pursued a variety of goals, including synthesizing reviews @xcite , summarizing dialogues @xcite , distilling news articles (notably, via duc foot_2 and tac foot_3 ), and generating reports @xcite . event-centric mds datasets include multinews @xcite and di-versesumm @xcite , which focus on new stories, but seamus is most similar to auto-hmds @xcite and wcep @xcite in being built on wikipedia articles and their sources. cdeks departs from all of these, however, in responding to an explicit information need. it is thus an event-centric form of query-oriented mds @xcite , where a query expressing the kind of information to be summarized is provided as additional input. but whereas queries from prior work are given in natural language-e.g. article titles @xcite or web searches @xcite -ours are structured event representations, drawing on the ie tradition of leveraging event ontologies to encode information needs, and enabling extraction-to-summarization pipelines. our work we summarize three key differences between prior work and our own. we focus on: ## annotation annotation of seamus was divided into two phases. in the first phase, abstractive report summaries were written for each event in famus (see §2) based only on its report document, and were then annotated for event arguments ( §3.1). in the second phase, abstractive cross-document summaries were written for each event based jointly on its report and source documents, and were then annotated for event arguments as in the first phase ( §3.2). in both phases, annotators were instructed to amend spurious, missing, or otherwise incorrect argument annotations in the report or source document before writing their summary. thus, both phases involve (1) correcting existing famus argument annotations; ## human evaluation setup lastly, we conduct a human evaluation of the reference and model-generated summaries. we focus our evaluation on the cross-document task, comparing the summaries generated by models presented in looking at intra-rater distributions, however, it's unclear how robust these preferences are. using wilcoxon rank-sum tests to evaluate pairwise differences in each rater's scores for a given pair of models, we find that some of these preferences are reliable at α = .05 (e.g. gpt-4o > t5 with @xmath0.016 for the first rater), but none holds up when applying the bonferroni correction for multiple comparisons. we take these results to indicate that our baselines are fairly effective at producing 18 see appendix d for further details. ## conclusion this work has extended the task of event-keyed summarization (eks) to the cross-document setting (cdeks). to enable this, we provided an expert reannotation of the famus cdae dataset, yielding high-quality event argument annotations on all 1,265 examples. we then leveraged these improved annotations to construct seamus-a collection of single-(report) and cross-document summaries on top of famus, further annotating the summaries themselves for event arguments ( §3). we benchmarked seamus on a diverse set of baselines, including smaller fine-tuned models, as well as zero-and few-shot prompted llms ( §4.2, §4.3). we then presented more detailed analysis, conducting a comprehensive set of input abalations ( §4.4), assessing the impact of degraded event extraction on summary quality ( §4.5), and finally concluding with a human evaluation of summary quality ( §5). we release seamus, along with our baseline results, to facilitate further work on eks in both the single-and cross-document settings.