Event-Keyed Summarization (2024)

Abstract: We introduce *event-keyed summarization* (EKS), a novel task that marries traditional summarization and document-level event extraction, with the goal of generating a contextualized summary for a specific event, given a document and an extracted event structure. We introduce a dataset for this task, MUCSUM, consisting of summaries of all events in the classic MUC-4 dataset, along with a set of baselines that comprises both pretrained LM standards in the summarization literature, as well as larger frontier models. We show that ablations that reduce EKS to traditional summarization or structure-to-text yield inferior summaries of target events and that MUCSUM is a robust benchmark for this task. Lastly, we conduct a human evaluation of both reference and model summaries, and provide some detailed analysis of the results.

## introduction traditional event extraction (ee) aims to produce structured event representations from unstructured text. as early as the message understanding conferences of the 1990s @xcite , the motivation for ee was fundamentally human-centric: a desire for adaptive systems that could "respond to a user's information need" @xcite . yet, the majority of ee research focuses intensively on improving metrics on major benchmarks without due consideration for how (or whether) those improvements translate into better results for users. arguably, the most human-centric way to convey information about complex events is with summaries. prominent summarization datasets such as cnn/daily mail @xcite , xsum @xcite , and gigaword @xcite rely on some conception of what is most event document …this situation, which has been blamed on the "nervousness" the country is experiencing, took place only a few hours after a man dressed in a camouflage military uniform killed 1 person and wounded 13 others with a machinegun at medellín's josé maría cordóva international airport. the killer, whose possible ties with drugtrafficking organizations are being investigated by the authorities, fired indiscriminately on people waiting in line for their tickets to bogotá, according to witnesses. ## task definition we define an event ontology as a tuple ⟨e, r, s⟩, consisting of a set of event types e, a set of role types r, and an assignment s : @xmath0. we define an event as a pair ⟨e, r⟩, consisting of an event type paired with a (possibly empty) set of event triggers @xmath1: s(e) → 2 σ * . we define event-keyed summarization (eks) as the task of mapping an input document @xmath2, r⟩ to a summary @xmath3, r⟩-with relevance determined by the role set s(e). muc-4 annotates 1,700 documents concerning political conflict in latin american countries, with terrorism-focused event types @xmath4, attack, bombing, kidnapping, robbery, forced work stoppage}. all event types are associated with the same set of 24 roles-i.e. s[e] = r, and each document may be associated with zero or more events of each type. since muc-4, it has become standard to focus on a five-role subset, consisting of the individual perpetrator(s) (perpind), the organization(s) they are affiliated with (perporg), the weapons they use (weapon), victims of the incident (victim), and damaged physical infrastructure (target) @xcite @xcite @xcite . we follow this practice here, but with two additions. first, we include the stageofexecution role, which conveys whether the event actually occurred, was (unsuccessfully) attempted, or was merely threatened. second, we include the location and date roles in cases where this information can actually be extracted from the text. foot_1 these properties (time, location, and reality status) contain essential details about an event, and are necessary for a complete summary when provided. mucsum given the gold template annotations in muc-4, we (the first three authors) wrote one abstractive summary per document-template pair ⟨d, ⟨e, r⟩⟩ that aims to convey all relevant information about ⟨e, r⟩ provided in d, given the roles listed above. to reduce the burden of writing summaries for so many events, we adopted a generatethen-edit approach: we first prompted chatgpt foot_2 to produce a succinct (≤ 3-sentence) candidate summary conditioned on ⟨d, ⟨e, r⟩⟩, then manually edited the result to ensure that it contained all r-relevant information represented in d about ⟨e, r⟩. additional information was included in the summaries if it provided important context or was otherwise necessary to ground the situation being described. in the course of writing summaries, we also re-annotated the date and location ## evaluation apart from our human evaluation ( §6), we report several standard summarization metrics, including rouge-{1, 2, lcs} f 1 @xcite and bertscore f 1 @xcite . since eks summaries focus on event participants, we also report the ceaf-ree f 1 metric of @xcite , a form of argument f 1 for stringfill roles. this provides a direct measure of how well a summary recovers the arguments from the input template. we train the span extractor of @xcite to extract and type arguments of the five entity-valued roles from the mucsum summaries. we then use the extractor to extract arguments from each summary and report ceaf-ree f 1 scores relative to the gold templates. finally, recent metrics based on natural language inference (nli) entailment (→) probabilities have been shown to exhibit higher correlation with human summary quality judgments than prior metrics @xcite . letting ⟨s p , s r ⟩ denote a predicted-reference summary pair for document d, we report the following probabilities: ## analysis lastly, we consider how the summaries generated by the above models actually fare under human evaluation. we solicited summary quality judgments from three fluent english speakers on 30 randomly selected ⟨d, ⟨e, r⟩⟩ pairs from the test split. for each pair, annotators provided a single, fivepoint likert-scale quality judgment for the summary generated for that pair by each of the five models in we find that all models produce summaries that are reliably worse than the reference (@xmath5.01): the best model outputs are rated 0.33 points worse than the reference on average (bart), with some models yielding an average difference of as much as 0.51 (gpt-4). the differences among models are generally much smaller: all are less than 0.2, with gpt-4 and pegasus tending to perform worse than bart and t5 (consistent with the results in §5), though no differences are reliable (@xmath6.1). appendix c contains further discussion. ## conclusion we have introduced the task of event-keyed summarization (eks), in which the goal is to generate a summary of a specific target event described in a document, given an underlying event ontology. we have introduced a robust benchmark for eks, mucsum, and presented a suite of fine-tuned and zero-shot baseline results across a diverse array of metrics. our ablations reveal that mucsum effectively synthesizes targeted event information with its document context. lastly, our human evaluation testifies to the quality of the reference summaries, while showing that our baselines also yield summaries of reasonable quality.