Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasing (2024)

Abstract: The most popular Cross-Document Event Coreference Resolution (CDEC) datasets fail to convey the true difficulty of the task, due to the lack of lexical diversity between coreferring event triggers (words or phrases that refer to an event). Furthermore, there is a dearth of event datasets for figurative language, limiting a crucial avenue of research in event comprehension. We address these two issues by introducing ECB+META, a lexically rich variant of Event Coref Bank Plus (ECB+) for CDEC on symbolic and metaphoric language. We use ChatGPT as a tool for the metaphoric transformation of sentences in the documents of ECB+, then tag the original event triggers in the transformed sentences in a semi-automated manner. In this way, we avoid the re-annotation of expensive coreference links. We present results that show existing methods that work well on ECB+ struggle with ECB+META, thereby paving the way for CDEC research on a much more challenging dataset. Code/data: https://github.com/ahmeshaf/llms_coref

## introduction cross-document event coreference resolution (cdec) involves identifying mentions of the same event within and across documents. an issue with cdec is that the widely used dataset, event coref bank plus (ecb+; @xcite ), is biased towards lexical similarities, both for triggers and associated event arguments, and therefore has a very strong baseline @xcite @xcite . to see this, consider the excerpts from ecb+ shown in figure 1 @xcite . this consists of three killing events selected from separate articles sharing a common trigger. an algorithm capable of matching the triggers and tokens within the sentences, such as "vancouver" and "office," can readily discern that event 2 is coreferent with event 3, and not event 1. this leads to the question of whether the state-of-the-art methods using this corpus @xcite learn the semantics of event coreference, or are merely exploiting surface triggers. figurative language, encompassing metaphors, similes, idioms, and other non-literal expressions, is an effective tool for assessing comprehension across cognitive, linguistic, and social dimensions @xcite @xcite @xcite . figurative language, by its nature, draws on a wide array of cultural, contextual, and imaginative resources to convey meanings in nuanced and often novel ways. consequently, it employs a broader vocabulary and more unique word combinations than literal language @xcite . most recent work on metaphors has been focused on generation @xcite @xcite , interpretation @xcite , and detection @xcite @xcite . yet, there is a dearth of event datasets for figurative language which limits an important research direction of event comprehension. in this paper, we address these two challenges by leveraging gpt-4 in constrained metaphoric paraphrasing of ecb+documents. we introduce a novel dataset named ecb+meta , which we generate using a semi-automatic approach. this involves applying metaphoric transformations to the event triggers within ecb+ and then hand-correcting the tagged triggers in the new corpus. as depicted in figure 1 (b), the trigger word killing in events 2 and 3 of ecb+ become slaying and snuffing out the flame of life of in ecb+meta, respectively. this approach preserves the coreference annotations from ecb+, thereby avoiding an expensive coreference re-annotation task. thus, we create several versions of "tougher" cdec benchmark datasets with enhanced lexical diversity with varying levels of metaphoricity. we present baseline 2018 )), for instance, was introduced as a way of adding ambiguity to the task. yet, both these datasets lack lexical diversity in terms of coreferent event triggers. @xcite is one such work that addresses the diversity question through crossdomain coreference, however, a dataset focusing cdec on figurative language does not exist to our best knowledge. even with the use of modern annotation tools @xcite , annotating cdec datasets is expensive. works such as @xcite ; @xcite use wikipedia as a way of bootstrapping ecr annotations automatically. in a similar vein, we bootstrap cdec annotations for figurative language in a synthetic way using gpt-4. ## methodology we first synthetically create ecb+meta by employing metaphoric paraphrasing of the original corpus. then we tag the event triggers of the original corpus in ecb+meta in a semi-automated manner. finally, we adopt two existing cdec methods to test this new dataset. we describe each of these steps: ## future work future research could explore applying more recent cdec techniques on ecb+meta. these techniques could include symbolic grounding, as discussed in @xcite , and event type categorical cross-encoding, as proposed by @xcite . another outcome of this research is to use cdec as a text complexity metric @xcite of a corpus. we argue that a corpus is more complex if a cdec algorithm is not able to identify that different explanations of the same event are the same. an interesting line of future work would be to automatically generate an optimally complex cdec corpus, i.e., a corpus that yields the lowest coreference score. in this work, we rely on the gpt-4's metaphor list and substitution choice. the only control we have is to make a coherent choice, however, we find ourselves subjected to the unpredictable outputs, colloquially referred to as "hallucinations", generated by gpt-4. in the future, we aim to integrate human feedback into the process of metaphor selection and to employ annotated metaphor databases from studies such as @xcite . ## conclusion in this paper, we introduced ecb+meta a lexically rich variant of ecb+ using constrained metaphoric paraphrasing of the original corpus. we provide hand-corrected event trigger annotations of two versions of ecb+meta differing in the kind of metaphoric transformation using either single words or phrases. we finally provide baseline results using existing sota methods on this dataset and show their limitations when there is substantial lexical diversity in the corpus. through the provided data and methodology, we lay a path forward for future research in cross-document event coreference resolution on more challenging datasets.