Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models (2024)

Abstract: Cross-document event coreference resolution (CDECR) involves clustering event mentions across multiple documents that refer to the same real-world events. Existing approaches utilize fine-tuning of small language models (SLMs) like BERT to address the compatibility among the contexts of event mentions. However, due to the complexity and diversity of contexts, these models are prone to learning simple co-occurrences. Recently, large language models (LLMs) like ChatGPT have demonstrated impressive contextual understanding, yet they encounter challenges in adapting to specific information extraction (IE) tasks. In this paper, we propose a collaborative approach for CDECR, leveraging the capabilities of both a universally capable LLM and a task-specific SLM. The collaborative strategy begins with the LLM accurately and comprehensively summarizing events through prompting. Then, the SLM refines its learning of event representations based on these insights during fine-tuning. Experimental results demonstrate that our approach surpasses the performance of both the large and small language models individually, forming a complementary advantage. Across various datasets, our approach achieves state-of-the-art performance, underscoring its effectiveness in diverse scenarios.

## introduction event coreference resolution is a useful task in information extraction @xcite . this is crucial for achieving a more comprehensive understanding of intricate narratives and facilitating knowledge extraction from diverse textual sources. the coreference of events typically relies on a thorough understanding of document-level context @xcite @xcite * corresponding author (a) existing approach (b) our approach figure 1: models for cross-document event coreference resolution, where the input comprises event mentions from different documents, and the output consists of event clusters formed by coreferential mentions, which are visually represented by icons sharing the same color and shape. @xcite . cross-document event coreference @xcite , involving the comparison of event mentions from different documents, presents additional challenges. on one hand, distinct events in different documents may be portrayed in a very similar manner, especially for events of the same type (challenge 1). on the other hand, the portrayal of the identical event may vary significantly across different documents (challenge 2). the model is required to grasp comparable coreference evidence from varied contexts and make judgments based on it (refer to the examples in existing work @xcite ) attempts to address cdecr based on fine-tuning small language models (slms) foot_0 , as shown in figure 1a . however, the complexity and diversity of the context make it prone to learning pseudofeatures by capturing simple co-occurrences rather than genuinely coreference-related terms, including contextual words, entity mentions and other event mentions associated with the given event mention. supporting this observation, cdecr remains a significant challenge for slms, as evidenced by achieving only around 70% conll f1 score on the fcc dataset @xcite . recent advancements in llms have significantly advanced the field of nlp, enabling the effective resolution for tasks like machine translation @xcite ) and text summarization @xcite , with just a few demonstrations. however, when it comes to information extraction (ie) tasks, llms encounter challenges in task-specific adaptation. specifically, llms struggle to achieve the same level of accuracy as supervised slms because a small number of demonstrations cannot comprehensively cover the complex annotation guidelines of these tasks @xcite . moreover, the inherent nature of the cdecr task, which involves processing multiple documents, imposes enhanced demands on understanding the lengthy context in the demonstrations. 2 instead of directly predicting cdecr structures, the relative strength of llms can enhance the generic understanding of individual documents, particularly the inherent meaning of diverse event mentions, which is complementary to the advantage of slms in understanding structures with thorough fine-tuning. to leverage the relative strengths of llms and slms, we propose a collaborative approach, as shown in figure 1b . first, we use the llm to summarize event mentions from different documents. then we feed these insights to the slm to enhance its understanding of event mentions, enabling it to make coreference judgments based on more focused contexts. for the llm summarization, we design a two-step workflow with separate generic prompts to guide its comprehension of the context of each mention, instead of task-specific in-context learning or fine-tuning. for the slm, we employ joint representation learning to integrate the origi-nal document and the generated summary. we conduct experiments on three datasets of cdecr, and the results demonstrate that our collaborative approach, as compared to methods solely relying on the llm or slm, exhibits significant improvements. across all three datasets (ecb+, gvc, and fcc), our approach achieves state-ofthe-art results, with increases of 1%, 2.7%, and 7% in conll f1, respectively (averaged over three independent experiments for each dataset). through analysis, it is demonstrated that our approach more thoroughly addresses the aforementioned challenge 1 of similarly portrayed contexts, making a substantial contribution to performance improvement. to the best of our knowledge, we are the first to propose a collaborative approach that leverages the universal capabilities of llms to address cdecr, achieving superior performance compared to the state-of-the-art baseline. 3 2 related work cdecr early work addresses cdecr by employing machine learning methods with manually designed features @xcite @xcite . recent neural approaches have utilized slms to encode event mentions, obtaining their embeddings for supervised coreference resolution. initial efforts involve encoding at sentence level and fusing the embeddings of mentions and the incomplete arguments extracted by srl as the representation of mentions @xcite @xcite . subsequent work incorporates extensive context directly into encoding, leading to noticeable improvements @xcite @xcite @xcite . more recently, @xcite and @xcite establish connections between event mentions using a discourse rhetorical structure constructor and a gpt-3 model fine-tuned with additional data for temporal reasoning, respectively. in comparison to existing work, we are the first to establish comprehensive connections between event mentions and their corresponding contextual elements, including contextual words, entity mentions, and other event mentions, by leveraging the intrinsic knowledge and out-ofthe-box context comprehension ability of llms. llm for ie several recent studies @xcite @xcite @xcite @xcite @xcite @xcite have evaluated the performance of llms, predominantly chatgpt, using in-context learning methods on various ie tasks. these investigations universally demonstrate that llms exhibit commendable performance in zero-shot and few-shot settings, yet there remains a substantial gap when compared to state-of-the-art supervised slms, with the performance gap widening for more complex tasks. in addition, there are also methods directly using labeled data from ie tasks to fine-tune llms @xcite @xcite . in general, training on these larger-scale models, such as code-llama and flan-t5, has yielded results comparable to supervised baselines and demonstrated improvements in zero-shot settings. however, when the training of llms does not result in significant performance gains, the training cost, compared to training slms, becomes less cost-effective. integration of llm and slm the integration of llm and slm is an emerging approach, with only a few explorations in complex ie tasks. @xcite prompts the llm to rerank a few difficult samples filtered by the supervised slm and achieves improvements on various few-shot ie tasks. their method is based on the observation that llms excel only at a small number of hard samples. @xcite first utilizes the llm to generate reasoning logic for demonstrations retrieved by a fine-tuned slm, then feeds this combined input back to the llm for relation extraction, surpassing supervised baselines on some datasets. an inherent challenge lies in finding reasonable demonstrations for null-type triples, leading to poor performance on complex tasks such as ace05. additionally, inducing complex reasoning logic for each of the k-demonstrations is costly, leading them to sample only a subset of ace05 and tacred test sets. @xcite and @xcite leverage llms for data enhancement in sentence and document-level relation extraction tasks, respectively. the gap between triples recognized by llms and those annotated under manually crafted rules introduces potential shifts in data distribution, making the effectiveness in applications unclear. overall, the aforementioned integration methods have exhaustively attempted to adapt llms to spe-cific tasks by prompting them to establish accurate connections with artificially defined labels. in contrast, our approach only requires llms to perform generic tasks, leveraging their inherent capabilities to assist specific tasks. concurrently, akin to our approach, ding et al. 2024) and nath et al. (2024 2024 ) employs similar task prompts to guide llm in generating coreference reasoning processes for mention pairs. unlike our approach, which involves a general task of having llm process each mention individually, their methods require llm to directly handle the relationships between mentions given coreference labels. in terms of efficiency, their methods are less effective than ours as they need to handle combinations of mention pairs, resulting in a quadratic increase in the number of processing entries. ## method we adopt the state-of-the-art method proposed by @xcite as our baseline (section 3.1), then summarize events using generic prompts for llm (section 3.2), and finally integrate the representations of events from both the summary and the original context into baseline system (section 3.3). ## conclusion we design generic tasks to leverage the potential of llms for cdecr, effectively bridging the gap between the general capabilities of llms and the complex annotation guidelines of specific ie tasks. results show that by harnessing the inherent knowledge and comprehension abilities of llms to gain a deeper understanding of events, our collaborative approach can alleviate the challenge of slms for complex contextual understanding, ultimately enhancing performance.