MCECR : A Novel Dataset for Multilingual Cross-Document Event Coreference Resolution (2024)

Abstract: Event coreference resolution (ECR) is a critical task in information extraction of natural language processing, aiming to identify and link event mentions across multiple documents. Despite recent progress, existing datasets for ECR primarily focus on within-document event coreference and English text, lacking cross-document ECR datasets for multiple languages beyond English. To address this issue, this work presents the first multiligual dataset for cross-document ECR, called MCECR (Multilingual Cross-Document Event Coreference Resolution), that manually annotates a diverse collection of documents for event mentions and coreference in five languages, i.e., English, Spanish, Hindi, Turkish, and Ukrainian. Using sampled articles from Wikinews over various topics as the seeds, our dataset fetches related news articles from the Google search engine to increase the number of non-singleton event clusters. In total, we annotate 5,802 news articles, providing a substantial and varied dataset for multilingual ECR in both within-document and cross-document scenarios. Extensive analysis of the proposed dataset reveals the challenging nature of multilingual event coreference resolution tasks, promoting MCECR as a strong benchmark dataset for future research in this area.

## introduction event coreference resolution (ecr) is a fundamental task in information extraction that aims to identify mentions of events referring to the same realworld event. an event mention refers to a word or phrase that indicates the occurrence of an event, known as event trigger. the goal of an ecr system is to accurately identify all triggers that pertain to the same event. for instance, consider the following example: "the leaders of 8 countries with the greatest industry gathered in paris to discuss global environmental challenges" and "the convention of industrialized countries in paris to combat global warming will be impactful." in these sentences, the event mentions "gathered" and "convention" both refer to a conference related to climate change. recognizing such coreferences between event mentions within a document or across multiple documents is crucial for achieving a comprehensive understanding of events. this knowledge can be leveraged in various downstream applications, including question-answering, summarization, information retrieval, and knowledge base population. numerous methods have been proposed for event coreference resolution, encompassing a range of approaches. early research explored featurebased models, incorporating traditional feature engineering techniques @xcite @xcite . additionally, graphbased models @xcite , integer linear programming (choubey and @xcite , markov logic networks @xcite , and multi-tasking @xcite approaches have been employed. more recently, there has been a surge of interest in leveraging large language models for event coreference resolution @xcite . for instance, the authors in @xcite propose an ecr approach that encodes both sentence-level and document-level context using longformer. however, most existing work on event coreference resolution has focused on within-document scenarios, limiting its scope. while some prior studies have investigated cross-document event coreference resolution (cdecr) @xcite @xcite , comprehensive exploration of this task remains limited due to the scarcity of available resources. moreover, multilingual cross-document event coreference resolution represents an even less-explored setting. the lack of large-scale manually annotated datasets poses a significant challenge in studying ecr in multilin-gual cross-document scenarios. existing datasets for cross-document event coreference resolution are primarily available in english @xcite , providing limited annotated samples. to address these limitations, this work introduces a novel dataset, namely the multilingual cross-document event coreference resolution (mcecr) dataset, specifically designed to facilitate research in this domain. to collect a diverse set of articles for annotation for event coreference, we employ news articles published in wikinews, an online news source covering various domains. specifically, news articles in the domains of politics, crime, health, technology, sports, economy and dissasters are collected from language-specific dumps of wikinews for five languages: english, spanish, ukrainian, turkish, and hindi. these articles serve as seed nodes to collect more related news articles from google searches. the news articles from google searches are then filtered based on some criteria to enhance the likelihood of encountering coreferring event mentions. to this end, the collected corpus consists of 1,456 news topics in 5,802 articles. to annotate the collected corpus, we hire nativespeaker annotators for each language. the annotation is performed in two phases: (1) event detection and (2) coreference resolution. in the first phase, the annotators read the news articles and identify event triggers. as such, we focus on open domain events to broaden the scope of our dataset. next, the annotators are provided with a pair of event triggers and they need to decide whether the provided event mentions refer to the same realworld event or not. since there might be a large number of event triggers in the articles of each news topic, we first employ a pre-trained model to identify the challenging event trigger pairs, which will be annotated by the human annotators to improve efficiency. in contrast, the non-challenging event mention pairs (determined by the confidence of the predictions from the pre-trained model) are automatically labeled and sampled for manual verification with the annotators. using this approach, out of 54,791 events, 4,266 non-singleton event clusters are detected, leading to a high-quality dataset for multilingual ecr. compared to previous benchmark dataset for this task, i.e. ecb+ @xcite , which contains only 722 nonsingleton event clusters and supports only english, our proposed dataset provides more annotated samples in a diverse set of languages, serving as a strong benchmark for multilingual cross-document event coreference resolution. in order to show the potentials of the proposed dataset for cross-document event coreference resolution in different languages, we conduct extensive experiments using the state-of-the-art ecr models. in particular, transformer-based models, joint clustering @xcite , and hierarchical models @xcite are employed to study the challenging nature of our mcecr dataset. additionally, we conduct experiments on the cross-lingual transfer learning settings for crossdocument ecr for the first time in the literature, revealing the unique challenges of this problem for future research. overall, our experiments demonstrate much room for further research on multilingual ecr with our dataset as the foundation to boost progress in this area. we will publicly release our dataset for future research. ## data collection given the limitations of existing resources for multilingual ecr, our objective is to construct a dataset covering a diverse set of languages and provide coreference links in the cross-document setting. to achieve this, we start by collecting event-rich articles from wikinews foot_0 , which is an online source for publishing news. we select this source as it provides news articles in different languages and covers a variety of categories. in particular, five languages are chosen, i.e., english, spanish, turkish, ukrainian, and hindi, and the news articles from their wikinews dumps are sampled for further processing and annotation. these languages represent different language families, and thus could better exhibit the challenges of cross-lingual ecr. moreover, we select the following categories for article selection to promote data diversity: (1) politics and conflicts, (2) crime and law, (3) health, (4) science and technology, (5) sports, (6) disasters and accidents, and (7) economy and business. note that we use language-specific categories to select the news articles for each language. the selected news articles from wikinews provide a valuable source of event-rich documents. however, in order to construct a richer dataset with many cross-document event coreference links, it is necessary to collect other articles that have more coreferring event mentions to the events mentioned in the wikinews articles. to this end, we employ google search results to obtain such articles. in particular, for each wikinews article, we retrieve news articles from other sources using the google search engine. the titles of the wikinews articles are employed as the queries in our searches as they provide short summaries for the main events in the articles. for each query, the first 50 search results are selected. these search results are further filtered to only keep the documents that have higher chance to contain referring event mentions to those in each wikinews article. our filtering criteria is based on the temporal correspondence heuristics (zhang and weld, 2013), which suggests the tendency to discuss the same events in different sources at the same or similar times (i.e., close to the occurring time of the events). as such, we only retain the searched articles whose publish dates are within 7 days of the publish dates of the original wikinews articles to improve the precision to obtain the same events. note that for each wikinews article, the search results are limited to being in the same language as the article itself. using this approach, on average, for each wikinews article (called a news topic), 2.9 related articles are selected from google search results. finally, the textual content of all wikinews articles and selected articles from google search results are extracted to be annotated by native speakers in each language. ## annotation this section discusses the details of our annotation for the collected corpus with event triggers and event coreference links. in this work, we recruit native speakers to annotate the documents in each language. in particular, three annotators per language are hired from the freelancer website upwork 2 . the hired freelancers are required to be native speakers of the target languages and fluent in english for training and communication. they also need to have experience in data annotation and pass an examination test to verify their ability on 2 www.upwork.com event and coreference annotation. annotators are paid a fixed hourly rate of $15 per hour, which is significantly higher than the minimum wage per hour in their countries. to annotate the corpus we employ two phases: (1) event detection: to annotate event triggers, and (2) event coreference resolution: to identify the event mentions that refer to the same real-world events in our collection of news documents. ## experiments to facilitate the evaluation and development of multilingual ecr models, this section studies how typical ecr models perform on our proposed dataset mcecr. in the literature, ecr is often modeled as a binary classification task @xcite . given two event mentions (in the same or different documents) along with their context, typical ecr models first concatenate the context for the two mentions to form a single input text @xmath0, w 2 , . . . , w n , where 1 ≤ s 1 ≤ e 1 < s 2 ≤ e 2 ≤ n are the start and end indexes for the spans of the first and second event mentions in w . the models then aims to classify w into 1 or 0 to indicate whether the two event mentions corefer to each other or not. in the inference time, once the labels for all pairs of event triggers in a document (for withindocument ecr) or across documents in a topic (for cross-document ecr) are predicted, the event chains will be constructed for coreference evaluation. to evaluate the performance of the models, following prior work @xcite , we report the coreference evaluation metrics muc @xcite , b 3 @xcite , ceaf e @xcite , and blanc @xcite . we use the implementations provided by @xcite to compute these metrics. to prepare the proposed mcecr dataset for model evaluation, we divide the topics (i.e., wikipedia articles) for each language into train/dev/test sets with a ratio of 80/10/10. for each set, we create two versions: (1) within-doc: to use only coreference links for event mentions in the same documents to create data, and (2) cross-doc: to use all coreference links for event mentions in the documents of the same topic to form event chains and data. note that in the cross-doc setting, two event triggers might be in the same document or in different documents of the same topic. for the experiments, to address documents with long context, for both settings, we limit the context of each event mention to three surrounding sentences, i.e., the sentence that contains the event trigger plus the previous and next sentences. ## related work event coreference resolution (ecr) is one of the important tasks for any event understanding and information extraction systems. to this end, there has been a considerable body of prior work for this problem. we study the prior work for eci in two dimensions, i.e., datasets and models: datasets: ecr data has already been provided in some of the existing event extraction datasets. specifically, @xcite , muc @xcite , tac kbp @xcite @xcite , ontonotes @xcite are the popular event datasets with manuallyannotated event coreference information. however, such datasets are mainly developed for english and some popular languages, i.e., spanish, arabic, or chinese. also, each previous ecr dataset on its own only supports at most 3 languages (e.g., ace 2005 and tac kbp). as such, these datasets cannot extensively evaluate models in less popular languages to better support multilingual research for ecr. most importantly, all of these datasets are only annotated for within-document event coreference that hinders model development for crossdocument ecr with multiple languages. regarding cross-document ecr, ecb+ (cybulska and vossen, 2014) has served as the major dataset to boost research progress for this problem. however, the lack of annotations in non-english documents is a critical shortcoming in ecb+ that prevents multilingual learning research for crossdocument ecr. also, it is noteworthy that ecb+ provides much less non-singleton event clusters than our dataset (i.e., 722 vs. 4,266), making ecb+ less suitable for developing data-hungry deep learning models. in contrast, mcemr represents the first dataset that annotates both within-document and cross-document event coreference for multiple languages (i.e., beyond english). compared to existing ecr datasets, our dataset supports the largest number of languages (i.e., five languages), covering turkish, ukrainian, and hindi for the first time in ecr research. with much more non-singleton event clusters, our dataset also enables training of larger models for ecr. in addition to manually-annotated datasets, there are some other ecr datasets that are automatically collected and annotated, including meantime @xcite , gvc @xcite , and wec @xcite foot_2 . however, due to the inherent noises in the fully automatic annotation, these datasets cannot guarantee the highest quality for multiple languages to provide reliable resources for model development for ecr. as such, our mcecr dataset leverages human annotation to control and produce a higher-quality dataset in multilingual languages for both with-document and cross-document ecr to significantly facilitate future research in this area. finally, due to the relatedness of eci and the event and event-event relation extraction (eere) tasks @xcite @xcite , we also note some recent multilingual datasets for event extraction @xcite and @xcite . models: the ecr task in the literature has been approached with different methods ranging from feature-based models to deep learning methods. in particular, for the feature-based approach @xcite , the typical models for ecr have employed svms @xcite , markov logic network @xcite , and integer linear programming (choubey and @xcite . recently, deep learning has been used extensively to solve ecr @xcite @xcite . the authors in @xcite employ lstm to encode input text and model the compatibility of arguments in event clusters. more recently, the application of large language models, e.g., bert or roberta, has increased in ecr models. the authors in @xcite employ a contrastive learning technique to train the roberta-based model for ecr. in @xcite , the authors leverage bert-based encoders to encode local and global context for events. for multilingual learning, @xcite explores cross-lingual transfer learning for within-document ecr with adversarial training. however, due to the lack of necessary datasets, none of those previous work has explored multilingual cross-document ecr as we do. finally, finally, it is worth noting that the modeling approaches for ecr shares some similarities with the popular task of relation extraction in information extraction @xcite . ## conclusion in this work, we introduce mcecr, a multilingual event coreference resolution dataset. compared to previous dataset, mcecr is the first ecr dataset that provides annotation for both within-document and cross-document event coreference for multiple languages. our dataset is annotated on wikipedia articles and related news articles obtained from google searches in 5 different languages (i.e., english, spanish, turkish, ukrainian, and hindi). we study the challenging nature of this dataset by evaluating the performance of strong baselines in monolingual and cross-lingual settings. our experiments reveal the necessity of further research on the proposed mcecr dataset to improve the performance of ecr models in multilingual learning.