MECI : A Multilingual Dataset for Event Causality Identification (2022)

Abstract: Event Causality Identification (ECI) is the task of detecting causal relations between events mentioned in the text. Although this task has been extensively studied for English materials, it is under-explored for many other languages. A major reason for this issue is the lack of multilingual datasets that provide consistent annotations for event causality relations in multiple non-English languages. To address this issue, we introduce a new multilingual dataset for ECI, called MECI. The dataset employs consistent annotation guidelines for five typologically different languages, i.e., English, Danish, Spanish, Turkish, and Urdu. Our dataset thus enable a new research direction on cross-lingual transfer learning for ECI. Our extensive experiments demonstrate high quality for MECI that can provide ample research challenges and directions for future research. We will publicly release MECI to promote research on multilingual ECI.

## introduction event causality identification (eci) is an important information extraction (ie) task that aims to identify causal relations between event mentions in text. for example, in the sentence "after inspection of his computer , officers found that he was interested...", a eci system should detect a causal relation between two events "inspection" cause ---→ "found". eci can provide valuable information for various applications such as event timeline construction @xcite , questionanswering @xcite , future event forecasting @xcite , and machine reading comprehension @xcite . due to its applications, eci has been extensively studied in the natural language processing community over the past decade. the vast majority of methods for eci involve feature engineering models @xcite @xcite @xcite and recent deep learning architectures @xcite @xcite @xcite . as such, the creation of large annotated datasets, e.g., eventstoryline @xcite , has been critical to the development of eci study. however, existing datasets for eci only annotate causal relations between event mentions in data of a single language, i.e., mainly for english @xcite @xcite . on the one hand, this leaves many other languages unexplored for eci, posing an important question about the generalization ability of existing methods to other languages. for instance, spanish, danish, and turkish are not covered in those separated datasets for eci. moreover, the current single-language datasets for eci tend to employ different annotation guidelines that prevent their combination into a larger corpus and cross-lingual transfer learning research to train and evaluate models in different languages. in all, the annotation discrepancy and limited language coverage hinder the research and development of the eci in various dimensions, necessitating a new dataset with broader coverage for eci. to address this issue, this paper introduces a multilingual event causality identification (meci) dataset to standardize and foster future research in multilingual eci. particularly, we present a largescale eci dataset for five languages, i.e., english, danish, spanish, turkish, and urdu foot_0 that are annotated with the same annotation guideline to enable cross-lingual transfer learning evaluation for the first time. as such, four languages, i.e., danish, spanish, turkish, and urdu, are not explored in any of the existing datasets for eci. to facilitate open access to the dataset, we obtain the texts from wikipedia for annotation in all examined languages. to make it consistent with prior research and benefit from the well-designed annotation guidelines of previous datasets, we inherit the event schema from the ace 2005 dataset @xcite , and the causal event relation guideline from eventstory-line @xcite (with both explicit and implicit causal relations) during the annotation process. in total, our meci dataset involves 46k events and 11k relations that are substantially larger than those in existing eci datasets. figure 1 illustrates our annotation interface in this work. in addition, we evaluate the proposed meci dataset using the state-of-the-art models for eci. we investigate the challenges of meci over all examined languages through the monolingual setting where the models are trained and evaluated in the same language. the experiments show that the performance of existing eci models, even with large pre-trained language models (plms), is far from satisfactory; models for non-english languages generally perform poorer than their english counterparts. we also observe the importance of choosing language-specific or multilingual plms for eci models as their effectiveness varies for different languages. moreover, we evaluate the models in the zero-shot cross-lingual setting, where the models are trained on english data and tested on the data of the other languages. the experiment suggests transferability of eci knowledge between english and urdu while showing a significant performance drop in other language pairs. these results can serve as baselines for future studies on cross-lingual transfer learning for eci. finally, we report the analysis and challenges of the meci dataset to provide insights for future eci research. we will publicly release meci to promote future studies in multilingual eci. ## experiments we randomly split the documents for each language in meci into three separate parts with a ratio of 3/1/1 to serve as training, development, and test data respectively for experiments. to study the challenges of eci presented in meci, we evaluate the performance of the state-of-the-art models for eci on this dataset. each model will be comprehensively evaluated in the monolingual learning (i.e., trained and tested on data of the same language) and multilingual learning (i.e., trained and tested on the data of different language) settings with meci. ## related work as an important task in ie, eci has attracted extensive research effort to develop effective models @xcite @xcite @xcite @xcite @xcite . to support model development for eci, several datasets have been introduced for this task, including pdtb @xcite , causal-timebank @xcite , ecb (cybulska and vossen, 2014), richer event description ( @xcite , because @xcite , and eventstoryline @xcite , caters @xcite . however, these previous work and datasets only focus on english data, presenting a strong demand for new research and datasets on other languages for eci. to this end, there are a few efforts on creating causality corpora for other languages, such as german @xcite , arabic @xcite and persian @xcite . however, these corpus consider not only event mentions, but also entities, clauses, and sentences, thus, not directly solving eci as we do. in addition, most existing annotation efforts for eci focus on explicit event causality relationships. eventstoryline @xcite and caterrs @xcite are the only two prior datasets that also explore implicit causal relationships between events. however, they do not provide annotation for multiple languages as we do in meci. finally, we also note recent efforts on creating multilingual datasets for other nlp tasks, including event detection @xcite , natural language understanding (e.g., slot filling) @xcite , and acronym extraction @xcite . ## conclusion we present a new dataset for event causality identification in five different languages across diverse typologies. the dataset is annotated consistently for all languages, offering a large number of event mentions/causal relations and covering four languages that have not been explored in the prior eci resources. our extensive experiments and analysis reveal the quality and challenges of our dataset for eci. in addition, our dataset enables cross-lingual transfer learning research that is not possible with current resources for eci. in the future, we plan to extend the dataset to include more languages such as arabic and hindi to broaden its coverage.