MEE : A Novel Multilingual Event Extraction Dataset (2022)

Abstract: Event Extraction (EE) is one of the fundamental tasks in Information Extraction (IE) that aims to recognize event mentions and their arguments (i.e., participants) from text. Due to its importance, extensive methods and resources have been developed for Event Extraction. However, one limitation of current research for EE involves the under-exploration for non-English languages in which the lack of high-quality multilingual EE datasets for model training and evaluation has been the main hindrance. To address this limitation, we propose a novel Multilingual Event Extraction dataset (MEE) that provides annotation for more than 50K event mentions in 8 typologically different languages. MEE comprehensively annotates data for entity mentions, event triggers and event arguments. We conduct extensive experiments on the proposed dataset to reveal challenges and opportunities for multilingual EE. To foster future research in this direction, our dataset will be publicly available.

## introduction event extraction (ee) is one of the major tasks of information extraction (ie) for text. in a complete ee pipeline, three major goals should be pursued: (1) entity mention detection (emd): to recognize mentions of real world entities; (2) event detection (ed): to identify event mentions/triggers and their types. an event trigger is a word or phrase that most clearly refers to the occurrence of an event; and (3) event argument extraction (eae): to find participants/arguments of an event mentioned in text. a participant is an entity mention that has an specific role in a given event mention. for instance, in the sentence "the soldiers were hit by the forces.", there are two entity mentions "soldiers" and "forces" of types person and orga-nization and an event trigger "hit" of type at-tack. also, the two event mentions "soldiers" and "forces" play the argument roles of victim and attacker (respectively) in the attack event. an ee system could be employed in other downstream applications such as question answering, knowledge base population and text summarization to assist extracting information about events in text. multiple methods have been proposed for event extraction. early work has employed feature-based models @xcite @xcite @xcite while later methods have explored deep learning to present state-ofthe-art performance for event extraction @xcite @xcite @xcite @xcite @xcite . however, despite all advancements on event extraction in recent years, a major limitation of current ee research is to overly focus on a few popular languages, thus failing to adequately reveal challenges and generalization of models in many other languages of the world. as such, a critical barrier for studying ee over multiple languages is the lack of high quality datasets that fully annotate data for many other languages for ee. for instance, the most popular dataset for ee, i.e., ace 2005 @xcite , only provide annotations for three languages english, chinese and arabic while tac kbp datasets @xcite only supports english, chinese and spanish. the tempeval-2 dataset @xcite involves 6 languages; however, it does not offer event argument annotation. even worse, recently created datasets, e.g., maven @xcite , rams @xcite , and wikievents @xcite , are only annotated for english. in all, such language and task limitations prevents research to comprehensively develop and evaluate ee methods over different languages and multilingual settings. moreover, the limited size of these datasets, i.e. less than 11k and 27k in ace 2005 and tempeval-2 respectively, hinders training of data-hungry deep learning models. finally, we note that important multilingual datasets for ee, e.g., ace 2005 and tac kbp, are not publicly available, which further restricts research on this domain. to address these limitations, in this work, we propose a large-scale multilingual event extraction (mee) dataset that covers 8 typologically different languages from multiple language families, including english, spanish, portuguese, polish, turkish, hindi, korean, and japanese. as such, portuguese, polish, turkish, hindi, and japanese are not explored in the popular multilingual datasets for ee, i.e., ace 2005 and tac kbp. importantly, to enable public data sharing and diversity the data, we employ wikipedia articles for the 8 languages in diverse topics (i.e., economy, politics, technology, crime, nature and military) for ee annotation. our dataset comprehensively annotates each document in a language for all the three sub-tasks emd, ed, and eae. to be consistent with prior ee research, we inherit the type anthologies for such tasks from the ace 2005 dataset that provides welldesigned guidelines and examples for the types. in particular, we include 7 entity types, 8 event types and 16 event sub-types, along with 23 argument roles in mee to facilitate ee annotation over multiple languages. overall, our dataset involves more than 415k entity mentions, 50k event triggers, and 38k arguments, which are much larger than previous multilingual ee datasets to better support model training and evaluation with deep learning. due to shared information schema over all the languages, our mee dataset enables cross-lingual transfer learning evaluation of mee models where training and test data comes from different languages. to this end, we conduct comprehensive experiments for both monolingual and cross-lingual learning settings to provide insights for languagespecific challenges and cross-lingual generalization of ee methods. by examining both pipeline and joint inference models for ee, our experiments show that the proposed dataset present unique challenges with less satisfactory performance of existing ee models, especially for cross-lingual settings, thus calling for more research efforts for multilingual ee in the future. ## data annotation we follow the entity/event type definition and annotation guidelines from the popular ace 2005 dataset to benefit from its well-designed documentation and be consistent with prior ee research. as such, entity mentions refer to mentions of realworld entities in text that can be expressed via names, nominals, and pronouns. entity mention extraction (emd) is more general than named entity recognition that only concerns names of entities. in addition, an event is defined as an incident whose occurrence changes the state of real world entities. an event mention is the part of input text that refers to an event that consists of two components: (1) event trigger: the words that most clearly refer to the occurrence of the event. it is noteworthy that we allow an event trigger to span multiple words to accommodate trigger annotation for multiple languages. for instance, in the turkish phrase "tayin etmek", both words are necessary to indicate an event trigger of type "appoint"; and based on the ace 2005 dataset, our dataset annotates entity mentions for 7 entity types: per-son (human entities), organization (corporations, agencies, and other groups of people), gpe (geographical regions defined by political and/or social groups), location (geographical entities such as landmasses or bodies of water), facility (buildings and other permanent manmade structures), vehicle (physical devices primarily designed to move an object from one location to another), and weapon (physical devices primarily used as instruments for physically harming). for event types, to avoid confusion and improve data quality, we prune the original ace 2005 event types to only include the types that are not ambiguous across multiple languages. for instance, in turkish, the event types sentence and convict are very similar (both can be evoked by the phrase "mahkum etmek") so they are not retained in our dataset. as such, we preserve 8 event types and 16 sub-types that are distinct enough for annotation in our dataset. finally, for event arguments, we preserve all 23 argument roles in the ace 2005 dataset. ## experiments this section evaluates the state-of-the-art models for event extraction to reveal challenges in our new dataset mee. to this end, the annotated article segments for each language in mee are randomly split into training/development/test portions with the ratios of 80/10/10. here, to prevent any information leakage, we ensure that different segments of an article (if any) are only assigned to one portion of the data split for each language. we examine ee models in two different settings: (1) monolingual learning where training and test data of models comes from the same language; (2) cross-lingual transfer learning where models are trained on training data of one language (i.e., the source language), but evaluated directly on test data of the other languages (i.e., the target languages). models: we evaluate two typical approaches for ee models with pipeline and joint inference in this work. first, for the pipeline approach, a model is trained separately for each of the three tasks in ee, i.e., entity mention detection (emd), event detection (ed), and event argument extraction (eae). here, the emd and ed tasks are modeled as sequence labeling problems, aiming to predict bio tag sequences for each input sentence to capture spans and types of entity and event mentions. as such, motivated by previous work @xcite second, for the joint inference approach, ee models simultaneously predicts entity mentions, event triggers, and arguments in end-to-end fashion to avoid error propagation and leverage inter-dependencies between tasks. to this end, we evaluate two state-of-the-art (sota) joint ee models, oneie @xcite and @xcite , in this work due to their language-agnostic nature. both oneie and fourie utilize pre-trained language models to represent input texts and capture cross-task dependencies for joint inference. note that these models are original designed to include the relation extraction task between entities. to adapt them to ee, we obtain their implementations from the original papers and remove the relation extraction components. finally, for performance measure, we report the performance (f1 scores) of ee models over three tasks emd (entity), ed (event), and eae (argument) using the same correctness criteria as in prior work @xcite (i.e., requiring correct prediction for both offsets and types of entity mentions, event triggers, and argument roles). hyper-parameters: to facilitate evaluation with multiple languages, we leverage the multilingual pre-trained language models (plms) mbert @xcite and xlm-roberta @xcite (base versions) to encode texts for ee models. for the pipeline approach, we fine-tune the hyper-parameters for the emd, ed, and eae models over development data for english and apply the selected values for all experiments for consistency. in particular, our hyper-parameters for the pipeline model include: 2 hidden layers with 250 hidden units in each layer for the feed-forward networks, 8 for mini-batch size, and 1e-2 for learning rate with the adam optimizer. for the joint ie models, we utilize the same hyper-parameters suggested in the original papers, i.e., oneie @xcite and @xcite . results: the results for monolingual experiments over different languages in mee are presented in tables 5 and 6 (i.e., with mbert and xlm-roberta encoders respectively). there are several observations from the tables. first, the models' performance on individual languages and on average for all three tasks emd, ed, and eae is still far from being perfect (i.e., all average performance is less than 69%), thus indicating considerable challenges in our multilingual ee dataset for future research. in addition, comparing the current stateof-the-art joint ie model (i.e., fourie) with the pipeline method, we find that fourie is better than the pipeline model on average, especially for the eae task with significant performance gap. as such, we attribute this to the ability of joint models to mitigate error propagation to eae from emd and ed to boost the performance. due to its best average performance, fourie will be leveraged in our next experiments. finally, we find that xlm-roberta generally has better performance than mbert (i.e., on average) for ee models. future research can thus focus on xlm-roberta to develop better ee models for multilingual settings. ## related works due to its importance, various datasets have been recently developed for ee in different domains, including cyseced (man et al., 2020) (for cybersecurity domain), litbank (for literacy) @xcite , maven @xcite , rams @xcite , and wikievents @xcite (for wikipedia texts). however, these datasets are only annotated for english texts. there exist several multilingual datasets for ee, ace @xcite , tac kbp @xcite , and tempeval-2 @xcite ; however, such datasets only provide annotation for a handful of popular languages with limited number of event mentions and might not fully support all ee tasks (e.g., missing eae in tac kbp and tempeval-2). regarding model development, existing ee methods can be categorized into feature-based @xcite @xcite @xcite or deep learning @xcite @xcite @xcite @xcite @xcite methods. while most prior ee methods have been designed for one popular language, there have been growing interests in multilingual and cross-lingual learning for ee in recent work, featuring multilingual plms (i.e., mbert and xlmr) as the key component for representation learning @xcite @xcite @xcite . however, as such works only rely on existing multilingual ee datasets, their evaluation is limited to a few popular languages and fails to evaluate the generalization over many other languages. ## conclusion we present a novel multilingual ee dataset, i.e., mee, that covers 8 typologically different languages with more than 50k event mentions to support training of large deep learning models. mee provides complete annotation for three ee subtasks, i.e., entity mention detection, event detection, and event argument extraction. to study the challenges in mee, we conduct extensive analysis and experiments with different ee methods in the monolingual and cross-lingual learning settings. our results demonstrate various challenges for ee in the multilingual settings that can be further pur-sued with mee. in the future, we will extend our dataset to include more languages and tasks for ie.