MINION : a Large-Scale and Diverse Dataset for Multilingual Event Detection (2022)

Abstract: Event Detection (ED) is the task of identifying and classifying trigger words of event mentions in text. Despite considerable research efforts in recent years for English text, the task of ED in other languages has been significantly less explored. Switching to non-English languages, important research questions for ED include how well existing ED models perform on different languages, how challenging ED is in other languages, and how well ED knowledge and annotation can be transferred across languages. To answer those questions, it is crucial to obtain multilingual ED datasets that provide consistent event annotation for multiple languages. There exist some multilingual ED datasets; however, they tend to cover a handful of languages and mainly focus on popular ones. Many languages are not covered in existing multilingual ED datasets. In addition, the current datasets are often small and not accessible to the public. To overcome those shortcomings, we introduce a new large-scale multilingual dataset for ED (called MINION) that consistently annotates events for 8 different languages; 5 of them have not been supported by existing multilingual datasets. We also perform extensive experiments and analysis to demonstrate the challenges and transferability of ED across languages in MINION that in all call for more research effort in this area. We will release the dataset to promote future research on multilingual ED.

## introduction event detection (ed) is one of the critical steps for an event extraction system in information extraction (ie) that aims is to recognize mentions of events in text, i.e., change of state of real world entities. specifically, an ed system identifies the word(s) that most clearly refer to the occurrence of an event, i.e., event trigger, and also detects the type of event that is evoked by the event trigger. for instance, in the sentence "the city was reportedly struck by f16 missiles.", the word "struck" is the trigger for an attack event. an ed model can be incorporated into other ie pipelines to facilitate the extraction of information related to events and entities, thereby supporting various downstream applications such as knowledge base construction, question answering and text summarization. due to its importance, ed has been extensively studied in the ie and nlp community over the past decade. existing methods for ed extend from feature-based models @xcite @xcite , to advanced deep learning methods @xcite @xcite @xcite @xcite . as such, the creation of large annotated datasets for ed, e.g., ace 2005 @xcite , has been critical to progress measurement and growing development of ed research. however, a majority of current datasets for ed only provide annotation for texts in a single language (i.e., monolingual datasets). for instance, the recent challenging datasets for ed, e.g., maven @xcite , rams @xcite , or @xcite , are all proposed for english documents only. in addition, there are a few existing datasets that include ed annotation for multiple languages (multilingual datasets), e.g., ace 2005 @xcite , tac @xcite , and tempeval-2 @xcite . however, those multilingual datasets only cover a handful of languages (i.e., 3 languages in ace 2005 and tac kbp, and 6 languages in tempeval-2), mainly focusing on popular languages such as english, chinese, arabic, and spanish, and leaving many other languages unexplored for ed. for instance, turkish and polish are not covered in existing multilingual datasets for ed. we also note that existing ed datasets tend to employ different annotation schema and guidelines that prevent the combina-tion of current datasets to create a larger one. in all, the limited coverage of languages and annotation discrepancy in current monolingual/multilingual ed datasets hinder comprehensive studies for the challenges of ed in diverse languages. it also limits thorough evaluations for multilingual generalization of ed models. finally, we note that the major multilingual datasets for ed are not publicly accessible due to the licence of involving documents, e.g., ace 2005 and tac kbp, thus further impeding research effort in this area. to address such issues, our goal is to introduce a new multilingual event detection dataset (called minion) to support multilingual research for ed. in particular, we provide a large-scale dataset that manually annotates event triggers for 8 typologically different languages, i.e., english, spanish, portuguese, polish, turkish, hindi, japanese and korean. among them, the five languages portuguese, polish, turkish, hindi, and japanese are not covered in existing popular datasets for multilingual ed (i.e., ace 2005, tac kbp, and tempeval-2). to facilitate public release and sharing of the dataset, we employ the event articles from wikipedia for annotation in 8 languages. in addition, to improve quality of the data, we inherit the annotation schema and guideline in ace 2005, the well-designed and widely-used dataset for ed research. in total, our minion dataset involves more than 50k annotated event triggers, which is much larger than those in existing multilingual ed datasets (i.e., less than 11k and 27k in ace 2005 and tempeval-2 respectively). we expect that the significantly larger size with more diverse set of languages and public texts in minion can contribute to accelerate and extend research in ed to a larger population. given the proposed dataset, we conduct thorough analysis on minion using the state-of-theart (sota) models for ed. in particular, we first study the challenges of ed in different languages using monolingual evaluations where ed models are trained and tested in the same languages. our experiments suggest that the performance of existing ed models is not yet satisfactory in multiple languages and the model performance on non-english languages is in general poorer than those for english. we also show that current pre-trained language models for specific languages (i.e., monolingual models) are less effective for ed models than multilingual pre-trained language models, e.g., mbert @xcite . in all, our findings highlight greater challenges of ed for non-english languages that should be further pursued in future research. in addition, our minion dataset also facilitate zero-shot cross-lingual transfer learning experiments that serve to reveal the transferability of ed knowledge and annotation across languages. in these experiments, ed models are trained on english data (the source language), but tested in other target languages. our results in this setting demonstrate a wide range of cross-lingual performance for different target languages in minion that introduces a diverse set of languages and data for ed research. finally, we report extensive analysis on minion to provide further data insights for future ed research, including challenges of data annotation, language differences, and cross-dataset evaluation. we will release minion to foster future research for multilingual ed. ## data annotation our dataset minion follows the same definition of events as the annotation guideline in ace 2005 @xcite . specifically, an event is defined as an occurrence that results in the change of state of a real world entity. moreover, an event mention is evoked by an event trigger which most clearly describes the occurrence of the event. while event triggers are mostly single words, we also allow multi-word event triggers to better accommodate ed annotation in multiple languages. for instance, the phrasal verb "tayin etmek" with two words in turkish, meaning "appoint", is necessary to express the event type start-position. we also inherit the annotation schema/ontology (i.e., to define event types for annotation) and guideline in ace 2005 to benefit from its well-designed documentation and be consistent with most of prior ed research. however, to improve the quality of the annotated data, we prune some event sub-types from the original ace 2005 ontology in our dataset. in particular, event sub-types that have very similar meanings in some language are not included in our final ontology. this promotes the distinction between event labels and avoids confusion for annotators to provide high-quality data in different languages. for instance, the event sub-types convict and sentence are very similar in turkish (i.e., both convict and sentence can be translated as mahkum etmek in turkish), thus being removed in our ontology. in addition, we also exclude event sub-types in ace 2005 that are not frequent in our collected data from wikipedia (more details on data collection later), e.g., nominate and declare-bankruptcy. finally, 16 event sub-types (for 8 event types) are preserved in the final event schema for our dataset. we provide detailed explanation and sample sentences for the event types in our dataset in the appendix a. ## experiments this section aims to study the challenges of ed for 8 languages in our minion dataset. as such, we evaluate the performance of the state-of-theart (sota) ed models in the monolingual situations where models are trained and tested on the annotated data of the same language. to prepare for the experiments, we randomly split the annotated data for each language in minion into separate training/development/test sets with the ratio of 80/10/10 (respectively). as minion allows multiword event triggers to accommodate language specialities in multiple languages, we model the ed task in our dataset as a sequence labeling problem. concretely, given an input text segment @xmath0, w 2 , . . . , w n ] with n words, ed models need to predict the label sequence @xmath1, y 2 , . . . , y n ] where y i indicates the label for the word w @xmath2. to this end, following prior work on multilingual ed @xcite and cross-lingual ed (m'hamdi et al., 2019), we examine the following representative sota models for sequencelabeling ed: (1) transformer: a pre-trained transformer-based language model (ptlm), e.g., mbert @xcite , is augmented with a feed-forward network to predict a label for each word in the input text; (2) transformer+crf: this model also employs an ptlm as the transformer model; however, a conditional random field (crf) layer is additionally introduced as the final layer to predict the label sequence y ; (3) transformer+bilstm: this model extends the transformer model by injecting a bidirectional long short-term memory network (bilstm) between the ptlm and the feed-forward network to further abstract the representation vectors; and (4) transformer+bilstm+crf: this model is similar to the transformer+bilstm model with an exception that a crf layer is employed in the end for label sequence prediction. as such, to implement the models, we explore two sota multilingual ptlms models, i.e., mbert @xcite and xlmr @xcite (their base versions) for text encoding. in the model notation, we will replace the prefix "transformer" with "mbert" or "xlmr" depending on the actual ptlm to use (e.g., mbert, mbert+crf, mbert+bilstm). following prior work @xcite , in the experiments, we evaluate the models using precision, recall and f1 scores for correctly predicting event trigger boundaries and types in text. our fine-tuning process suggests similar values of hyper-parameters for the models across languages in minion. in particular, for english, we use one layer for bilstm mod ## related work early attempts for ed have employed feature-based models @xcite @xcite @xcite @xcite while deep learning has recently been proven to be a better approach for ed @xcite @xcite @xcite @xcite @xcite ; pouran ben veyseh et al., 2021a). there have also been recent efforts on creating new datasets for ed for different domains, including biomedical texts @xcite , literary texts @xcite , cybersecurity texts @xcite , wikipedia texts @xcite , fine-grained event types @xcite , and historical texts @xcite . however, such prior works and datasets for ed are mainly devoted to english, ignoring challenges in many non-english languages. non-english datasets for ed also exist @xcite ; however, these datasets are only annotated for one language with divergent ontology and annotation guidelines, thus unable to support comprehensive studies and transferability research for ed on multiple languages. existing ed datasets that cover multiple languages involve ace 2005 @xcite , tac @xcite , and tempeval-2 @xcite . among such datasets, ace 2005 is the most popular dataset used in prior multilingual/cross-lingual ed research @xcite @xcite @xcite . however, such multilingual datasets suffer from the issues of small data size, limited language coverage with greater focus on popular languages, and inaccessibility to the public as discussed in the introduction. finally, we also note some prior works that claim event detection datasets for non-english datasets @xcite @xcite . however, such datasets are not comparable to our dataset as their event detection task is indeed a sentence classification problem where established definition of events with event triggers are not followed and annotated. ## conclusion we introduce a new dataset for ed in 8 typologically different languages. the dataset is significantly larger and covers more and newer languages than prior resources. specifically, 31,226 text segments from language-specific articles of wikipedia are manually annotated in the dataset. our experiments and analysis demonstrate the high quality of the dataset and the multilingual challenges of ed, providing ample room for future research in this direction. in the future, we will extend the dataset to include event argument annotations. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of aro, odni, iarpa, the department of defense, or the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. this document does not contain technology or technical data controlled under either the u.s. international traffic in arms regulations or the u.s. export administration regulations. we thank the anonymous reviewers and tracy king for their helpful feedback.