MAVEN : A M assive G eneral D omain E vent D etection D ataset (2020)

Abstract: Event detection (ED), which means identifying event trigger words and classifying event types, is the first and most fundamental step for extracting event knowledge from plain text. Most existing datasets exhibit the following issues that limit further development of ED: (1) Data scarcity. Existing small-scale datasets are not sufficient for training and stably benchmarking increasingly sophisticated modern neural methods. (2) Low coverage. Limited event types of existing datasets cannot well cover general-domain events, which restricts the applications of ED models. To alleviate these problems, we present a MAssive eVENt detection dataset (MAVEN), which contains 4,480 Wikipedia documents, 118,732 event mention instances, and 168 event types. MAVEN alleviates the data scarcity problem and covers much more general event types. We reproduce the recent state-of-the-art ED models and conduct a thorough evaluation on MAVEN. The experimental results show that existing ED methods cannot achieve promising results on MAVEN as on the small datasets, which suggests that ED in the real world remains a challenging task and requires further research efforts. We also discuss further directions for general domain ED with empirical analyses. The source code and dataset can be obtained from https://github.com/THU-KEG/MAVEN-dataset .

## introduction event detection (ed) is an important task of information extraction, which aims to identify event triggers (the words or phrases evoking events in text) and classify event types. for instance, in the sentence "bill gates founded microsoft in 1975", an ed model should recognize that the word "founded" is the trigger of a found event. ed is the first stage to extract event knowledge from text @xcite and also fundamental to various nlp applications @xcite @xcite . due to the rising requirement of event understanding, many efforts have been devoted to ed in recent years. the advanced models have been continuously proposed, including the feature-based models @xcite @xcite and advanced neural models @xcite @xcite @xcite @xcite @xcite . nevertheless, the benchmark datasets for ed are upgraded slowly. as event annotation is complex and expensive, the existing datasets are mostly small-scale. as shown in figure 1 , the most widely-used ace 2005 english dataset @xcite only contains 599 documents and 5, 349 annotated instances. due to the inherent data imbalance problem, 20 of its 33 event types only have fewer than 100 annotated instances. as recent neural methods are typically data-hungry, these small-scale datasets are not sufficient for training and stably benchmarking mod-ern sophisticated models. moreover, the covered event types in existing datasets are limited. the ace 2005 english dataset only contains 8 event types and 33 specific subtypes. the rich ere ontology @xcite used by tac kbp challenges @xcite covers 9 event types and 38 subtypes. the coverage of these datasets is low for general domain events, which results in the models trained on these datasets cannot be easily transferred and applied on general applications. recent research @xcite has shown that the existing datasets suffering from the data scarcity and low coverage problems are now inadequate for benchmarking emerging methods, i.e., the evaluation results are difficult to reflect the effectiveness of novel methods. to tackle these issues, some works adopt the distantly supervised methods @xcite to automatically annotate data with existing event facts in knowledge bases @xcite @xcite or use bootstrapping methods to generate new data @xcite . however, the generated data are inevitably noisy and homogeneous due to the limited number and low diversity of event facts and seed data instances. in this paper, we present maven, a humanannotated massive general domain event detection dataset constructed from english wikipedia and framenet @xcite , which can alleviate the data scarcity and low coverage problems: we reproduce some recent state-of-the-art ed models and conduct a thorough evaluation of these models on maven. from the experimental results, we observe significant performance drops of these models as compared with on existing ed benchmarks. it indicates that detecting general-domain events is still challenging and the existing datasets are difficult to support further explorations. we also explore some promising directions with empirical analyses, including modeling the multiple events shown in one sentence, using the hierarchical event schema to handle long-tail types and distinguish close types, and improving low-resource ed tasks with transfer learning. we hope that all contents of maven could encourage the community to make further breakthroughs. ## event detection definition in our dataset, we mostly follow the settings and terminologies defined in the ace 2005 program @xcite . we specify the vital terminologies as follows: an event is a specific occurrence involving participants @xcite . in maven, we mainly focus on extracting the basic events that can be specified in one or a few sentences. each event will be labeled with a certain event type. an event mention is a sentence within which the event is described. as the same event may be mentioned multiple times in a document, there are typically more event mentions than events. an event trigger is the key word or phrase in an event mention that most clearly expresses the event occurrence. the ed task is to identify event triggers and classify event types for given sentences. accordingly, ed is conventionally divided into two subtasks: trigger identification and trigger classification @xcite . trigger identification is to identify the annotated triggers from all possible candidates. trigger classification is to classify the corresponding event types for the identified triggers. both the subtasks are evaluated with micro precision, recall, and f-1 scores. recent neural methods typically formulate ed as a token-level multiclass classification task @xcite or a sequence labeling task @xcite , and only report the trigger classification results (add an additional type n/a to be classified at the same time, indicating that the candidate is not a trigger). in maven, we inherit all the above-mentioned settings in both dataset construction and model evaluation. ## experiments our experiments and analyses will show the challenges of maven and promising ed directions. ## related work as stated in section 2, we follow the ed task definition specified in the ace challenges, especially the ace 2005 dataset @xcite in this paper, which requires ed models to generally detect the event triggers and classify them into specific event types. the ace event schema is simplified into light ere and further extended to rich ere @xcite to cover more but still a limited number of event types. rich ere is used to create various datasets and the tac kbp challenges @xcite @xcite . nowadays, the majority of ed and event extraction models @xcite @xcite @xcite @xcite are developed on these datasets. our maven follows the effective framework and extends it to numerous general domain event types and data instances. there are also various datasets defining the ed task in different ways. the early muc series datasets @xcite define event extraction as a slot-filling task. the tdt corpus @xcite and some recent datasets @xcite @xcite follow the open-domain paradigm, which does not require models to classify events into pre-defined event types for better coverage but limits the downstream application of the extracted events. some datasets are developed for ed on specific domains, like the biomedical domain @xcite @xcite @xcite , literature @xcite , twitter @xcite and breaking news @xcite . these datasets are also typically small-scale due to the inherent complexity of event annotation, but their different settings are complementary to our work. ## conclusion and future work in this paper, we present a massive general domain event detection dataset (maven), which significantly alleviates the data scarcity and low coverage problems of existing datasets. we conduct a thorough evaluation of the state-of-the-art ed models on maven. the results indicate that general domain ed is still challenging and maven may facilitate further research. we also explore some promising directions with analytic experiments, including modeling multiple event correlations (sec-tion 5.3), utilizing the hierarchical event schema to distinguish close types (section 5.6), and improving other ed tasks with transfer learning (section 5.5). in the future, we will extend maven to more event-related tasks like event argument extraction, event sequencing, etc.