Fine-Grained Event Trigger Detection (2021)

Abstract: Most of the previous work on Event Detection (ED) has only considered the datasets with a small number of event types (i.e., up to 38 types). In this work, we present the first study on fine-grained ED (FED) where the evaluation dataset involves much more fine-grained event types (i.e., 449 types). We propose a novel method to transform the Semcor dataset for Word Sense Disambiguation into a large and high-quality dataset for FED. Extensive evaluation of the current ED methods is conducted to demonstrate the challenges of the generated datasets for FED, calling for more research effort in this area.

## introduction understanding events in text is an important aspect of natural language processing (nlp). toward this end, event detection (ed), a task of information extraction (ie), aims to identify event triggers in sentences and classify them into some predefined types of interest. event triggers represent the most important words (usually single verbs or nominalizations) in the sentences that evoke the events. the current state-of-the-art methods for ed feature the deep learning models where many new network architectures are introduced in the last couple of years @xcite @xcite @xcite . among others, the rapid development of the deep learning models for ed can be partly attributed to the availability of the large datasets to evaluate the models (e.g., the ace 2005 and tac kbp 2015 datasets @xcite ). unfortunately, a major issue in these existing datasets for ed is that they tend to only focus on a limited set of event types. for example, the popular ace 2005 dataset is only annotated * corresponding author. for 33 event subtypes (e.g., attack, start-position, elect) while the number of events in the tac kbp dataset @xcite is 38. on the one hand, the limited numbers of types are unable to cover a wide range of possible events in practice @xcite . on the other hand, the small label sets often amount to the coarse-grained event types in the existing datasets that cannot capture the slightly different nuances (i.e., fine-grained distinction) of the events. for instance, both the words "quit" and "fired" in the two sentences "he decided to quit the job." and "he was fired due to a policy violation." (respectively) would be considered as the trigger words of the same event type of end-position in the ace 2005 dataset. however, the nuances in these two events are quite different (i.e., in term of the willingness of the job termination) and the ability to characterize such subtle distinction would be useful for the downstream applications @xcite . in order to address these problems, we propose to explore the problem of fine-grained event detection (fed) that seeks to solve ed with much larger and finer-grained sets of event types (motivated by the fine-grained entity typing task @xcite ). to our knowledge, this is the first work to explicitly study fed in the literature. a major challenge in this research direction is the creation of the evaluation datasets to enable effective model development and analysis. in particular, it is non-trivial to design a large set of fine-grained event types to be applied to annotate the datasets. in addition, with such a large number of fine-grained event types (i.e., 449 in this work), the traditional labeling procedure with human involvement might be too expensive and error-prone when it comes to the generation of large datasets for fed. to this end, we introduce a novel method to address these challenges and produce a large dataset for fed based on wordnet and word sense disambiguation (wsd) datasets. our method involves two major steps where we first leverage the synset typology in wordnet to formulate the finegrained event types and then convert the annotated datasets for wsd to establish the datasets for our fed problem. this novel data generation procedure minimizes the human effort and allows us to create a large and high-quality dataset with 449 fine-grained event types for fed. finally, we extensively evaluate the state-of-the-art ed models on the proposed fed dataset. the experiments show that the performance of the current ed models is not yet satisfactory for fed and further research is needed to advance the performance in this area. we will publicly release the proposed dataset to promote the future research on fed. ## data generation procedure the goal of this section is to generate a large dataset for ed with many fine-grained event types to evaluate the fed models. our proposed procedure to achieve this goal involves two major steps. first, we identify the eventive synsets/senses in wordnet 3.0 @xcite and group them into classes with similar eventive meanings. these classes would serve as the fine-grained event types in the resulting fed dataset. as the result, we obtain a mapping from the set of wordnet synsets to the set of the fine-grained event types for our problem (some wordnet synsets might not be mapped to any event type in our case). afterward, we leverage the semcor dataset for wsd @xcite and map the synsets annotated for the words in this dataset into the event types in our setting. this conversion process produces a dataset whose words are assigned with the fine-grained event types in our fed problem. as the semcor dataset is manually annotated, the resulting fed dataset would be large and have high quality if the synset-event type mapping is constructed well. in particular, for eventive synset/sense identification, we first start with nouns. following @xcite , we assume that any synset for a noun subsumed by one of the three following synsets via the wordnet hyponyms would be considered as eventive: state 2 n (i.e., the way something is with respect to its main attributes), process 6 n (i.e., a sustained phenomenon or one marked by gradual changes through a series of states), and event foot_0 n (i.e., something that happens at a given place and time). in this way, we find 13,166 even-tive synsets over 82,115 synsets for nouns. we gloss: the act of changing location from one place to another example: he would walk miles into the campagna to visit with them, and in particular to see their horses. gloss: a change of position that does not entail a change of location example: styka shook his head slowly. event.n.01 act.n.02 action.n.01 level 1 level 2 change.n.03 motion.n.03 motion.n.06 level 3 level 4 level 5 root synsets event type the synsets for the event type corresponding to the core synset motion.n.06 state.n.02 process.n.06 hyponym link call these three general synsets as the eventive root synsets in the following. starting from these root synsets, we traverse the synset graph in wordnet by following the hyponym links. the graph traversal procedure will generate three different trees whose nodes are the eventive synsets and roots correspond to the three selected synsets. for convenience, we call the synsets in wordnet that can be reached by one of the three synsets above after n hyponym links as the synsets at the n-th level 1 (so the root synsets are at the zero level). in order to form the fine-grained event types for fed, we select the wordnet synsets at the 4 th level as the core meanings (called the core synsets) for the event types in our dataset (there are 2,637 core eventive synsets found in this way). we empirically choose the synsets at the 4 th level to balance two factors. on the one hand, the synsets at the shallower levels lead to too general event types that cannot achieve the expected fine-grained property. on the other hand, going deeper for the core event meanings reduces the numbers of examples per event type in the final fed dataset converted from semcor. given a core synset a, we identify the other synsets with similar meaning to a and combine them to represent a fine-grained event type (called e) in our dataset (i.e., the event type e will involve several semantically similar synsets in wordnet). in this work, we include two following classes of synsets in the event type e for a: up to this point, we obtain a set of 2,637 event types, each represented by a core synset and a set of related synsets. we combine all the other synsets (i.e., the ones that do not appear in any of the 2637 types) to create a single event type called other as in the traditional ed task. with these grouping information, we can now create a mapping from the synsets/senses in wordnet to the 2638 established event types for our dataset (called the m-wordnet-event mapping). based on this mapping, we transform each example in the wsd semcor dataset, which involves a sentence and a word of interest, into an example in the new dataset for fed (called fedsemcor) where the synset/sense label for the word in the original example of semcor is mapped into the corresponding event type in fedsemcor. as the final processing step, we remove from fed-semcor any event types that have less than 10 examples to ensure that the event types are adequately represented in our dataset. this step significantly reduces the number of event types in fedsemcor, leaving us 449 event types (not including other) with 34,666 examples. these are called the positive examples where we have 77.2 examples per event type in average. as a result, the synsets for the removed event types are also included in the synset set for the other type in the mapping. the number of examples with the other type in the final fedsemcor dataset (called the negative examples) is 98,309. figure 1 illustrates the synset levels and some examples for the event types. ## related work ed has been studied extensively in the last decade, featuring feature-based models @xcite @xcite , deep learning models @xcite 2016b,a; nguyen and grishman, 2016; @xcite @xcite @xcite , and few/zero-shot learning models @xcite @xcite . the rapid development of such models has been facilitated by the availability of the ed datasets in different domains, including the general domain with the popular ace and tac kbp datasets @xcite @xcite , the biomedical domain @xcite , literature @xcite , cybersecurity @xcite , and the open domain @xcite . however, these datasets only involve a small number of event types and none of them has considered ed with many fine-grained event types as we do. our fed task is also related to fine-grained entity typing that aims to classify entity mentions into a fine-grained set of types @xcite @xcite . the techniques to generate datasets for fine-grained entity typing include distant supervision @xcite and manual annotation @xcite . notably, @xcite also uses wordnet to establish the fine-grained entity types, applying different entity mention extractors over external corpus. our work is different as we focus on fine-grained event types using the manually annotated corpus semcor to generate data. ## conclusion we study a new task of fed, featuring 449 finegrained event types in the dataset for ed. a novel method to generate the evaluation dataset for fed is introduced, leveraging manually annotated wsd datasets (i.e., semcor) and the eventive synsets in wordnet. we evaluate the state-of-the-art ed models on the new dataset to show the opportunities for the future research on fed.