GLEN : General-Purpose Event Detection for Thousands of Types (2023)

Abstract: The progress of event extraction research has been hindered by the absence of wide-coverage, large-scale datasets. To make event extraction systems more accessible, we build a general-purpose event detection dataset GLEN, which covers 205K event mentions with 3,465 different types, making it more than 20x larger in ontology than today’s largest event dataset. GLEN is created by utilizing the DWD Overlay, which provides a mapping between Wikidata Qnodes and PropBank rolesets. This enables us to use the abundant existing annotation for PropBank as distant supervision. In addition, we also propose a new multi-stage event detection model specifically designed to handle the large ontology size in GLEN. We show that our model exhibits superior performance compared to a range of baselines including InstructGPT. Finally, we perform error analysis and show that label noise is still the largest challenge for improving performance for this new dataset.

## introduction as one of the core ie tasks, event extraction involves event detection (identifying event trigger mentions and classifying them into event types), and argument extraction (extracting the participating arguments). event extraction serves as the basis for the analysis of complex procedures and news stories which involve multiple entities and events scattered across a period of time, and can also be used to assist question-answering and dialog systems. the development and application of event extraction techniques have long been stymied by the limited availability of datasets. despite the fact that it is 18 years old, ace 2005 foot_1 is still the de the clinical testing services holding company is based in san juan capistrano, calif. figure 1 : the glen event detection dataset is constructed by using the dwd overlay, which provides a mapping between propbank rolesets and wikidata qnodes. this allows us to create a large distantlysupervised training dataset with partial labels. the focus of current benchmarks on restricted ontologies in limited domains is harmful to both developers and users of the system: it distracts researchers from building scalable general-purpose models for the sake of achieving higher scores on said benchmarks and it discourages users as they are left with the burden of defining the ontology and collecting data with little certainty as to how well the models will adapt to their domain. we believe that event extraction can, and should be made accessible to more users. to develop a general-purpose event extraction system, we first seek to efficiently build a highquality open-domain event detection dataset. the difficulty of annotation has been a long-standing sentence : a police officer taking part in recovering the bodies said about 17 people were killed. trigger identification recovering killed recovery killing violent_death ... massacre event node : recovery -return of a lost or stolen item to its owners encoder ... ... ## related work as the major contribution of our work is introducing a new large-scale event detection dataset, we review the existing datasets available for event extraction and the various ways they were created. event extraction datasets ace05 set the event extraction paradigm which consists of event detection and argument extraction. it is also the most widely used event extraction dataset to date. the more recent maven @xcite dataset has a larger ontology selected from a subset of framenet @xcite . fewevent @xcite is a compilation of ace, kbp, and dataset #documents #tokens #sentences #event types #event mentions ace2005 599 303k 15,789 33 5,349 maven 4480 1276k 49,873 168 118,732 glen train 5607 3641k 187,468 3,464 184,806 dev 311 204k 10,359 1720 9878 test 306 201k 10,627 1334 8290 total 6224 4045k 208,454 3,465 205,045 wikipedia data designed for few-shot event detection. a number of datasets (dcfee @xcite , chfinann @xcite , rams @xcite , wikievents @xcite , docee @xcite ) have also been proposed for document-level event extraction, with a focus on argument extraction, especially when arguments are scattered across multiple sentences. within this spectrum, glen falls into the category of event detection dataset with a heavy focus on wider coverage of event types. ## the glen benchmark to build the glen benchmark, we first build a general-purpose ontology based on the curated dwd overlay and then create distantly-supervised training data based on the refined ontology (section 3.1). in order to evaluate our model, we also create a labeled development set and test set through crowdsourcing (section 3.2). ## method in the event detection task, the goal is to find the trigger word offset and the corresponding event type for every event. the main challenge for our dataset is the large ontology size and partial labels from distant supervision. to mitigate label noise, we first separate the trigger identification step from the event typing step, since trigger identification can be learned with clean data. then, to handle the large ontology, we break the event typing task into two stages of type ranking and type classification to progressively narrow down the search space. finally, in the type classification model, we adopt ## analysis in this section, we investigate the following questions: (1) which component is the main bottleneck for performance? (section 6.1) and (2) does our model suffer from label imbalance between types? (section 6.2) ## conclusions and future work we introduce a new general-purpose event detection dataset glen that covers over 3k event types for all domains. glen can be seen as our first attempt towards making event extraction systems more accessible to the general audience, removing the burden on users of designing their own ontology and collecting annotation. we also introduce a multi-stage model designed to handle the large ontology size and partial labels in glen and show that our tailored model achieves much better performance than a range of baselines. our model cedar can be used as an off-theshelf event detection tool for various downstream tasks, and we also encourage members of the community to improve upon our results, (e.g. tackle the noise brought by partial labels) or extend upon our benchmark (e.g. include other languages).