O mni E vent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding (2023)

Abstract: Event understanding aims at understanding the content and relationship of events within texts, which covers multiple complicated information extraction tasks: event detection, event argument extraction, and event relation extraction. To facilitate related research and application, we present an event understanding toolkit OmniEvent, which features three desiderata: (1) Comprehensive. OmniEvent supports mainstream modeling paradigms of all the event understanding tasks and the processing of 15 widely-used English and Chinese datasets. (2) Fair. OmniEvent carefully handles the inconspicuous evaluation pitfalls reported in Peng et al. (2023), which ensures fair comparisons between different models. (3) Easy-to-use. OmniEvent is designed to be easily used by users with varying needs. We provide off-the-shelf models that can be directly deployed as web services. The modular framework also enables users to easily implement and evaluate new event understanding models with OmniEvent. The toolkit is publicly released along with the demonstration website and video.

## introduction correctly understanding events is fundamental for humans to understand the world. event understanding requires identifying real-world events mentioned in texts and analyzing their relationships, which naturally benefits various downstream applications, such as stock prediction @xcite , adverse drug event detection @xcite , narrative event prediction @xcite , and legal case analysis @xcite . as illustrated in figure 1 , event understanding covers three complicated information extraction tasks: (1) event detection (ed), which is to detect the battle of sultanabad occurred on feb. 13, 1812. … the persians won the battle by moving faster than the russians and attacking them in pinevale at 7:30 am. … in the end, however, the persians lost the invasion due to the russians maneuvering around aras river. … the event triggers (keywords or phrases evoking events in texts) and classify their event types, (2) event argument extraction (eae), which is to extract the event arguments for each trigger and classify their argument roles, and (3) event relation extraction @xcite , which is to identify the complex relationships between events, typically including temporal, causal, coreference, and subevent relations. ed and eae together constitute the conventional event extraction (ee) task. in recent years, event understanding research has grown rapidly @xcite @xcite , and multiple practical systems @xcite @xcite @xcite have been developed. however, as shown in to address these issues, we develop omnievent, a comprehensive, fair, and easy-to-use toolkit for event understanding, which has three main features: (1) comprehensive support for task, model, and dataset. omnievent supports endto-end event understanding from plain texts, i.e., all the ed, eae, and ere tasks. for ed and eae, we classify the mainstream methods into four paradigms, including classification, sequence labeling, span prediction, and conditional generation. we implement various representative methods for each paradigm. for ere, we provide a unified modeling framework and implement a basic pairwise classification method @xcite . we also cover the preprocessing of 15 widely-used english and chinese datasets. (2) fair evaluation. as found in @xcite , there are three major pitfalls hidden in ee evaluation, including data processing discrepancy, output space discrepancy, and absence of pipeline evaluation. omnievent implements all the proposed remedies to help users avoid them. specifically, we implement unified pre-processing for all the datasets and a method to convert the predictions of different paradigms into a unified space. omnievent also provides unified prediction triggers of supported datasets for fair pipeline comparisons. (3) easy-to-use for various needs. we design a modular and extensible framework for omnievent, which appeals to users with various needs. we provide several off-theshelf models that can be easily deployed and used by users interested in applications. model developers and researchers can train implemented methods within several lines of code or customize their own models and evaluate them. by integrating transformers @xcite and deepspeed @xcite , omnievent also supports efficiently system ee ere #supported models #supported datasets llm support dygie 1 1 oneie 1 4 openue 1 2 eventplus 1 n/a fourie 1 n/a resin-11 1 n/a deepke 2 1 @xmath0: comparisons between omnievent and other event understanding systems. the number of supported models and datasets only includes those of event understanding tasks. n/a denotes that the system is an integrated service and does not process benchmark datasets. for omnievent, the module combination enables many possible models and 20 is the number of models we have tested for usability. to demonstrate the effectiveness of omnievent, we present the results of several implemented methods on widely-used benchmarks. we also conduct experiments with models at different scales and show that fine-tuning llms helps achieve better event understanding results. we hope omnievent could facilitate the research and applications of event understanding. ## related work with the advancement of research in nlp, various toolkits or systems for event understanding have been developed. they tend to focus on developing advanced ee systems to achieve improved results on public benchmarks @xcite @xcite or perform robustly in real-world scenarios @xcite . however, these toolkits or systems, designed based on a specific ee model, do not support comprehensive implementations of ee models and are inconvenient for secondary development. there is also some work that has meticulously designed user-friendly algorithmic frameworks @xcite , which are convenient for usage and secondary development. however, they are not specifically designed for event understanding, hence the corresponding support is limited. eventplus @xcite is the only work supporting the entire event understanding pipeline but it only supports temporal relation extraction and does not provide comprehensive implementations of event understanding models. moreover, existing work also neglects the discrepanomnievent also supports large language models (t5-xxl @xcite and flan-ul2 @xcite ). considering the mentioned issues, we present omnievent, a comprehensive, fair, and easy-to-use toolkit for event understanding. compared to other systems in ## the omnievent toolkit we introduce the overview ( § 3.1) and main features of omnievent ( § § 3.2 to 3.4), as well as an online demonstration ( § 3.5) powered by omnievent. ## evaluation in this section, we conduct empirical experiments to evaluate the effectiveness of the omnievent toolkit on widely-used datasets. ## conclusion and future work in the paper, we present omnievent, a comprehensive, fair, and easy-to-use toolkit for event understanding. with the comprehensive and modular implementation, omnievent can help researchers and developers conveniently develop and deploy models. omnievent also releases several off-theshelf models and deploys an online system for enhancing the applications of event understanding models. in the future, we will continually maintain omnievent to support more models and datasets.