MUSIED : A Benchmark for Event Detection from Multi-Source Heterogeneous Informal Texts (2022)

Abstract: Event detection (ED) identifies and classifies event triggers from unstructured texts, serving as a fundamental task for information extraction. Despite the remarkable progress achieved in the past several years, most research efforts focus on detecting events from formal texts (e.g., news articles, Wikipedia documents, financial announcements). Moreover, the texts in each dataset are either from a single source or multiple yet relatively homogeneous sources. With massive amounts of user-generated text accumulating on the Web and inside enterprises, identifying meaningful events in these informal texts, usually from multiple heterogeneous sources, has become a problem of significant practical value. As a pioneering exploration that expands event detection to the scenarios involving informal and heterogeneous texts, we propose a new large-scale Chinese event detection dataset based on user reviews, text conversations, and phone conversations in a leading e-commerce platform for food service. We carefully investigate the proposed dataset’s textual informality and multi-domain heterogeneity characteristics by inspecting data samples quantitatively and qualitatively. Extensive experiments with state-of-the-art event detection methods verify the unique challenges posed by these characteristics, indicating that multi-domain informal event detection remains an open problem and requires further efforts. Our benchmark and code are released at https://github.com/myeclipse/MUSIED.

## introduction event detection (ed), which aims to identify event triggers and classify them into specific types from unstructured texts, has been widely researched and applied in various downstream tasks @xcite @xcite . advanced models have been continuously proposed, ranging from feature-based models @xcite @xcite to recent neural-based models @xcite @xcite @xcite . despite the significant progress, we find that previous works have the following two limitations in practical scenarios. to address these two problems, in this paper, we expand event detection to the scenarios involving informal and heterogeneous texts. we construct a new large-scale chinese event detection dataset based on meituan * , the most popular chinese ecommerce platforms for food service, which provides users with multiple ways to feed back on food safety issues (events), such as posting reviews and communicating with after-sale staff. these reviews and conversations yield a large-scale multi-source heterogeneous informal text repository, which contains valuable information about food safety events and hence can serve as a corpus. we collect the desensitized data from three typical scenarios: i) users posting reviews, ii) users communicating with after-sale staff through text messages, and iii) users communicating with after-sale staff on the phone. by extracting user reviews, text conversations, and phone conversations, we create a massive dataset consisting of multi-source heterogeneous informal texts for event detection (musied). we investigate musied's textual informality ( § 4.3) and multi-source heterogeneity ( § 4.4) by carefully inspecting data samples. the textual informality leads to event descriptions involving long triggers ( § 4.3.1), multi-event sentences ( § 4.3.2), and user typos ( § 4.3.3), while the multi-source heterogeneity brings notable diversity of event type distribution and event density across domains ( § 4.4). we re-implement the state-ofthe-art ed methods and conduct extensive evaluation on musied ( § 5). the experimental results clearly verify the unique challenges posed by the above characteristics. specifically, the proposed dataset requires more robust models towards identifying long triggers ( § 5.4.1), capturing multievent interaction ( § 5.4.2), and alleviating typo noises( § 5.4.3). meanwhile, musied also facilitates future research on tackling multi-source heterogeneity, e.g., with multi-domain learning and ( § 5.5.1) and domain adaptation ( § 5.5.2). our contributions can be summarized as follows: ## event detection definition we follow the classical settings and terminologies adopted by @xcite and maven @xcite , and specify the vital event terminologies as follows. event: a specific occurrence involving participants (location, time, subject, object, etc.). event mention: a phrase or sentence within which an event is described. event trigger: the main word or phrase that most clearly expresses the occurrence of an event. event type: the semantic class of an event. event detection aims to identify event trigger words and classify their event types for a given text. accordingly, ed is conventionally divided into two subtasks: (1) trigger identification, which aims to identify the event triggers. (2) trigger classification, which aims to classify the recognized trigger into predefined categories. both subtasks are evaluated with micro precision, recall, and f-1 scores. most recent works @xcite @xcite perform trigger classification directly (add an additional type "n/a" to be classified at the same time, indicating that the candidate is not a trigger). we also inherit these settings in this paper. ## related work most existing works towards event detection adopt general domain datasets such as ace 2005 @xcite , tac kbp datasets @xcite and maven @xcite as benchmarks. also, some works present domain-specific datasets and valuable explorations. for example, event extraction from biomedical texts are extensively researched @xcite @xcite . @xcite present a new dataset of literary events. casie @xcite ## conclusion and future work we have presented musied, a massive multisource heterogeneous informal text dataset for event detection, based on user reviews, text conversations and phone conversations of online food services. the extensive evaluation verify the unique challenges posed by the textual informality and multi-source heterogeneity characteristics. our in-depth investigations present multiple promising directions worth pursuing, including exploiting document-level information, multi-domain learning and domain adaptation. in the future, we are interested in extending musied to more eventrelated tasks such as event argument extraction.