T itle2 E vent: Benchmarking Open Event Extraction with a Large-scale C hinese Title Dataset (2022)

Abstract: Event extraction (EE) is crucial to downstream tasks such as new aggregation and event knowledge graph construction. Most existing EE datasets manually define fixed event types and design specific schema for each of them, failing to cover diverse events emerging from the online text. Moreover, news titles, an important source of event mentions, have not gained enough attention in current EE research. In this paper, we present Title2Event, a large-scale sentence-level dataset benchmarking Open Event Extraction without restricting event types. Title2Event contains more than 42,000 news titles in 34 topics collected from Chinese web pages. To the best of our knowledge, it is currently the largest manually annotated Chinese dataset for open event extraction. We further conduct experiments on Title2Event with different models and show that the characteristics of titles make it challenging for event extraction, addressing the significance of advanced study on this problem. The dataset and baseline codes are available at https://open-event-hub.github.io/title2event.

## introduction event extraction (ee) is an essential task in information extraction (ie), aiming to extract structured event information from unstructured plain text. extracting events from news plays an important role in tracking and analyzing social media trending, and facilitates various downstream tasks including information retrieval @xcite , news recommendation system @xcite and event knowledge graph construction @xcite @xcite . figure 1 shows an example of extracting events from multiple news titles. based on the extracted events, news reporting the same event could be aggregated and sent to users to provide comprehensive views from different sources. event extraction can be categorized into two levels: sentence-level ee and document-level ee. sentence-level ee identifies event entities and attributes in a single sentence @xcite , while document-level ee aims to extract entities of the same event scattered across an article @xcite . in scenarios such as news aggregation, human-written news titles often preserve the core information of the news event, while news articles may contain too many trivial details. therefore, performing sentence-level ee on news titles is more efficient than document-level ee on news articles to aggregate relevant news. however, most ee models trained on traditional sentence-level datasets could not reach ideal performance when extracting events from titles @xcite @xcite @xcite @xcite . on the one hand, these models request predefined event types and a specific schema for each of them. each event schema consists of manually designed argument roles such as event trigger, person, time, and location. then the extraction of events will be decomposed into sub-tasks of extracting each argument role separately. despite the success in traditional ee, the manual design of specific event schema is costly and time-consuming, and the limited predefined event types could not handle a great variety of events emerging from the internet where most news titles nowadays are derived from. on the other hand, extracting events from chinese titles could be more challenging than traditional sentence-level ee such as the ace 2005 benchmark. 1 this is because some unique writing styles are observed in news titles on chinese social media, as shown in figure 2 . first, the writing of many titles does not strictly obey the correct grammar. for example, some titles will omit the agent when describing an action for brevity, while others may place the action before the first mention of the agent for emphasis. second, the role overlap problem, i.e., the same entity may play different roles in multiple events, usually occurs when the events in the text have certain associations with each other. although there are about 10% events in ace 2005 having this problem, it has not gained enough research attention for quite a long time @xcite . however, the role overlap problem is much 1 https://catalog.ldc.upenn.edu/ldc2006t06 considering the significance of title event extraction and a lack of corresponding benchmarks, we present title2event, a dataset with more than 42,000 chinese titles collected from the internet. in general, title2event has the following important features: we believe title2event could further facilitate current ee research in real-world scenarios. we experiment with different methods on ti-tle2event and analyze their performance to address the challenges of this task. ## dataset construction this section describes the process of data collection and annotation details. ## data analysis on title2event this section describes the statistics and characteristics of title2event from various perspectives. ## methods formally, given a sequence of tokens @xmath0, w 2 , . . . , w n >, open ee aims to output a list of triplets @xmath1, t 2 , . . . , t @xmath2, p i , o @xmath3, p i , o i denote the subject, predicate and object of the event respectively. the object of an event could be empty, and the total number of events per sentence m is not fixed. open ee can also be aligned with traditional ee task formulation by considering the predicate as the event trigger as well as a unique event type, while the subjects and objects both taken as event arguments. based on the task formulation, we first implement an unsupervised method using an existing toolkit. then, we split the task into trigger extraction and argument extraction, and implement different supervised methods on them. ## experiments we conduct experiments on title2event with the methods described in section 5 and analyze their performance. ## conclusions in this paper, we present title2event, a chinese title dataset benchmarking the task of open event extraction. to the best of our knowledge, ti-tle2event is the largest manually-annotated chinese dataset for sentence-level event extraction. we experiment with different methods and conduct detailed analysis to address the challenges observed in title2event, which are rather scarce in existing datasets yet common in real-world scenarios. we believe title2event could further facilitate advanced research in event extraction.