D oc EE -zh: A Fine-grained Benchmark for C hinese Document-level Event Extraction (2024)

Abstract: Event extraction aims to identify events and then extract the arguments involved in those events. In recent years, there has been a gradual shift from sentence-level event extraction to document-level event extraction research. Despite the significant success achieved in English domain event extraction research, event extraction in Chinese still remains largely unexplored. However, a major obstacle to promoting Chinese document-level event extraction is the lack of fine-grained, wide domain coverage datasets for model training and evaluation. In this paper, we propose DocEE-zh, a new Chinese document-level event extraction dataset comprising over 36,000 events and more than 210,000 arguments. DocEE-zh is an extension of the DocEE dataset, utilizing the same event schema, and all data has been meticulously annotated by human experts. We highlight two features: focus on high-interest event types and fine-grained argument types. Experimental results indicate that state-of-the-art models still fail to achieve satisfactory performance, with an F1 score of 45.88% on the event argument extraction task, revealing that Chinese document-level event extraction (DocEE) remains an unresolved challenge. DocEE-zh is now available at https://github.com/tongmeihan1995/DocEE.git.

## introduction event extraction (ee) aims to detect events from text, encompassing both event classification and event element extraction. ee is an important task of information retrieval in natural language processing @xcite with a wide range of applications. for instance, it can automatically detect and analyze major events in news reports, providing timely information for decision-makers @xcite @xcite . in conclusion, advancements in event * corresponding author. extraction technologies and systems can benefit numerous domains. significant progress has been made in event extraction, particularly in the english domain. notable datasets such as ace2005 foot_0 have been extensively used for sentence-level event extraction, laying a foundation for numerous research studies. the tac kbp foot_1 event nugget dataset extends event extraction to a broader context by including event nuggets and their arguments. the rich ere (entities, relations, events) @xcite dataset further advances the field by offering a more detailed annotation schema and expanding the scope to document-level extraction. recently, the do-cee @xcite dataset has emerged as a comprehensive resource for document-level event extraction, offering wide coverage of event types and fine-grained annotations, greatly contributing to the advancement of this field. in contrast, chinese language processing predominantly relies on the chinese portion of the ace2005 dataset, which mainly focuses on event extraction at the sentence level. however, events are often spread across entire documents, resulting in event arguments being dispersed across multiple sentences. as depicted in figure 1 , identifying the "date" argument may require information from sentence @xcite , while understanding the "reason" may involve synthesizing data from sentences @xcite . this highlights the need for multi-sentence reasoning and modeling long-range dependencies, which go beyond the scope of sentence-level event extraction. therefore, advancing event extraction from individual sentences to entire documents is critically necessary. currently, there are few chinese datasets available for document-level event extraction, most of which focus on the financial domain, such as ch-穆里尼奥下课：热刺结束与名帅的短暂婚姻 [1] 当地时间周一，穆里尼奥被热刺解雇。在得知自己被解雇前，穆里尼奥像往常一样穿戴好装备准备开始训练。 on monday local time, mourinho was dismissed by tottenham. before learning about his dismissal, mourinho, as usual, geared up and prepared to start training. in the end, tottenham announced the dismissal of coach mourinho. after just 17 months of coaching tottenham, mourinho was let go. event type: resignation or dismissal event arguments: date resignee/dismissed employee age reason position approver organization term successor mourinho's dismissal: tottenham ends brief marriage with the renowned coach finann @xcite and duee-fin @xcite . moreover, a significant portion of the event arguments in these datasets are generic and used across multiple events, with specific arguments tailored to particular event types being relatively scarce. for instance, in chfinann, 60% of the arguments are general, and in duee-fin, this figure is 51%. this prevalence of generic arguments limits the ability of models to accurately capture the nuances of specific events, reducing the effectiveness of event extraction systems in identifying and differentiating between unique event types. in summary, existing datasets for chinese document-level ee fail in the following aspects: limited coverage of domains, and insufficient refinement of argument types. in our paper, we introduce docee-zh, a finegrained chinese dataset for document-level event extraction. docee-zh focuses on the extraction of the main event, following a one-event-perdocument approach. figure 1 illustrates an example of docee-zh. our contribution encompasses two key aspects: 1) high-interest event types: doceezh has curated 59 event types derived from various news categories, encompassing domains such as politics, military, entertainment, sports, and others. 2) fine-grained event argument types: docee-zh incorporates a total of 344 argument types, personalized event-specific arguments have been devised for each event type. in docee-zh, 86% of the event arguments are specific to individual events. ## related datasets in recent years, the field of event extraction has made significant advancements, particularly with the development of various datasets tailored for both sentence-level and document-level tasks. these datasets provide valuable resources for research and facilitate the development of more complex models. in this section, we overview the most representative chinese event extraction datasets and compare them with several important englishlanguage datasets, analyzing their differences and characteristics. ## constructing docee-zh our main goal is to construct a fine-grained chinese dataset to promote the development of event extraction from the sentence level to the document level. in the following sections, we will first introduce how to build event schema, and then discuss how to collect candidate data and label them through crowdsourcing. ## data analysis of docee-zh in this section, we conduct a comprehensive analysis of docee-zh to provide a deep understanding of the dataset and the task of document-level event extraction. overall statistics docee-zh contains annotations for 36,729 document-level events and 216,496 event arguments, averaging 5.9 arguments per document. notably, the event type awards ceremony exhibits the highest average number of event arguments per document at 11.6, while the financial crisis event type shows the lowest at 3.3. the average document length in our dataset is 1005 characters, encompassing an average of 23.36 sentences per document. this highlights the substantial amount of information available for analysis. each document is dedicated to a single event, focusing on the core event described in the news. given the high volume of daily news, our goal is to extract the most impactful events of interest to users. this approach facilitates more focused and efficient subsequent tasks such as event fusion and event reasoning. additionally, docee-zh contains the largest number of documents (36,729) and tokens (36,012k), reflecting its richness in textual resources. the dataset's extensive sentence count (817,085) and event arguments (216,496) further underscore its comprehensive coverage and the fine granularity of its event argument annotations, especially across such a large corpus. compared to ch-finann and duee-fin, docee-zh not only includes a broader variety of event types, but also features more detailed event argument annotations, capturing nuanced semantic information and supporting more complex event extraction tasks. while deie surpasses docee-zh in the total number of event arguments, docee-zh still excels in the precision and depth of its annotations, making it particularly suitable for handling more complex and diverse task scenarios. event type statistics figure 4 illustrates the distribution of the top 15 most common event types in docee-zh, representing the highest frequency of occurrences. these event types include categories such as sports competitions (9.8%), organization fines (9.4%), fires (6.9%), appointments/inaugurations (6.1%), and resignations/dismissals (5.3%), among others. our annotated data exhibits a long-tail distribution typical of real-world datasets, where class distributions are often uneven. notably, event types with over 500 instances constitute 36.2%, while those with over 200 instances represent 79.3%. further details can be found in the appendix. ## experiments on docee-zh in this section, we elucidate the challenges posed by docee-zh through comprehensive experimentation employing state-of-the-art models. we commence by delineating the experimental setup, followed by conducting experiments on event classification and event argument extraction tasks. finally, we discuss the implications of our findings and suggest potential directions for future development in chinese document-level event extraction. ## conclusion in this paper, we propose docee-zh, a documentlevel event extraction dataset, to foster the development of chinese document-level event extraction. docee-zh contains over 36,000+ events and 210,000+ arguments, and includes more finegrained event arguments. experiments demonstrate that chinese document-level event extraction remains an open problem.