Classical C hinese Historical Event Detection Evaluation (2024)

Abstract: “Event detection involves identifying and extracting event information from natural language texts. The complex syntax and semantics of Classical Chinese, coupled with its limited usage, pose significant challenges for information extraction tasks on classical Chinese texts. At the 23rd China National Conference on Computational Linguistics (CCL 2024), we launched an evaluation task focused on the extraction of historical events from Classical Chinese. We used our constructed Classical Chinese Historical Event Logical Schema to identify event triggers and classify event types. The evaluation utilized the Classical Chinese Historical Event Detection Dataset (CHED), annotated from The Twenty-Four Histories corpus, with the aim of enhancing event extraction technologies and advancing the digital study of classical Chinese historical texts. The evaluation included two subtasks and attracted 28 teams, with 15 teams submitting valid results. In the subtask of trigger identification, the best-performing system achieved an Exact match score of 63.6%. In the subtasks of coarse-grained and fine-grained event type classification, the top systems achieved F1-scores of 84.5% and 81.4%, respectively.”

## introduction event detection is the process of identifying and extracting relevant event information from natural language texts. given the complex syntax and semantics of classical chinese, coupled with its limited usage scope, the task of information extraction for classical chinese texts remains a significant challenge. constructing high-quality datasets tailored to specific domains is essential for event detection tasks. while several high-quality event detection (ed) datasets exist for english and modern chinese, including ace 2005 @xcite , leven @xcite , maven @xcite , poe @xcite , and duee @xcite , classical chinese lacks such datasets due to its semantic complexity and unique historical context. large-scale datasets in english and modern chinese are not directly applicable to classical chinese ed. research in deep learning for event detection has explored historical event detection in classical chinese texts, such as the study of war events in shiji and zuozhuan @xcite (jiuming ji, 2015) @xcite . however, there still remain challenges such as sparse training data, complex text structures, semantic ambiguity. these issues highlight the need for constructing specific datasets and refining text features. to address these critical challenges and enhance the accuracy and efficiency of classical chinese event detection , we have developed the classical chinese historical event dataset (ched) @xcite . this dataset is intended to serve as a benchmark for advancing the development of event detection algorithms in classical chinese historical texts. the language and culture computing laboratory (lcc-lab) of beijing language and culture university conducted the historical event type extraction evaluation as part of the ccl 2024 conference. the task aimed to assess the performance of algorithms in detecting event types in classical texts and to promote research and development in classical chinese event extraction technology. the competition attracted 28 teams, with 15 submitting valid results. the best-performing systems achieved a match score of 76.3% in trigger identification and f1-score of 84.5% and 81.4% in fine and coarse event type classification tasks, respectively. the evaluation included two subtasks: trigger identification and event type classification, corresponding to the two steps of event type detection. we summarize the two main features as follows: ## task description in this section, we will provide a detailed description of the two subtasks. subtask 1: trigger identification. this subtask involves identifying event triggers within the text and marking their locations. our dataset is constructed based on the principle of minimal triggers, primarily using single-syllable words that best represent the occurrence of events within the text. the goal is to ensure that the identified triggers are concise yet accurately indicative of the events described. as shown in figure 1 , in the sentence "九月乙丑，太尉李修罢。" (in september of yi chou, general li xiu was dismissed.), the word "罢" (ba) means dismiss. therefore, the trigger in this sentence is "罢" (ba). subtask 2: event type classification. in this subtask, each identified trigger word is classified into an event type based on a predefined event type schema. the classification is conducted at both coarsegrained and fine-grained levels, allowing for a detailed understanding of the event's nature. coarsegrained event types include 9 major categories, and fine-grained event types include 67 subcategories. as shown in figure 1 , in the sentence "进军建德，擒贼帅赵桑干。" ( advancing to jiande, capturing the enemy leader zhao sanggan ), the word "进军" (advance) could represent the event of dispatching troops to jiande, and the word "擒" (capture) could represent the event of capturing the enemy leader zhao sanggan. therefore, the trigger "进军" (advance) corresponds to the coarse-grained event type "军事" (military) and the fine-grained event type "军事-备战-出兵" (military -prepare for war -send troops). the trigger "擒" (capture) corresponds to the coarse-grained event type "军事" (military) and the finegrained event type "军事-作战-俘虏" (military -combat -capture). 1 event type schema and dataset is released on https://github.com/nlpinblcu/ched2024 the construction of an event type schema in a given context should meet the criteria of comprehensive coverage, precise granularity, and high accuracy. as shown in figure 2 , the initial construction was based on word frequency statistics and semantic clustering of the translated corpus, and it was finalized through trial annotation and expert evaluation. ## evaluation metrics this section introduces the evaluation metrics for the two subtasks and the comprehensive assessment.