CHED : A Cross-Historical Dataset with a Logical Event Schema for Classical C hinese Event Detection (2023)

Abstract: “Event detection (ED) is a crucial area of natural language processing that automates the extrac-tion of specific event types from large-scale text, and studying historical ED in classical Chinesetexts helps preserve and inherit historical and cultural heritage by extracting valuable informa-tion. However, classical Chinese language characteristics, such as ambiguous word classes andcomplex semantics, have posed challenges and led to a lack of datasets and limited research onevent schema construction. In addition, large-scale datasets in English and modern Chinese arenot directly applicable to historical ED in classical Chinese. To address these issues, we con-structed a logical event schema for classical Chinese historical texts and annotated the resultingdataset, which is called classical Chinese Historical Event Dataset (CHED). The main challengesin our work on classical Chinese historical ED are accurately identifying and classifying eventswithin cultural and linguistic contexts and addressing ambiguity resulting from multiple mean-ings of words in historical texts. Therefore, we have developed a set of annotation guidelinesand provided annotators with an objective reference translation. The average Kappa coefficientafter multiple cross-validation is 68.49%, indicating high quality and consistency. We conductedvarious tasks and comparative experiments on established baseline models for historical ED inclassical Chinese. The results showed that BERT+CRF had the best performance on sequencelabeling task, with an f1-score of 76.10%, indicating potential for further improvement. 1Introduction”

## introduction event detection (ed) is a significant research area in natural language processing (nlp). the ed task mainly includes two steps. firstly, recognizing and labeling triggers (words that best represent the occurrence of events) in the text, and secondly, determining the event types to which triggers belongs. for example, in the sentence "九月乙丑，太尉李修罢。" (in september of yi chou, general li xiu was dismissed.), the word "罢" (ba) means "dismiss". therefore, the trigger in this sentence is "罢" (ba), and we label this sentence as a "职位-官位-免职" (position-official position-dismiss from a position) event triggered by the word "罢" (ba). constructing high-quality datasets for specific domains is critical for ed tasks. several high-quality ed datasets exist for english and chinese, such as ace 2005 @xcite , @xcite , maven @xcite , poe @xcite and duee @xcite . however, classical chinese lacks such datasets due to complex semantics and special era. large-scale datasets in english and modern chinese are not directly applicable to classical chinese ed. the current research on ed in classical chinese is limited by the lack of high-quality datasets that are specific, systematic, and scalable. to address these crucial issues and enhance the accuracy and efficiency of classical chinese ed, we have constructed the classical chinese historical event dataset (ched). this dataset has the potential to serve as a benchmark for developing and evaluating ed algorithms for classical chinese historical texts. the hierarchical and logical event schema of the ched can be extended and adapted to other nlp domains, making it a valuable resource not only for nlp researchers but also for scholars in other humanities fields. moreover, ched offers a unique historical perspective for exploring ancient societies, enhancing our comprehension of their cultures and interconnections. it also supports the digital humanities research and helps preserve cultural heritage through the study of classical chinese texts. during the construction of our dataset, we encountered three primary challenges: 1) developing an event schema that could encompass the majority of events described in classical chinese literature; 2) accurately identifying and classifying events within cultural and linguistic contexts while accounting for the ambiguity resulting from multiple meanings of words in historical texts; 3) ensuring consistent annotation results, which was essential throughout the entire dataset construction process. to address these challenges, we proposed several approaches. one such approach involved subjecting the processed data and preliminary event schema to trial annotation and expert review. through several revisions and validations, we constructed a hierarchical and logical event schema with fine granularity, consisting of 9 major event categories and 67 subcategories that cover significant events in ancient chinese history. the 9 major categories of events include life, position, communication, movement, ritual, military, law, economy, and nature. the complete event schema has been placed in the appendix a, as shown in figures 11 and 12 . in addition, we have annotated a total of 8,122 valid sentences. to ensure further accuracy, our annotators possessed extensive knowledge of classical chinese and actively sought expert opinions while constructing the dataset. multiple cross-validation were also conducted, yielding an average kappa coefficient of 68.49%, which denotes a high level of consistency and quality. additionally, we conducted various tasks and comparative experiments on established baseline models for historical ed in classical chinese. the outcomes indicated that bert+crf exhibited the highest performance on sequence labeling task, achieving an f1-score of 76.10%. we conclude three main contributions as follows: 1) we constructed the ched, which provides a rich cross-historical data foundation for classical chinese ed, making it a valuable resource for scholars and researchers. the dataset contains 8,122 valid sentences; 2) we proposed a hierarchical and logical event schema, which has a fine-grained structure that can be adapted more effectively to other nlp domains; 3) we excavated a unique and profound historical perspective from the ched, promoting the advancement of digital humanities research. ## related work in the realm of event detection (ed) tasks in deep learning, sparse and imbalanced training data, complex text, and semantic ambiguity still pose problems, highlighting the importance of dataset construction and feature extraction through text refinement processing. a high-quality dataset is essential for ed tasks. it should be large enough to support various learning algorithms, has high accuracy and consistency in labeled data, and contains diverse event types. several high-quality annotated ed datasets have been constructed, including the widely used english dataset ace 2005 @xcite , the legal ed dataset @xcite , the large-scale crossdomain ed dataset maven @xcite , the electrical power ed dataset poe @xcite and the chinese event dataset duee @xcite based on real-world scenarios. while many studies have summarized the primary methods of chinese ed based on literature, classical chinese field ed faces challenges due to differences in context and expression of historical texts. there have been studies using deep learning methods to investigate historical ed in classical chinese texts @xcite , such as researching the war events in the zuozhuan (左传). for example, the roberta-crf model was established @xcite , and pattern matching and crf models were used to extract events from the zuozhuan (左传) (zhongbao liu, 2020). additionally, mixed techniques using information extraction have been applied to classical chinese texts, including entity recognition and event extraction, with the extracted information being visualized using electronic charts @xcite . furthermore, studies have been conducted on extracting historical events and event elements from shiji (史记) and zuozhuan (左传) @xcite . however, these studies have only produced coarse-grained event type constructions, mostly focused on a single text and based on relatively small dataset sizes. ## event schema construction the construction of event types in a given context should fulfill the criteria of comprehensive coverage, precise granularity, and high accuracy. to achieve these goals, we mainly carried out work in four aspects, as shown in figure 1 : 1) word frequency statistics; 2) word semantic clustering; 3) trial annotation; 4) expert evaluation. eventually, we constructed an event schema that includes 9 major categories and 67 subcategories. figure 2 depicts the structure of one of the major categories, position. we assume that the words with higher frequency in the text reflect the main content and central theme of the text, which is closely related to the event types. therefore, it is necessary to conduct comprehensive word frequency statistics on the text to ensure the coverage of event types. we selected the translated works of the twenty-four histories from niutrans foot_1 and used hanlp foot_2 for basic word segmentation and part-of-speech tagging on the corpus, and conducted word frequency statistics based on the results. after removing stop words and irrelevant part-of-speech tags, we analyzed the word frequency statistics results of nouns, verbs, and gerunds. we discovered that certain high-frequency words, such as "进攻" (attack), could serve as event types for historical events in classical chinese. semantic clustering analysis was further conducted on words to automatically classify similar semantic words, aiming to provide more refined classification references for the construction of classical chinese event types. we used fasttext foot_3 to generate vector representations for each word and the k-means clustering algorithm to cluster words with high semantic similarity. based on the analysis, and inspired by the ace @xcite , meven @xcite and other datasets, we preliminarily summarized the classical chinese historical event types, including 15 major categories and 73 subcategories. to evaluate the actual event coverage in real-world texts, we randomly selected 15 volumes from the benji (本纪) and liezhuan (列传) sections of each book in the corpus of the twenty-four histories pro- the twenty-four histories improve the consistency and quality of annotations optimize the event schema trial annotation total @xmath0. historical records 2. hanshu ...... 24. history of ming dynasty formulation of annotation guidelines alignment of classical chinese and its translation formal annotation ched total @xmath1 figure 3 : the entire annotation process from raw corpus to dataset is presented, including two main stages, as well as the measures taken to ensure the quality of annotation. in addition, to ensure the accuracy of classical chinese historical event types and avoid personal subjective bias, we invited experts and students with linguistic and computer science backgrounds to evaluate our event types. after these efforts, we constructed the final event schema for ched. ## annotation process we used figure 3 to illustrate our process. ## data analysis in this section, we mainly introduce the scale and distribution of the dataset, as well as the phenomenon of data sparsity that has been observed, and provide possible explanations for it. ## conclusion and future work in conclusion, we have constructed a hierarchical and logical schema for classical chinese events and used it to create the ched based on the twenty-four histories corpus. the ched can effectively facilitate the advancement of digital humanities research by providing a unique and profound historical perspective. despite encountering various challenges during the construction of the dataset, we ensured the consistency and quality of the annotations. we assessed the effectiveness and quality of the dataset by testing it against several baselines and calculating kappa scores, and we obtained satisfactory results. nevertheless, there is scope for further enhancement, and our future work will concentrate on expanding and optimizing the dataset to meet a wider range of application needs. our dataset is a valuable resource not only for natural language processing but also for classical literature and cultural studies. furthermore, it makes a significant contribution to the field of event detection in classical chinese, and we anticipate that it will inspire further research and exploration.