Multi-Model Classical C hinese Event Trigger Word Recognition Driven by Incremental Pre-training (2024)

Abstract: “This paper addresses the task of identifying and classifying historical event trigger words in Classical Chinese, utilizing both small-scale and large-scale language models. Specifically, we selected the small-scale language model GujiBERT for intelligent processing of classical texts, and the large-scale language model Xunzi-Qwen-14b. Both models underwent continued pretraining and fine-tuning, resulting in GujiBERT-CHED-mlm and Xunzi-Qwen-14b-CHED. For the small-scale language model, we used a BiLSTM as the feature extraction module and a CRF as the decoding module, employing a sequence labeling paradigm to complete the evaluation experiments. For the large-scale language model, we optimized the prompt templates and used a sequence-to-sequence paradigm for evaluation experiments. Our experiments revealed that GujiBERT-BiLSTM-CRF achieved the best performance across all tasks, ranking fourth in overall performance among all participating teams. The large-scale language model demonstrated good semantic understanding abilities, reaching a preliminary usable level. Future research should focus on enhancing its ability to produce standardized outputs.”

## introduction ancient texts contain rich information on historical events, figures, geographical locations, and more, which are of significant value for historical research, cultural heritage preservation, and exploration of historical patterns. events, as the smallest granular units describing historical knowledge, are critical for the detection and organization of information in ancient texts. this facilitates the enhancement of knowledge services for ancient literature from a digital humanities perspective. due to the diverse content forms and complex language structures of ancient texts, information extraction typically relies on expert manual annotation and analysis, which is time-consuming and labor-intensive, making it difficult to scale for large volumes of ancient texts. in recent years, the rapid development of natural language processing (nlp), particularly the application of deep learning methods, has provided effective solutions for the automatic detection and identification of events in ancient texts. the objective of this task is to evaluate and further improve algorithmic models for detecting historical events in ancient texts. pre-training and fine-tuning are conducted on a prepared dataset of ancient historical texts to develop an optimal performance model. this model should possess two key capabilities: first, accurately identifying the corresponding trigger words and their locations, meaning it should precisely pinpoint the words that best represent the occurrence of a historical event in ancient texts and clearly mark their positions within the text; second, correctly determining the event type associated with each trigger word according to the ccl-ched official classification system for ancient text events, ensuring the accuracy and consistency of detection results. based on an extensive survey of research on chinese ancient text event information extraction, we employed both natural language understanding models and natural language generation models to complete this evaluation task. specifically, trigger word recognition and classification were treated as a sequence labeling task, with a domain-specific bert model selected for continued pre-training and fine-tuning to construct the trigger word recognition and classification model. additionally, to explore the applicability of large language models to this task, a base model tailored for intelligent processing of ancient chinese texts was selected for continued pre-training and fine-tuning. comparative testing was conducted to select the optimal prompts, enabling the large language model to generate character indices and trigger word categories for the given text. figure 1 illustrates the overall technical approach employed in this study. through a series of experiments, we discovered that smaller language models, exemplified by bert, outperform in terms of prediction accuracy and the standardization of output. although large language models have advantages in terms of ease of use, the standardization of their output is challenging to control, necessitating further manual processing. overall, large language models are less suitable for tasks that require high precision in information extraction. 2 related work currently, there are three main methods for event extraction from ancient chinese texts: patternmatching-based event extraction, statistical machine learning-based event extraction, and deep learningoverall, pattern matching can achieve good results for specific types of texts, but its transferability is poor. statistical machine learning methods heavily depend on the richness of training samples and perform poorly in parsing sentences that did not appear in the training corpus. currently, deep learning methods have become mainstream. among these, models based on the bert architecture have achieved c c l 2 0 2 4 good results under the sequence labeling paradigm. with the rise of transfer learning techniques, methods based on reading comprehension and generative models have also been applied to event information extraction from ancient texts, providing new perspectives for event extraction research. in addition to changing task paradigms and designing, comparing, and analyzing different architectures of deep learning models, many scholars also aim to provide the most basic and foundational optimizations for various natural language processing tasks by constructing domain-specific pre-trained language models (dongbo @xcite . by the end of 2022, the emergence of chatgpt once again revolutionized the paradigm of natural language processing tasks. subsequently, large language models continued to innovate, leading to the creation of models tailored for different vertical domains. this includes the "xunzi" series of models designed for intelligent processing of classical chinese. however, as of now, there have been no studies observed that apply large language models to the extraction of trigger words in ancient texts. given the extensive research showing that domain-specific continued pre-training of deep pretrained language models can effectively enhance the model's performance on downstream tasks @xcite . related studies also indicate that sequence labeling models still maintain certain advantages in specific task scenarios. furthermore, according to @xcite , it is evident that within the sequence labeling task paradigm, the bert-bilstm-crf model achieves the best performance. considering the current lack of relevant practices involving large language models in the extraction of trigger words from ancient texts, this paper proposes constructing incremental pre-training for bert specifically for the evaluation task. it combines this with bert-bilstm-crf to enhance the effectiveness of event information extraction. additionally, attempts are made to incrementally pretrain and fine-tune large language models, exploring their applicability in extracting trigger words from ancient texts.