Unlocking Transitional C hinese: Word Segmentation in Modern Historical Texts (2023)

Abstract: This research addresses Natural Language Processing (NLP) tokenization challenges for transitional Chinese, which lacks adequate digital resources. The project used a collection of articles from the Shenbao, a newspaper from this period, as their study base. They designed models tailored to transitional Chinese, with goals like historical information extraction, large-scale textual analysis, and creating new datasets for computational linguists. The team manually tokenized historical articles to understand the language’s linguistic patterns, syntactic structures, and lexical variations. They developed a custom model tailored to their dataset after evaluating various word segmentation tools. They also studied the impact of using pre-trained language models on historical data. The results showed that using language models aligned with the source languages resulted in superior performance. They assert that transitional Chinese they are processing is more related to ancient Chinese than contemporary Chinese, necessitating the training of language models specifically on their data. The study’s outcome is a model that achieves a performance of over 83% and an F-score that is 35% higher than using existing tokenization tools, signifying a substantial improvement. The availability of this new annotated dataset paves the way for refining the model’s performance in processing this type of data.

## chinese word segmentation chinese word segmentation (cws) has been a subject of prolonged debate within the field of nlp. for specific tasks and datasets, adhering to the character level proves to be a more suitable approach, yielding higher-performance results and simplifying the process. thus, in certain scenarios, it is deemed less essential to perform word segmentation. for instance, a study by @xcite strongly asserts that with the advent of neural methods in nlp, cws is gradually becoming an irrelevant or even detrimental step in the nlp pipeline. in the context of computational methods in the humanities, the task of addressing cws continues to be an area of significant interest. in fact, this crucial task enables multiple analyses of chinese text, empowering researchers and practitioners to extract valuable insights and knowledge. to accomplish this, it becomes essential to have a tokenizer that is specifically tailored and adapted to the data being processed. the effectiveness of the tokenizer greatly impacts the accuracy and efficiency of subsequent nlp tasks on chinese texts. over time, cws has been extensively studied, leading to the development of sophisticated systems capable of achieving near-perfect results. some of these systems boast an impressive f-score close to 100%, indicating a high level of precision and recall in identifying word boundaries within chinese texts. these remarkable achievements in segmentation accuracy further enhance the overall performance of nlp applications when working with chinese language data. during the early stages of cws, the predominant methods employed were lexicon-and rule-based approaches, such as forward maximal matching, reverse maximal matching, and least word cut. while these techniques were relatively straightforward to implement, they suffered from low accuracy in segmenting words effectively. to address this challenge, from a machine learning perspective, cws started to be approached as a sequence labeling task. this method involved predicting whether each input character should be separated from its neighboring characters @xcite . in the 2000s, many researchers turned to conditional random fields or maximum entropy markov models to tackle this task. as the field progressed, approaches utilizing supervised or unsupervised features emerged @xcite , contributing to the state-of-the-art performance in cws. these techniques made significant strides in improving segmentation accuracy and efficiency. a major turning point came around 2013 when researchers began incorporating neural networks into their work, revolutionizing the research landscape for cws. whether adopting feed-forward, recurrent, or convolutional neural network architectures, these approaches offered substantial benefits, particularly in reducing the need for labor-intensive data engineering tasks @xcite @xcite . despite the promise of neural network-based methods, early trials did not consistently outperform non-neural systems. refining these neural approaches became necessary to achieve their full potential in cws tasks. however, over time, researchers made remarkable progress, and neural network-based methods eventually surpassed the capabilities of their non-neural counterparts, leading to significant advancements in the field. in the realm of nlp, the emergence of transformers marked another significant milestone. recent advancements in the field utilized a customized transformer model for sequence tagging, resulting in the achievement of state-of-the-art performance @xcite . however, a common issue with many nlp models is that they are often trained and evaluated on contemporary datasets, limiting their applicability to historical or ancient texts. cws is a field with a long history in chinese nlp, resulting in the creation and continual evaluation of numerous datasets, like the ones listed below. it comprises diverse types of chinese text, including news articles, fiction, and academic papers from various sources, such as ckip (academia sinica) and city university of hong kong for traditional chinese, and beijing university and microsoft research (china) for simplified chinese. recently, to address the need for models capable of handling ancient chinese texts, the evahan 2022 dataset foot_2 was created. it consists of annotated ancient chinese text developed as part of the evahan project, a collaborative effort by several chinese universities to build resources for ancient chinese nlp. the ancient chinese language dates back to around 1000bc-221bc. thanks to the impressive performance of the models trained on these contemporary data, many off-the-shelf solutions are widely adopted in the context of digital humanities research, facilitating the exploration and analysis of chinese texts from various time periods and genres: these tools were designed with a strong emphasis on achieving high performance and userfriendliness. nevertheless, it remains crucial to evaluate their performance when applied to specific datasets and chinese language variants. ## dataset creation within the scope of our project, our primary objective was to evaluate various tokenizers on our specific dataset. the aim was twofold: first, to estimate the performance of off-the-shelf tokenizer tools, and second, to address the possibility that these readily available solutions might not yield suitable results for our unique data. in such a scenario, we planned to develop a custom model tailored to the requirements of our dataset. in order to delve deeper into the development of the language during the particular period under examination, we engaged in manual tokenization of sentences extracted from the shenbao. this involved carefully segmenting the text into individual words. we proceeded in two steps: for the first round, we aimed at annotating a large select of texts published between the years 1872 and 1947. three annotators were trained on a sample of the selected corpus, after which they annotated separately 741 articles. for the second round, we selected 52 articles published between the years 1872 and 1907 that were annotated, curated, and used for evaluating our model. by tokenizing these historical articles, we sought to gain relevant insights into the linguistic patterns, syntactic structures, and lexical variations present in the chinese language during that specific historical timeframe. this approach allowed us to address the challenges posed by the absence of modern linguistic resources and the particularities that might arise in historical chinese texts. through this comprehensive evaluation and analysis of word segmentation methods on our wellcurated dataset, we anticipated obtaining a clearer picture of the tokenizer's efficacy in handling historical chinese texts. ultimately, this process played a vital role in shaping our subsequent decisions regarding the selection of the most appropriate tokenizer or the development of a custom model best suited to our research objectives and historical data. to achieve this goal, we engaged two groups of three distinct annotators who were tasked with tokenizing the documents based on our guidelines. each annotator was asked to add basic punctuation to mark the end of sentences. a blank space was introduced between tokens when their part of speech was different. only place names, job titles, proper names of persons were left unspeparated. additionally, for the second round, a separate individual -a historian with high skills with classical and modern chinese -was entrusted with curating the annotated data. this systematic approach ensured that the word segmentation process was carried out consistently and according to the specified guidelines, while also guaranteeing the quality and accuracy of the curated dataset. upon the successful completion of the annotation and curation processes, we obtained a carefully curated dataset, which we will call enp-tok foot_5 , documented in ## experiments by using this new annotated dataset, we have effectively assessed the performance of off-the-shelf tokenizers, particularly when applied to texts from the period of interest. we conducted a thorough evaluation of the five most commonly used word segmentation tools. the results of this evaluation are summarized and presented in ## model training to validate this hypothesis, we conducted extensive model training using three distinct datasets: ckip, evahan, and enp-tok. our aim was to assess the significance of both the training and inference datasets in shaping model performance. beyond the influence of the cws training data, we also delved into the impact of the language models employed to train these models. all results presented in this section are averaged over 10 random initializations. initially, we used the bert @xcite ) language model for chinese. because of its ability to deal with simplified and traditional chinese (automatically translated), and its popularity. in view of the results presented in based on the results, presented depending on the quality of the original document in the findings suggest that incorporating data from historical chinese sources can enhance the performance of the models for the task at hand. however, to fully explore the potential benefits, it is essential to consider the impact of using language models pre-trained on historical data @xcite , which may provide a more contextually relevant starting point for the training process. ## conclusion our study aimed to identify the most effective methods for cws, specifically focusing on historical texts written in transitional chinese. we created a novel, manually annotated dataset, derived from the shenbao. this dataset provided us with a rich linguistic resource, allowing a comprehensive analysis of word segmentation methods for historical chinese texts. the evaluation of five commonly used off-theshelf tokenizers revealed that while these models offer ease of use and quick implementation, their performance on our historical dataset was significantly lower than on contemporary datasets. this finding underscores the importance of validating models before using them for larger-scale analyses, to ensure the reliability of their outcomes. further investigation with the hanlp library and the fasthan model demonstrated notable performance improvements, suggesting potential advancements in cws using contemporary, state-of-the-art models without the need for additional manual annotations. however, the disparity between these models and the language models used for training makes it challenging to predict which models would yield superior results on different datasets. our exploration of different datasets and language models for training our segmentation model led us to a critical realization: using high-quality, relevant training data closely aligned with our target texts allowed us to achieve the best performance. this insight emphasized the importance of prioritizing data quality over quantity, and that customization of the training data to suit the unique characteristics of the target texts can lead to more accurate and reliable results. ultimately, our efforts culminated in the development of a model that achieved an impressive f-score of over 83% , a significant leap compared to using the jieba tokenizer. while these results approach the levels attainable on contemporary datasets, there are several avenues for further improvement. the availability of enp-tok dataset opens up new possibilities for enhancing the model's performance, ranging from linguistic analysis of the annotated dataset to optimization at the training level by incorporating specific language models. the improved cws of transitional chinese has significant implications for historians conducting research on china. the better the cws, the more accurate and efficient the text analysis becomes. this improved accuracy can help streamline the research process, allowing historians to accurately segment and analyze large volumes of text, thus opening up new avenues of inquiry and enabling the exploration of previously unmanageable datasets. as most of the texts published in the period under consideration -including periodicals, books, diaries, etc. -were written in a style that closely matches the various forms on which we have trained our model, the tokenizer we propose can serve to unlock vast corpora of historical sources on which existing models fail. this can lead to more nuanced understandings of the wide range of historical texts made available through digitization, offering deeper insights into china's history. this study illuminates the importance of dataset relevance and quality in achieving optimal results for cws, particularly for transitional chinese texts. the insights and methods derived from this study contribute significantly to the field of historical chinese text analysis and provide valuable tools for historians working with these rich and complex linguistic resources.