Period Classification in C hinese Historical Texts (2021)

Abstract: In this study, we study language change in Chinese Biji by using a classification task: classifying Ancient Chinese texts by time periods. Specifically, we focus on a unique genre in classical Chinese literature: Biji (literally “notebook” or “brush notes”), i.e., collections of anecdotes, quotations, etc., anything authors consider noteworthy, Biji span hundreds of years across many dynasties and conserve informal language in written form. For these reasons, they are regarded as a good resource for investigating language change in Chinese (Fang, 2010). In this paper, we create a new dataset of 108 Biji across four dynasties. Based on the dataset, we first introduce a time period classification task for Chinese. Then we investigate different feature representation methods for classification. The results show that models using contextualized embeddings perform best. An analysis of the top features chosen by the word n-gram model (after bleaching proper nouns) confirms that these features are informative and correspond to observations and assumptions made by historical linguists.

## introduction traditional studies of language change largely rely on close reading and require researchers to have solid training in understanding historical texts. as a wealth of historical data has been digitized, there has been a surge in recent years in using computational methods to investigate language change (e.g. @xcite @xcite . however, most of these studies are limited to a small number of indo-european languages. many languages, especially typologically very different ones, are understudied. one task that often precedes an investigation of language change is automatic text dating, also called period classification or diachronic text evaluation. in this task, a machine learning model's performance in capturing temporal information is evaluated. such tasks, along with datasets, exist for languages such as irish @xcite , polish @xcite , hebrew @xcite , and italian @xcite . to our best knowledge, there is no study focusing on dating historical texts in chinese. therefore, we first built a diachronic chinese biji (literally "brush notes") corpus with biji from four dynasties, spanning about 1300 years. this corpus contains around 33,000 paragraphs from 108 different biji labeled with dynasty information. then, we investigate two major questions: 1) can a classifiers successfully distinguish paragraphs in historical texts from different time periods? 2) do the features used for classification correspond to what we know from a linguistic perspective, and are the features informative for linguists? for the first question, we are interested in the classification systems' performance on chinese, since this is the first approach for this language, and written chinese is renowned for its lack of change from century to century @xcite . more specifically, we investigate the performance of different feature representation methods (char/word ngram, word2vec, contextualized embeddings models) in classifying the time periods of a text. these feature representation methods have been shown effective in previous studies of dating historical texts or lexical semantic change @xcite @xcite . our next question looks closer at the interrelation between text dating and language change. the classification results provide useful information, but they cannot directly tell us which features drive language change. since language change is an accumulating process and usually happens in neighboring time periods (dynasties), we use a task of distinguishing paragraphs from neighboring dynasties to target lexical change. we can then look at the features that ranked highest in this task and investigate whether a high relevance for this task corresponds to linguistic insights. results show that most of the top features are informative and fit into the observation made by historical linguists. the paper is structured as follows: we discuss related work in section 2. in section 3, we discuss the process of building a classical chinese biji dataset. we introduce the feature representation methods for classifying historical texts in section 4 and the experimental setup in section 5. we report results of the different feature representation methods and present an error analysis in section 6. in section 7, we analyze the classification for neighboring dynasties, focusing on important features drawn from the training data. we conclude in section 8. ## corpus of classical chinese biji a reliable dataset is the key to dating historical texts. taking into account the goal of our task, our dataset needs to satisfy the following requirements: biji are "fundamentally a set of discontinuous notes of description and reflection written by the biji author" @xcite . as suggested by @xcite , biji can be roughly divided into three types based on structural characteristics: 1) short stories, anecdotes, or biographies; 2) descriptions and reflections on a wide range of affairs; 3) recordings of a single subject such as history or poetry. this abundance of content that biji cover also leaves us with the issue of definition: biji and its boundaries with other genres are always controversial in academia. as we are more interested in those biji containing more informal language, we focus on the first type of biji, which contains more conversations. the first type of biji is normally novel-like, also called biji fiction. to have a professional data selection standard, during dataset creation, we choose biji according to the book series biji xiaoshuo daguan "the collection of biji fiction" published by shanghai classics publishing house @xcite . since these biji fiction collections are published based on dynasties, this also offers us trustworthy dynasty information about biji with decisions made by professional editors. given the difficulties involved in dating biji fiction @xcite , we use dynasties as our classes, for several reasons: first, since biji are a set of discontinuous notes written by an author, the writing time could span dozens of years. it is unrealistic to label a specific publication time. second, most classical chinese databases, such as scripta sinica and the chinese text project, label texts with their dynasty instead of the exact publication date. this makes our dataset consistent with the widely used corpus format; third, the most important reason is that labeling books with dynasties is sensible based on studies in historical linguistics @xcite @xcite , since dynasties are widely used as an anchor point for periodization. thus, dynasty labels can easily fit into historical linguists' periodization framework. we collect biji titles from four books in this series: biji fiction in tang dynasty and five dynasties, biji fiction in song and yuan dynasty, biji fiction in ming dynasty, and biji fiction in qing dynasty. all four books have a total of 140 biji and cover four dynasties sequentially: tang dynasty (618-907), song dynasty (960-1279), ming dynasty (1368-1644), and qing dynasty (1644-1911) 4 . we used the wiki section in the chinese text project 5 as the primary source of data. in this section, digitized texts are submitted by the site users, and they are freely available without copyright restrictions. we searched all biji titles mentioned in the books and successfully collected 127 of them. however, we deleted some biji for the following consideration: 1) biji without punctuation. the original biji were written without punctuation, and punctuation was generally inserted during modern editing. thus we take the presence of punctuation as a sign for careful editing and discard the nonedited biji. 2) biji with encoding and formatting issues (mainly paragraph segmentation errors) and a significant amount of annotations within the text. 3) biji which are not collections of short stories or anecdotes. for example, some biji mainly aim to explain concepts in previous classical work. a majority of the content is sentences citing from the classical work. such books clearly carry linguistic features from previous dynasties and are not good representations for the dynasty in which they were published. after applying these filtering rules, 89 biji were left. we found two problems with this dataset: 1) due to the issues described above, some biji from tang dynasty have been deleted, which causes an imbalance of the dataset. 2) since biji from ming and qing are usually lengthy, the collection of biji fiction covers only few biji from these dynasties. to balance the dataset, we used two open-access data sources, daizhige 6 and xuge shang 7 to collect additional data. for tang dynasty, collection of biji fiction already covers most of the existing biji in tang dynasty, so we looked in these new data sources for alternative editions of those biji with format issues. for ming and qing dynasty, we added books from the biji category and made sure that all these newly added biji satisfy our filtering rules. our final dataset includes 108 biji. since biji are usually a collection of short stories, we essentially treat each story as one instance in the corpus. we also notice that stories can be lengthy in biji from ming and qing dynasties; for such books, we treat each paragraph as a single instance. to balance the length of each instance, we discard those too long (larger than 1000 characters) or too short instances (shorter than 50 characters). given our selection process, the quality of our dataset may not be as high as professional classical chinese databases since we could not manually examine the correctness of each text. ## feature representation methods in this section, we present the feature representations that we use for time period classification. these feature representations can be categorized into three major types: n-grams, word embeddings, and contextualized embeddings features. ## experimental settings since biji show considerable differences in length, we split them into single paragraphs for classification. however, we need to avoid having instances from the same author distributed over training and testing data since the system may learn to classify authorship instead of time period. therefore, we perform five-fold cross-validation with a split based on books rather than individual paragraphs. in this way, we use different books for training and testing. we additionally balance the number of paragraphs from each dynasty in each fold as much as possible. for the baseline, we use the majority class, i.e., all instances are labeled as song dynasty. since our aim is to compare the performance of different feature representation methods, we need to investigate all models using the same machine learning algorithm. we compare three machine learning algorithms widely used in classification tasks: svm, logistic regression, and naive bayes. all the classification experiments use the python package scikit-learn @xcite . ## dynasty classification the next best feature type is n-gram features. we notice that word n-grams outperform character n-grams by more than 2 points in f score. a possible reason may be the increase of disyllabic words after the tang dynasty since disyllabic words increase in frequency from old chinese to middle chinese and from middle chinese to early modern chinese @xcite . therefore, word n-grams become more informative than simple character ngrams even when taking segmentation errors into consideration. surprisingly, using word2vec embeddings as features is not very effective in this task. the f score of word2vec features is 44.6, i.e., about 10 points lower than that of character n-grams. this implies that word2vec embeddings are not very effective in capturing language change outside a neural network architecture. figure 1 shows the confusion matrix of the svms with contextualized embeddings, averaged over the five folds. it shows that the most challenging time period is ming dynasty. less than 50 percent of its instances are predicted correctly. about 26% instances from ming dynasty are wrongly labeled as qing dynasty, and we see the same trend for the qing dynasty instances as well. notice that this similarity may be due to the property of language usage and similar content (see section 7). most instances from song and tang are labeled correctly, and we also see that they are seldom labeled as qing dynasty. this indicates that qing and ming dynasties are similar to each other, but song and tang are each very distinct from any other dynasty. we have a closer look at the 10 biji with the lowest accuracy, see ## neighboring dynasties classification we now turn to the second research question, investigating whether the features that our classifier uses correspond to linguistic insights. instead of classifying the four dynasties, we now look at the classification of neighboring dynasties, which is where the effects of language change should be the most visible. since contextualized embeddings features are difficult to interpret, we use n-gram features. however, a first trial showed that many of the high ranking features were person names, etc., which are inherently linked to a specific dynasty. thus, we decided to bleach proper nouns, i.e., we mask words tagged as proper nouns by jiayan toolkit. then, we use chi-square feature selection on the bleached n-gram model, and select the 1,000 highest ranked features for the investigation. to look into the features selected by the classifiers, we list the top 20 features extracted from the bleached n-gram models foot_10 , shown in another example concerns the features 皆 and 俱. for both words, the main sense is 'all (adverb)'. our results show that 皆 is useful in distinguishing instances from tang to song dynasty while 俱 is helpful for classifying song and ming dynasty. these results can be compared to a study by @xcite , who found that the usage of 皆 decreases time, and both 皆 and 俱 are substituted by 都 gradually in modern chinese. however, the change of 皆 and 俱 is not addressed by @xcite . in contrast, surprisingly, 都 does not appear in our top 20 features. the differences need to be investigated more closely in the future. we also see cultural changes reflected. for example, 相国 is a common title for prime minister from qin to tang dynasty, but after tang dynasty, 宰 相 is used to refer to the prime minister. however, even after bleaching proper nouns, there remain features containing indirect information about the dynasty. for example, 鬼 and 女 are the names of frequently used main characters in biji from qing dynasty. this makes them informative features, and they rank as the top features in the comparison of instances from ming and qing dynasty. ## conclusion & future work in this paper, we have approached the task of dating chinese historical texts and examined the performance of different feature representations and machine learning algorithms. the first public available diachronic biji dataset is created for tracking language change 16 . our main findings are that even though dating historical chinese texts is a challenging task for different types of feature representation methods, we still find informative features concerning language change, which correspond to historical linguists' observations. however, currently our methods only capture lexical change. in future work, we will investigate models for syntactic change.