Towards Context-aware Normalization of Variant Characters in Classical C hinese Using Parallel Editions and BERT (2024)

Abstract: For the automatic processing of Classical Chinese texts it is highly desirable to normalize variant characters, i.e. characters with different visual forms that are being used to represent the same morpheme, into a single form. However, there are some variant characters that are used interchangeably by some writers but deliberately employed to distinguish between different meanings by others. Hence, in order to avoid losing information in the normalization processes by conflating meaningful distinctions between variants, an intelligent normalization system that takes context into account is needed. Towards the goal of developing such a system, in this study, we describe how a dataset with usage samples of variant characters can be extracted from a corpus of paired editions of multiple texts. Using the dataset, we conduct two experiments, testing whether models can be trained with contextual word embeddings to predict variant characters. The results of the experiments show that while this is often possible for single texts, most conventions learned do not transfer well between documents.

## introduction a lack of orthographic norms is a common feature of ancient writing systems. in the case of classical chinese, the written language of ancient china, this manifests prominently in a high number of variant characters (yitizi 異體字), that is in a broad sense, characters that are graphically distinct from each other but are used to write the same morpheme. for many downstream tasks such as full-text search, identification of parallel passages or the analysis of vocabulary, normalization of variant characters is desirable, as often, they are completely interchangeable and merely reflect arbitrary choices of copyists or woodblock carvers. however, there is also a class of quasi-variant characters that are only interchangeable in some contexts, with one variant often being preferred for one of multiple words foot_0 that can be written with the characters, more strongly associated with a particular word sense, or only found in certain compounds. for example, the two homophonous and etymologically related words "li to experience, to undergo" and "li calender" should, according to most dictionaries, be written with the two charac-ters「li 歷」 foot_1 and「li 曆」respectively. while the usage of these two characters in some editions of classical chinese texts agrees with this distinction, in others, we find either character used to write both words, or other variant forms such as 「li 厯」 replacing them. hence, a simplistic approach to normalization based on lists of variant characters must either risk conflating variants that were intentionally kept apart such as「li 歷」and「li 曆」, potentially impacting the understanding of the text, or ignore such cases, which could e.g. mean missing a parallel passage in two texts just because one scribe decided to use 「li 厯」for both "to experience" and "calender". the complex history of many characters further complicates matters, with specialised variants appearing and disappearing and characters being borrowed to write additional words over time (for a detailed overview, see @xcite . also, in china there traditionally was a taboo on using characters from the ruler's name, which was sometimes avoided by using existing variant forms or even coining new ones @xcite foot_2 . furthermore, the physical quality of texts might vary, and ocr systems as well as preferences of human transcribers can have an impact on which variant characters are presented to us in digital versions of the texts. for example, in figure 1 four characters are shown that represent the same word but are transcribed into two different, but very similar variant forms, which the dictionary of chinese character variants (dccv) (ministry of education, r.o.c) lists as having overlapping but not identical usage. one case matches fine variations in the writing style of the original text while the other appears to be a transcription error. thus, we anticipate a considerable amount of variability and noise in the data, and it is to be expected that there is no single normal form that "normalization" will result in. in order to cope with these difficulties, an intelligent system for character normalization should ideally satisfy the following conditions: towards the development of such a system, in this study, we have extracted a dataset of variant characters in context from a corpus of texts in two editions. in two experiments, we have tested whether contextual word embeddings can be used to train models to predict variant characters. ## related work given the fact that many quasi-variant characters are distinguished from each other by being preferred for specific words or word senses, we expect the problem to be highly similar to word sense disambiguation. for classical chinese, @xcite and @xcite have recently used bert for this task, with some success, although results for some characters were mixed. our approach of using a parallel corpus has already been successfully applied for learning word sense disambiguation, using alignments of translated sentences @xcite . @xcite have developed a dataset of loangraphs, i.e. characters used to write a word that is commonly written with another character, and used bert embeddings to detect them and predict the more usual character for writing the word in question. many variant characters originate from loangraphs @xcite , and the tasks share the problem of having to decide whether a character should be replaced by another character, so the study is very similar to the subject of this study. however, the authors use a handannotated dataset, which compared to ours, has the advantage of higher accuracy, and greater coverage of rare loangraph usage. on the other hand, the number of samples for each type of character is quite limited in comparison to our automatically derived dataset, and since there is no systematic annotation of an entire corpus, it is impossible to quantify how widespread the phenomenon is, and how the usage of loangraphs differs between texts. a somewhat comparable task for modern chinese is conversion from simplified to traditional characters, as one simplified character often replaces several traditional ones, such as「li 历」replacing both 「li 歷」and 「li 曆」. hence, machine learning techniques that take context into account have been investigated for this task @xcite . the problem of substituting one character with another, more common character is shared with spelling correction, for which bert has also been used @xcite ). an important difference to modern languages is of course that for classical chinese, there is no uniformly accepted normative authority, so it is not a-priori clear which character is "correct" in a given context. for western languages, normalization of historic spelling variations has been intensively stud-ied. @xcite gives an overview over different techniques, including machine learning approaches. @xcite and among others more recently @xcite introduce techniques to take context into account to differentiate words, similar to what is attempted here. however, a key difference between alphabetical languages and chinese is that techniques for the former often rely on edit distances between words, which is not directly transferable to chinese characters. ## building a dataset to the best of our knowledge, no comprehensive annotated corpus to train and test a system for variant normalization exists. however, there is a readily available data source which has high potential: works for which different prints or handwritten editions are digitally available. since the usage of variant characters can vary considerably between several editions of the same text, aligning them allows for the mining of variant characters. crucially, using an algorithm that will be described in detail below, we were able to automatically extract instances of variant characters that are used concurrently in one edition but correspond to only a single variant in another, giving potential cases of quasi-variant characters differentiated by one writer but not the other. ## conclusions in this article, we have demonstrated the general viability of using parallel editions and contextual embeddings for context-aware variant character normalization for classical chinese, by showing that a simple logistic model can be trained to predict which of more than one differentiated variants could replace a character in a given context. at the same time, our analysis has also revealed that meaningful variation of variant characters is quite a rare phenomenon, while in the digital editions surveyed, alteration between variant characters without meaningful difference is ubiquitous. this confirms the need for some form of variant normalization. in this regard, the failure of the model to learn to distinguish variants can actually be highly useful, because it can increase confi-dence that for those cases, a simple list based normalization approach does not run the risk of losing information. for those cases where the model was able to learn a differentiation, the results of the second experiment indicate that idiosyncratic usage of variant characters is quite common in the corpus. training a model on the conventions used by one writer of one edition of a document does often not generalize to other documents. taken together with the high overall number of variant characters, this confirms that copyists had considerable freedom in choosing variant characters, and highlights the importance of considering the transmission process when reading received versions of ancient texts. in terms of the two design goals for an intelligent system for variant normalization stated in the introduction, we have achieved more progress towards the first goal. as we have seen with the example of「er 尓」 and「er 爾」, the system has shown itself capable of discovering deliberate variation in a case where we would not expect it to occur based on consulting a dictionary. it could be a worthwhile endeavour to rerun the experiments with the full list of substitution profiles, i.e. without removing instances that are completely interchangeable according to the dccv, to see how widespread such cases are. towards the second goal of normalizing variants towards specialised forms, we have made significant progress only for a single case, 「li 暦」 and「li 歴」. in differentiating these two characters, our simple approach that did not require any manually annotated data achieved high accuracy. since the second experiment has shown an apparent lack of uniform conventions in the usage of many variant characters, further endeavours in this direction will first need to decide which conventions to adopt.