Two Approaches to Diachronic Normalization of P olish Texts (2024)

Abstract: This paper discusses two approaches to the diachronic normalization of Polish texts: a rule-based solution that relies on a set of handcrafted patterns, and a neural normalization model based on the text-to-text transfer transformer architecture. The training and evaluation data prepared for the task are discussed in detail, along with experiments conducted to compare the proposed normalization solutions. A quantitative and qualitative analysis is made. It is shown that at the current stage of inquiry into the problem, the rule-based solution outperforms the neural one on 3 out of 4 variants of the prepared dataset, although in practice both approaches have distinct advantages and disadvantages.

## introduction this paper discusses two solutions to the problem of diachronic normalization, that is, the task of determining contemporary spelling for a given historical text. diachronic normalization may concern the writing of individual words, punctuation, hyphenation, or separation of tokens. we believe that the methods described in this paper may be useful for linguistic research on historical texts. a practical use case for our work is to facilitate fulltext search in historical texts -a query written in contemporary spelling may trigger a search for historical variants through the use of reversed-order diachronic normalization. similar experiments, for text normalization in a speech synthesis system from text, were described in @xcite . those authors claim that text normalization remains one of the few tasks in the field of natural language processing where handcrafted rules may yield better results than machine learning. this is due to the following reasons: in our experiments, we compare the results of a rule-based approach with one based on machine learning. the rule-based approach relies on a set of handcrafted rules to normalize text. in the ml approach, we train a supervised normalization model on the basis of a corpus of polish books for which both historical and current spellings are available. ## related work the first attempts at rule-based diachronic normalization used for historical text in english were described by @xcite and @xcite . similar studies were conducted for german @xcite . there, context rules operated at the level of letters instead of words. the normalization rules may be derived from corpora, as @xcite showed for german. diachronic normalization may be also performed using a noisy channel model, as described by @xcite using the example of old hungarian texts. research on diachronic normalization has also been conducted for portugese @xcite , swedish @xcite , slovene @xcite , spanish @xcite , and basque @xcite . @xcite surveys historical spelling normalization methods for eight languages. he reports word-level accuracy for the evaluated systems. he claims that using cer is not justified, because it strongly correlates with wer for systems showing reasonable accuracy. @xcite use bi-directional lstms and multi-task learning to normalize texts in early new high german. their dataset consists of 44 texts from the aselm corpus. he model presented is evaluated with respect to word-level accuracy. @xcite discuss the problem of evaluating historical text normalization systems. they emphasize the necessity of reporting accuracy for unobserved tokens and recommend confronting the normalization systems with a simple baseline that memorizes training samples. @xcite present an automatic method for diachronic normalization of polish texts. the proposed method uses a formal language to model diachronic changes. @xcite introduce a method for finding spelling variants in a diachronic corpus using word2vec. ## data training and evaluation of a diachronic normalizer requires a corpus of texts that preserve historical spelling along with their contemporized counterparts. as our aim is the normalization of polish prose, we decided to collect texts for our corpus from two sources. texts that preserve historical spelling were drawn from the polish edition of the wikisource project (wikimedia foundation, 2023), which provides proof-read transcriptions of printed books that have fallen into the public domain, encoded in the mediawiki format. for contemporized texts, we used wolne lektury (modern poland foundation, 2023), a digital library that aims to deliver new editions of school readers, free of charge. although both sources encompass a wide variety of texts, ranging from poems and works of philosophy to dictionaries and historical documents, we narrowed our attention to novels, to facilitate the process of matching the original texts from wikisource to their contemporized versions in wolne lektury with the use of metadata information available for novels in both sources. we initially sourced 308 novels from wikisource and 279 from wolne lektury. ## discussion after performing a qualitative analysis of the results obtained using rule-based and neural normalization models, we observed for the neural networks: (1) flexibility in context interpretation, i.e., the ability to adapt to various contexts and understand linguistic nuances; (2) recognition of irregular patterns, i.e., the ability to identify and process non-standard and complex language forms; the spelling changes -from egzystencya to egzystencja, anglja to anglia, etc. -were part of the polish orthographic reform of 1936. this reform was aimed at simplifying and standardizing the polish language's spelling. it introduced several changes, including the replacement of the letter 'y' with 'j' or 'i' in certain contexts, and the introduction of the letter 'j' in place of 'i' in some cases to better reflect pronunciation. this reform significantly influenced the modern polish language, aligning it more closely with its phonetics. ## conclusion this paper has discussed two approaches to the diachronic normalization of polish texts. we presented transducers, a rule-based solution that relies on a set of deterministic, handcrafted rules, and a family of neural normalization models based on a text-to-text transfer transformer architecture. the experiments that we conducted showed that the rule-based approach is effective in the diachronic normalization task. however, the neural model surpassed the rule-based solution in the case of a test set that consists of a separate set of novels and excludes samples that should remain unchanged in the normalization process. as the presented research is preliminary in nature, there are several promising directions to explore, which we are committed to doing in the near future. among other ideas, we want to test the performance of hybrid solutions combining both approaches in distinct ways. we are also considering testing different model architectures and conducting further work on improving the quality of the training data used for the neural approach, as we believe it has the potential to eventually surpass the rule-based solution in most typical scenarios.