Normalizing without Modernizing: Keeping Historical Wordforms of M iddle F rench while Reducing Spelling Variants (2024)

Abstract: Conservation of historical documents benefits from computational methods by alleviating the manual labor related to digitization and modernization of textual content. Languages usually evolve over time and keeping historical wordforms is crucial for diachronic studies and digital humanities. However, spelling conventions did not necessarily exist when texts were originally written and orthographic variations are commonly observed depending on scribes and time periods. In this study, we propose to automatically normalize orthographic wordforms found in historical archives written in Middle French during the 16th century without fully modernizing textual content. We leverage pre-trained models in a low resource setting based on a manually curated parallel corpus and produce additional resources with artificial data generation approaches. Results show that causal language models and knowledge distillation improve over a strong baseline, thus validating the proposed methods.

## introduction normalizing orthographic variations of historical texts is a crucial task for digital humanities. it allows for both conservation and easy consultation of ancient documents. archives conservation is a task conducted mostly manually by trained experts who could benefit from advances in automatic normalization and modernization of historical texts. previous work has mainly focused on transforming the spelling of historical texts into their modern counterpart in order to apply computational tools @xcite @xcite . however, reducing orthographic variation while keeping historical spelling for a given era is the cornerstone of reliable diachronic studies. yet, variations in wordforms lead to data scarcity and the lack of corpora containing ledictz jour, vendredy 28 octobrix 1547, en l'evesché ledit jour vendredi 28 octobris 1547 en l'évêché said day friday october 28 1547 in the bishop's house l'on fasse respondre aut president de sadicte lectre l'on fasse répondre au président de sadite lettre we answer to the president about his letter ayme richard, habitant et ferratier, filz de feu thivent richard, de sonzier aimé richard habitant et ferratier fils de feu thivent richard de scionzier aimé richard inhabitant and ironworker son of the late thivent richard of scionzier figure 1 : segments sampled from our middle french corpus in their original form (top, colored), normalized version (middle, in black, normalized words underlined) and english translation (bottom, italic). in the first sample, the bishop's house, translation of evesché in this example, refers to the house inhabited by the previous bishop which was converted into a prison. during the process of digitization and modernization of historical texts, researchers in digital humanities could benefit from the data produced during each steps of this process. in this work, we focus on the normalization without modernization of archives from the 16th century written in middle french. to the best of our knowledge, there are no datasets of archival documents from this era and language containing non-normalized wordforms aligned to their normalized counterparts. we first manually normalize and align a set of historical texts with their spelling-normalized forms, keeping the syntactic and semantic content identical to the originally authored manuscripts. second, we investigate how to automatically normalize these texts inspired by low-resource machine translation approaches. empirical results show that two orthogonal solutions, namely sequence-level knowledge distillation @xcite @xcite and language model transfer learning combined with back-translation @xcite , allow to produce reliable synthetic parallel corpora. our summarized contributions are: (i) we pave the way towards archival documents conservation with expert targeted normalization and spelling variants reduction, keeping historical wordforms without modernization, (ii) we show evidence that leveraging pre-trained models and synthetic data improves normalization performances over a strong baseline, (iii) we release a middle french handcrafted parallel corpus aiming at orthographic normalization along with several fine-tuned models. 1 the remainder of this paper is organized as follows. we briefly introduce the background work in section 2. we describe the data production methods in section 3, followed by our experiments and results in section 4. finally, conclusions and future work are presented in section 5. ## background work this section presents previous work on normalization and modernization of historical texts, followed by background work on synthetic data generation methods. finally, we formalize our spelling normalization as machine translation approach. ## historical documents normalization models we trained individual models based on the three parallel datasets described in section 3, namely hand-crafted, distillation and generation. all normalization models were based on the pre-trained model m2m100 with 418m parameters @xcite . the causal language model used to generate target synthetic data from prompts was bloomz @xcite with 560m parameters. the training procedure for normalization models using synthetic datasets followed a two-step process. first, continued training @xcite with mixed synthetic and hand-crafted corpora. during this first step, we applied either sequence-level knowledge distillation or synthetic data generation followed by back-translation, leading to two models. second, fine-tuning using the hand-crafted parallel corpus only, applied to each model resulting from the first step. we considered three baselines: pre-trained model without finetuning, copy of the source side of the test set (identity function), and a previously released model for early modern french normalization (modfr) @xcite . training and evaluation due to the small size of our hand-crafted parallel corpus and the need to divide it into training, validation and test sets, all our experiments were based on a 5-fold crossvalidation setup (train, validation and test splits represent approx. 63%, 17% and 20% respectively). we monitored the models performance during training based on the validation set with the bleu metric @xcite and the best performing model for each fold was kept to normalize the test set. finally, the 5-fold combined outputs were evaluated using four additional metrics, namely chrf @xcite , translation edit rate (ter) @xcite (wer) and segment-level accuracy. 5 the five metrics used in our evaluation measure the performances of automatic normalization models at various granularities. more precisely, bleu measures the n-gram precision (with 0 < @xmath0, chrf is the f-score at the character level, ter measures the translation edit rate at the word-level including shifts, wer is the word error rate without shifts and accuracy is the number of exact matching segments. the latter metric is interesting as it indicates how many segments are exactly matching the reference, thus reducing the manual revision requiring human experts. results obtained on the test set are presented in ## conclusion this paper paved the way towards archival documents conservation with expert targeted normalization and spelling variants reduction while preserving historical wordforms of 16th century middle french. compared to previous work in this field, the proposed methods do not modernize the historical spelling and syntactic structure but rather reduce the orthographic variability observed in originally authored texts. our approaches were inspired by low-resource machine translation and leveraged pre-trained models to achieve significant performance gains compared to a strong baseline. furthermore, our approaches are orthogonal and will benefit from collecting additional relevant corpora for fine-tuning and prompt engineering, which is one of our goals in future work. another research direction considered as future work stems from the large amount of publicly available modern french data and pre-trained llms which could be further leveraged for our task.