Context-Aware Text Normalisation for Historical Dialects (2020)

Abstract: Context-aware historical text normalisation is a severely under-researched area. To fill the gap we propose a context-aware normalisation approach that relies on the state-of-the-art methods in neural machine translation and transfer learning. We propose a multidialect normaliser with a context-aware reranking of the candidates. The reranker relies on a word-level n-gram language model that is applied to the five best normalisation candidates. The results are evaluated on the historical multidialect datasets of German, Spanish, Portuguese and Slovene. We show that incorporating dialectal information into the training leads to an accuracy improvement on all the datasets. The context-aware reranking gives further improvement over the baseline. For three out of six datasets, we reach a significantly higher accuracy than reported in the previous studies. The other three results are comparable with the current state-of-the-art. The code for the reranker is published as open-source.

## introduction historical languages rarely have enough resources to satisfy the needs of state-of-the-art supervised machine learning methods. manual labeling of large amounts of training data is frequently out-of-thequestion as it demands rare and expensive expertise of historical linguists. historical dialects, however, are frequently grammatically and lexically similar to their modern high-resource descendants. thus, being able to normalise the historical spelling to the modern standards can enable processing of historical dialects with nlp tools trained on modern languages. modern high-resource languages are well-standardized and have prescriptive grammar, spelling and lexica usage standards. historical languages, on the contrary, did not have such standardization. while modern german also has a plenty of dialects, there exists its standardized variety i.e. standard high german. middle high german (mhg), on the contrary, does not have contemporary standardization. when we deal with a historical text, we deal with a historical dialect (e.g. middle high german, middle low german) that text is written in. even more, there is a plenty of variability within the dialects caused by the lack of orthographic standards, geographical differences, the exact time when a text was written etc. the normalisation task presupposes that there exists a standard to which the historical variants are normalised. as historical languages do not have such a standard, the closes possible goal for historical dialect normalisation would be their modern standardized descendants. throughout centuries, languages undergo systematic phonological changes such as the great vowel shift in english, the first and second consonant shifts in germanic dialects or the first and second palatalization in slavonic languages. this can be captured systematically by machine learning algorithms and applied to unseen words. thus, the current state-of-the-art approaches to the historical normalisation rely on statistical or neural machine translation methods and define the task as a problem of translating between characters or substrings @xcite instead of words. historical texts pose an additional challenge of non-standardized orthography. spelling variations can depend on the region, authorship and time of writing. these kinds of irregularities complicate the training of a normaliser as it fails to unambiguously align the modern word to its historical form. in this paper, we show that providing the dialectal context during the training disambiguates the alignment between the historical and modern words and, thus, leads to a significant improvement of the normaliser's performance. most of the research on historical text normalisation has focused on context-agnostic approaches that do not consider the immediate surrounding of the word. however, the context-agnostic approaches do not have an efficient method to deal with homographs, polysemy and homonyms (e.g. the middle high german (mhg) word in can be normalised as a modern german preposition in or a pronoun ihn, an accusative form of the pronoun er "he"). as a context-agnostic neural normaliser cannot disambiguate between them, we hypothesise that all the valid normalisations are expected to appear among the top five normalisation results. we have tested this hypothesis and confirmed that a correct normalisation is found among the top five candidates for 96.22% to 98.83% of words depending on the dataset. we propose a reranking method by applying a word-based n-gram language model to the top five normalisation candidates. we show that reranking improves the word accuracy and delivers state-of-the-art performance. ## related work the early work on text normalisation relied on rule-based methods @xcite ; @xcite ) and dictionary lookups @xcite . @xcite compare the performance of two statistical normalisation tools on portuguese and arrive at the conclusion that context-agnostic normalisation can only achieve limited results. later, statistical machine translation (smt) methods became the state-of-the-art. the normalisation problem was defined in the smt context as a problem of translation between characters which maximizes the probabilities of p (f |e) × p (e) with p (f |e) being the probability that the word f is the historical variant of the modern word e and p (e) is the probability of observing the normalised form in the modern language. @xcite ; @xcite ; @xcite ; @xcite have all successfully applied characterbased smt to various normalisation tasks. as neural machine translation has gained momentum in the past few years, the neural normalisers have also started producing state-of-the-art results. @xcite compares smt-based methods to sequence-to-sequence models and concludes that nmt methods perform consistently better than smt. @xcite also shows that nmt methods perform consistently better than smt under the condition that enough data is available. joint training and multitask learning have long been successfully used for various nlp applications. @xcite introduced a multitask learning approach to historical german normalisation. they trained a bidirectional lstm network on the whole anselm corpus and retrained its prediction layer on each text separately which led to an improvement over the baseline. @xcite conducted a detailed evaluation of various setups of joint and multitask training. pairwise dataset combination showed that multitask learning on two datasets of different languages (e.g. german and slovene) did not give a consistent improvement but the common trend was that closely related datasets benefit the most from the multitask training. the joint training setup with no indication to which language the dataset belongs by @xcite had mostly a negative effect on the normalisation accuracy. the general mt research gets more interested in context-aware normalisation. @xcite propose an decoder architecture that takes into consideration previously translated sentences while generating new texts. @xcite show that integrating document level information helps to improve french and chinese translation into english. there is still not much work done in the area of context-aware normalisers. @xcite proposed to use sequence-to-sequence models to normalise full sentences for conversational systems. @xcite proposed to use hidden markov models to choose over the normalised candidates in a sentential context. @xcite used a language model for candidate correction of an unsupervised normaliser. ## data historical languages did not have universally accepted prescriptive grammar or standardized orthography and, thus, spelling varies depending on the authorship, region or time when the document was written. some historical languages could have even utilized various alphabets (e.g. slovene) depending on the region or time. spelling variations can also be found even within the same document. for our experiments, we have used an openly available normalisation dataset foot_1 by @xcite . we selected the languages that have regional or diachronic dialects in the dataset. thus, we used four historical languages (german, slovene, spanish and portuguese). the dataset includes the following languages: german, hungarian, icelandic, portuguese, slovene, spanish, swedish. the german, portuguese, slovene and spanish datasets are not homogeneous and are comprised of several diachronic and geographical varieties of the corresponding languages that are normalised to its modern standard variation. ## multidialect text normalisation previous work has shown that multitask and joint training setups are particularly useful for closely related datasets. the historical texts are a perfect testbed for multitask and joint setups: the spelling varies depending on the dialect, region, time of writing or even the author. while many of the previous normalisation approaches do not incorporate this information into the training, we assume that the dialectal context can disambiguate homographs and help a normaliser to learn region-and time-specific diachronic spelling changes. in this section we show that providing a neural normaliser with dialectal context leads to significantly foot_2 better performance over all the datasets. to incorporate the dialectal information into the normaliser, we follow a standard approach to multitask learning used in multilingual machine translation @xcite : after splitting the input words into characters we modify the input by adding a tag that identifies the word's historical dialect. during the training we observed that balancing the data can play a crucial role in the normaliser's performance. thus, we ensured that all the dialects are equally represented in the training and in the validation by means of oversampling. this means that if a dialect has five times as few data as the dialect with the largest amount of data, each normalisation pair of the underrepresented dataset will appear five times in the training data. ## context-aware reranking the weakness of context-agnostic text normalisation is its inability to differentiate between the homographs, homonyms and polysemy. example 1 from the anselm normalisation dataset shows a mhg sentence with a homograph das that can be either normalised as a modern german (de) demonstrative pronoun das ("this") or a relative pronoun dass ("that"). context-agnostic normalisation can only produce the exact same normalised form for both words. yet, the correct normalisation can usually be still found in the top five candidates. if the normaliser has observed multiple homographs in the source training data for which correct normalised forms differ, the normaliser's uncertainty about the output is very high which leads to it assigning nearly equal probabilities to both normalisations. thus, we propose a context-aware reranking of the normalisation output. we borrow the idea from the statistical machine translation that rescores the translation probabilities with the probabilities of observing a n-gram in a large monolingual corpus. for the context-aware reranking, all the multidialect normalisation models were trained as described in the previous section (sec. 4) and balanced accordingly with the only exception of spanish for which the best performing model was trained on an unbalanced dataset. furthermore, we train an n-gram model on the monolingual target corpus and apply a beam search to the top five normalisation results. ## evaluation in order to facilitate the beam search and to exclude error propagation, we use a threshold over the negative log probability assigned by the normaliser. figure 3 shows the percentage of the correct normalisations found among the top five results filtered within a given threshold window. the threshold is defined as -log(p (w 1 )) + d i.e. the sum of the negative log probability assigned to the first-ranked candidate and a variable d that determines the threshold's window (the horizontal axis of figure 3 ). the vertical axis shows how many correct normalisations are found within the given window i.e. if at least one candidate is correct. the graph shows that most of the correct normalisations can be found within a narrow window of @xmath0.75 and a steep surge is observed at the d between 0.2 and 0.3. this is an expected observation as neural normalises typically assign closer scores to uncertain outcomes. having made this observation, we propose a context-aware reranking method for the normalisation results. all the language models were trained on the target side of the normalisation datasets and on a bible acquired from the opus parallel corpus foot_4 . we downloaded the pt, es, de and sl bibles aligned to an english bible so that all the bibles have the same amount of verses. we have attempted to train larger language models on news corpora but they did not perform better than the bible-based lms. this is likely due to the fact that normalised historical texts have different syntax and lexica than news corpora. this assumption is also confirmed by the fact that we achieve a very significant improvement over the multidialect and state-of-the-art (sota) baselines for the german anselm dataset which is a collection of religious texts. the improvement was much lower for the botanical texts of the german ridges corpus. thus we conclude that having an in-domain monolingual corpus is extremely beneficial for this approach. we studied the effect of the threshold window by applying the language model to rerank the normalisation results through beam search as described in the previous chapter. figure 1 shows the relations between the normaliser's accuracy and the window size. for all the datasets except for spanish, the accuracy significantly improved over the baseline starting from the window set to 0.15 and started to deteriorate at 0.3. the best accuracy for the spanish, portuguese, slovene and german ridges datasets was achieved with the window value of 0.2. the best results for the german anselm dataset was achieved with the window set to 0.25. this is also the only dataset for which the accuracy for the threshold of 0.2 is significantly lower than for the reranked results over the window of 0.2. for the sota comparison, we used the results reported by bollmann-2019-large on the same datasets. the normalisers that achieve the best results on the dataset were: a neural normaliser by tang-etal-2018evaluation and a character-based normaliser enhanced with a language model trained on a large amount @xcite . the * means that the results are not significantly different from the best result. figure 3 : the normalisation accuracy (the vertical axis) changes depending on the threshold's window (the horizontal axis). the larger the window, the more candidates are considered. for three datasets we achieved a significant (@xmath1.05) improvement over the reported state-of-theart. also for all the datasets except for gaj slovene applying the context-aware reranking leads to a significant improvement over the multidialect normaliser. the greatest improvement over the sota baseline is observed for the german anslem dataset with +2.2 to the accuracy. the balanced multidialect joint training also resulted in the significant improvement over the sota (+1.07) for the german ridges dataset and the application of the reranking further significantly improved the accuracy by +0.55. the context-aware reranking also led to a significant accuracy improvement as compared to sota and to the multidialect normaliser for slovene bohoric. there is no significant difference between the performance of the models on the gaj slovene and portuguese as compared to the sota. the reranking, however, has significantly improved the accuracy for the portuguese as compared to the multidialect baseline. the spanish dataset was the most challenging for the multidialect normaliser and while the reranking significantly improved the results as compared to the multidialect setup, the final accuracy score was still insignificantly below the sota. ## qualitative evaluation of the results the evaluation showed that the largest improvement was achieved on the german datasets and slovene bohoric while the performance on the slovene gaj, spanish and portuguese stayed the same as sota. the german texts are older, the spelling is more chaotic and more distant from the modern standard. in order to understand why adding the german side of wmt corpus foot_6 to the data does not help to improve the performance, we analysed frequent corrections that were made by the language model on the german datasets. a qualitative analysis by a german speaker showed that the vocabulary and syntax of the normalised side of the training data contains many archaisms and set expressions and syntax follows the rules of mhg rather than modern german. for example, liebes "dear" is normalized by the context-agnostic normaliser as leibes "bodily" in liebes kind "dear child". the phrase occurred 0 times in wmt and 190 times in german anselm training set. another common correction in german anselm is sandte 'sent' changed to "sankt" saint. the correction occurred 116 times in the testset. in the wmt corpus "sankt" occurs exclusively as a part of geographic locations and never in the same context as in the german anselm corpus (names of saints). not only historical spelling variations, but also modern homographs pose a challenge that contextagnostic normalisation cannot overcome. the top ranked incorrectly normalised wordforms in the german anselm (see the german ridges corpus poses an additional challenge of abundance of botanical terminology: we have calculated that 90% of words, that were assigned the incorrect wordform by the reranker, occur only once in the test set and are mainly botanical terms that are not found in the training set. as these terms also are neither found in the wmt corpora, it again explains why adding more modern data did not result in any improvement. we have examined the effect of the window in which the candidates are considered for the beam search and observed that the model is very sensitive to the error propagation. in particular, allowing a large window or considering all the top five normalisation results irrespective of their score decrease the accuracy. this mostly happens when the lower ranked candidates are valid words that are frequently observed in this contexts (e.g. "spoke" and "speaks"). the normaliser would assign a much lower probability to a lower ranked normalisation but the language model would have an approximately equal n-gram counts for the both of them. thus, it is important to determine the correct window. the general observation is that if the model is trained on in-domain data, the window can be slightly larger than if the model is trained on out-of-domain data. this is probably due to the fact that it better covers domainspecific lexica. the only dataset for which the context-aware reranking did not result in significant improvement over the multidialect model is the gaj slovene dataset. figure 2 gives a clue why this is the case: the slovene dataset plateaus faster than any other dataset and does not have a steep surge near the window of 0.3 like the german datasets and slovene bohoric. thus, the possible improvement can only be rather limited and enlarging the window would cause the error propagation as described above as too many correct firstranked normalisations would be unnecessary reranked by the language model which would eliminate the benefit of its usage in the first place. ## conclusion we have presented a reranking method for a multidialect normaliser. we have shown that neural historical normalisers perform better when trained on the data with explicitly provided dialect information. the context-aware reranking of the normalisation results implemented by means of a statistical n-gram model has resulted in improvement over the baseline and performs comparable or better than the state-of-the-art results reported in previous studies. we will publish all the code used in this study as open-source. thus, future work will focus on exploring more elaborated methods of context-aware reranking i.e. comparison with neural language models.