Spelling Normalization of Historical Documents by Using a Machine Translation Approach (2018)

Abstract: The lack of a spelling convention in historical documents makes their orthography to change depending on the author and the time period in which each document was written. This represents a problem for the preservation of the cultural heritage, which strives to create a digital text version of a historical document. With the aim of solving this problem, we propose three approaches—based on statistical, neural and character-based machine translation—to adapt the document’s spelling to modern standards. We tested these approaches in different scenarios, obtaining very encouraging results.

## introduction with the aim of preserving the cultural heritage, there is an increased need for the digitalization of historical documents, a procedure which strives for creating digital text which can be searched and automatically processed @xcite . however, the linguistic properties of historical documents create an additional difficulty. on the one hand, human language evolves with the passage of time. on the other hand, the lack of a spelling convention makes orthography to change depending on the author and the time period in which a given document was written. this makes historical documents harder to read, and makes it even more difficult to search for certain information in a collection of documents, or any other process that must be applied to them. spelling normalization aims to resolve these problems. its goal is to adapt the document's c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. in this work, we propose three approaches to tackle spelling normalization: a method based on smt; another method based on neural machine translation (nmt); and another method based on character-based machine translation (cbmt). our main contribution are the followings: pérez-ortiz, sánchez-martínez, esplà-gomis, popović, rico, martins, van den bogaert, forcada (eds.) proceedings of the 21st annual conference of the european association for machine translation, p. 129-137 alacant, spain, may 2018. the rest of this document is structured as follows: in section 2, we introduce the machine translation approaches used in our work. section 3 presents the different approaches taken to achieve spelling normalization. then, in section 4, we describe the experiments conducted in order to assess our proposal. after that, in section 5, we present and discuss the results of those experiments. finally, in section 6, conclusion are drawn. ## machine translation approaches in this section, we present the machine translation approaches used in our work. ## spelling normalization in this section, we propose different approaches to adapt the spelling of historical documents to modern standards. our first approach is based on smt. considering the document's language as the source language and its normalized version of that language as the target language, we propose to use smt to adapt the document's spelling to modern standards. in our second approach, we wanted to assess how well nmt works for normalizing the spelling of a historical document. therefore, similarly as to with the previous approach, considering the document's language as the source language and its normalized version of that language as the target language, we propose to use nmt to adapt the document's spelling to modern standards. finally, since in spelling normalization changes frequently occur at a character level, it seemed fitting to use a character-based strategy. therefore, our third approach is based on cbmt. similarly as to with the previous approaches, considering the document's language as the source language and its normalized version of that language as the target language, we propose to use cbmt to adapt the document's spelling to modern standards. as a starting point and to have the same conditions in both smt and nmt, in this work we chose to use the simplest character-based approach: to split words into characters and, then, apply conventional smt/nmt. ## experiments in this section, we describe the experiments conducted in order to assess our proposal. additionally, we present the corpora and metrics. ## results in this section, we present and discuss the experiments conducted in order to assess our proposal. the slovene language had a big restructuring in the 18 th century. for this reason, bohoric-whose documents were written during this period-is the corpus whose orthography differs the most compared to modern standards. evaluating the document's spelling differences with respect to modern orthography results in a low bleu value, a high ter value and a fairly high cer value. however, just by applying a statistical dictionary we achieved great improvements: bleu and ter improved highly, and cer decreased significantly. with our first approach, we achieved even greater improvements for all metrics. furthermore, when using bpe to reduce the vocabulary, we achieved new improvements. these improvements were more notorious when evaluating with cer and bleu, although they were significant with ter as well. our second approach achieved less satisfying results. the document's spelling differences were significantly reduced when measuring with bleu and ter. however, the results were significantly worse than the ones obtained using a statistical dictionary. furthermore, using cer to measure the spelling differences resulted in the document having more differences than the original document. using bpe to reduce the vocabulary did not help. in fact, results were significantly worse. most likely, this was due to the properties of the corpus: being the smallest of the corpora (less than four thousand sentences and with a vocabulary of around ten thou- finally, our third approach was both the most and least satisfying. while character-based smt achieved the best results for all metrics-all of them were significantly better than the results achieved by smt with bpe-character-based nmt achieved the worst results. once more, this was most likely due to the corpus being too small for the neural systems. entremeses y comedias, the oldest of the corpora, is the next corpus with greater orthographic difference. nonetheless, the quality of the original document shows fairly good bleu value, a considerable good ter value, and a low cer value. in spite of this, the statistical dictionary achieved significant improvements, the most noteworthy being the increase of bleu, which was the metric that showed the lowest quality. the smt approach reduced significantly the spelling differences from the original document. however, in this case, results were not significantly different to the ones obtained by the statistical dictionary, except when evaluating with cer, which results were slightly (around two cer points) worse. moreover, reducing the vocabulary with bpe did not achieved a significant difference with using the full vocabulary. the nmt system behave in a similar fashion as with the previous corpus: bleu and ter showed a significant reduction of the spelling difference from the original document, but smaller than the reduction achieved by the statistical dictionary. in this case, however, the differences with the statistical dictionary were smaller (around eight points of bleu and ter). moreover, although cer still showed more spelling differences than in the orig-inal document, its value was not as bad as with bohoric (around six points of cer). furthermore, despite still being worse than the statistical dictionary, bpe helped to improve results. it is worth noting the improvement in cer (around ten points), which represents an improvement with respect to the spelling differences in the original document. once more, the character-based approach yielded the best results. character-based nmt was the neural approach which yielded the best results, although these results were not significantly different to the ones obtained by the statistical dictionary. however, character-based smt did significantly improved the statistical dictionary. similarly to what happened with the other corpora, the statistical dictionary significantly reduced the spelling differences in the quijote corpus. it is worth noting, however, that these differences are considerable smaller in this corpus: measuring the spelling differences in the original document shows a fairly good bleu and ter values, and fairly small cer values. in this case, the smt approach did not yield results as satisfactorily as with the previous corpora. results showed a significant improvement with respect to the original document. however, this improvement was not significantly different than the one achieved by the statistical dictionary-except when measuring with cer, whose value was significantly worse. nonetheless, bpe improved the results, and the generated document was significantly better (for all metrics except for cer) than the document generated by the statistical dictionary. the results yielded by the nmt system showed a significant improvement with respect to the spelling differences from the original document (except when measuring with cer), but this improvement was significantly worse than the one achieved by the statistical dictionary. reducing the vocabulary with bpe helped to improve the results-specially when measuring with cer, whose results were now significantly better than the original document-but they were similar to the statistical dictionary's results. finally, the character-based approached achieved, once more, the best results. however, while using cer to measure the document's spelling differences with respect to modern standards yielded a significant improvement (for character-based smt), measuring with bleu and ter yielded similar results to using the smt approach combined with bpe. similarly, character-based nmt achieved a significant improvement in terms of cer, but similar bleu and ter results to the nmt-bpe approached. being the newest corpus, gaj contains fewer spelling differences with respect to modern orthography. in fact, measuring the spelling differences from the original document already yielded satisfactory bleu and ter values, and a low cer value. nonetheless, the statistical dictionary managed to improve bleu and ter results, although yielded a worse cer value. the smt approach managed to significantly improve results for all metrics. however, reducing the vocabulary with bpe yielded similar results, except when measuring with ter, whose results were significantly better. gaj being a fairly small corpus (thirteen thousand sentences and with a vocabulary of around thirty thousand words), the nmt systems behaved similarly as with bohoric: the generated document had more spelling differences than the original document. using bpe improved results, but the generated document still contained more spelling differences than the original one. character-based smt yielded the best results when using cer to measure the spelling difference. however, measuring with bleu and ter yielded similar results to the smt approach. characterbased nmt, however, was the nmt approach which yielded the worst results-specially when measuring with ter and cer. in general, except for one exception (gaj, whose best results-when evaluating with ter-were achieved by the approach that combined smt with bpe), character-based smt was the approach that yielded the best results for all metrics. it is also worth noting how well-for being such a simplistic approach-using an statistical dictionary behave: except for one exception (gaj, which yielded an increase of spelling differences when evaluating with cer), all results showed a significant reduction of spelling differences with respect to the original document and, in some cases, not too much worse than character-based smt. the bleu and ter from the original document, and how much these values have significantly improved, seem to indicate that the final document is quite different to the original one. however, cer seems to indicate otherwise. most likely, since spelling differences occur more frequently at a character level (i.e., most orthographic changes consist in a few letters per word), bleu and ter-which evaluate at a word-level-are being penalized. nonetheless, all metrics show that the spelling differences have been significantly reduced. ## conclusions and future work in this work, we proposed three machine translation approaches to update the spelling of a historical document to match modern standards, increasing the document's readability and helping in the preservation of the cultural heritage. additionally, as an extra baseline, we proposed a simplistic approach: based on the frequency of which, on the training corpora, the spelling of a word is changed, to build a statistical dictionary. then, on a given document, we checked, word by word, if it was on the dictionary. if the search was positive, we changed that word by the translation that appeared in the dictionary. otherwise, we left the word as it appeared in the original document. we tested our proposal with four datasets formed by documents from three different time periods, two different languages and three distinct alphabets, obtaining very encouraging results. in general, approaches based on smt yielded better results than those based on nmt. this was specially true for the smallest corpora, in which the neural systems were not able to learn properly and yielded more spelling differences than the ones contained in the original document. as it was to be expected due to the task characteristics (in spelling normalization, changes frequently occur at a character level), the characterbased approaches-both phrased-based and neural-yielded the best results for each kind of system (i.e., character-based was the best smt approach, and character-based nmt was the best nmt approach). the exception was character-based nmt, which yielded worse results when applied to the smallest corpora. finally, it is worth noting how well the statistical dictionary behaves. although its results were not the best, they were close enough to take this approach into consideration. being the simplest and fastest to compute, it could be useful in cases in which its worth sacrificing quality to increase speed. as a future work, we would like to try new character-based approaches. in this work, we tested the simplest approach (to split the words into characters and, then, apply conventional smt/nmt). however, more complex approaches have been developed in recent years @xcite @xcite . finally, a frequent problem when working with historical documents is the scarce availability of parallel training data @xcite . therefore, we would like to obtain more diverse corpora to be able to experiment in broader domains: older time periods, documents written by a great variety of authors, etc. additionally, we would like to explore the generation of synthetic data @xcite to create new training data.