Name Consistency in LLM -based Machine Translation of Historical Texts (2025)

Abstract: Large Language Models (LLMs) excel at translating 16th-century letters from Latin and Early New High German to modern English and German. While they perform well at translating well-known historical city names (e.g., Lutetia –> Paris), their ability to handle person names (e.g., Theodor Bibliander) or lesser-known toponyms (e.g., Augusta Vindelicorum –> Augsburg) remains unclear. This study investigates LLM-based translations of person and place names across various frequency bands in a corpus of 16th-century letters. Our results show that LLMs struggle with person names, achieving accuracies around 60%, but perform better with place names, reaching accuracies around 90%. We further demonstrate that including a translation suggestion for the proper noun in the prompt substantially boosts accuracy, yielding highly reliable results.

## introduction large language models (llms) have shown remarkable capabilities in translating 16th-century texts in latin and early new high german into modern-day german or english, surpassing other machine translation systems by a considerable margin @xcite . llms also excel at translating well-known proper nouns, be they current or historical names. however, the extent to which llms accurately translate lesser-known person names and place names (toponyms), is underexplored. proper nouns are important content words; mistranslations can have far-reaching consequences. if, as it occurred in our experiments, vallistellinensi is mapped to the valais region in switzerland instead of italy's valtellina, or bayonensis to basel instead of bayonne (france), the differences amount to distances of several hundred kilometers! in this study, we investigate llm-based translation of person names and place names (countries, cities and regions) in latin and early new high (enh) german from our 16th-century letter corpus into modern german. the translation involves normalizing inflected forms into base forms (e.g. @xmath0.g. @xmath1. this study is part of a project to translate and provide access to the letter correspondence of the zurich reformer heinrich bullinger (1504-1575) 1 . the corpus consists of 12,000 letters, three quarters of which are in latin, most of the others in enh-german @xcite . the corpus is annotated with manually checked person and location names. the following sentence, with three person names and four place names, exemplifies the subject matter. note that the gpt-translation splits the long input sentence into two, which helps readability. to evaluate llm translation performance across a range of name frequencies, we selected a balanced sample of names: the 10 most frequent, 10 of the least frequent (with a minimum of 10 occurrences), and 10 randomly chosen names from intermediate frequency bands. these names appear in many forms, up to 200, as in the case of heinrich bullinger himself: huldrice, heinricho, heimrych, heilrich, ..., bullingerus, bulliger, bulingero, etc. for each sampled name, we extracted one sentence for each different form of that name, as well as the context of the three preceding and three following sentences. we tested with two settings: without context (just the sentence that contains the name) and with a context window of the three sentences before and after. within these two settings, we explored two different prompting strategies to guide the translation of proper nouns. the two strategies involved including or leaving out historical background information in the prompt (i.e., where the sentence to be translated is taken from), and adapting the system prompt or keeping it generic. additionally, we investigated how marking the target proper noun affects the translation quality. this step is interesting in that it might attract the llm's focus to a particular word or word sequence, and that it structures the output automatically. this facilitates evaluation and postprocessing, since the proper nouns can be easily extracted. finally, we evaluated the translation quality if a translation suggestion for the name is included in the prompt. by evaluating and comparing these approaches, we assess the abilities of llms in translating proper nouns and how contextual and formatting cues influence their performance. we also examine the reasons for the discrepancies in translation quality between different names. this work provides insights into how llms address complex translation challenges that involve historical texts and proper nouns. ## the 16th-century letter corpus our corpus of 16th-century letters consists of the correspondence of the zurich reformer heinrich bullinger. it encompasses around 12,000 documents, which include 3,100 letters professionally edited by the institute for swiss reformation studies foot_1 and an additional 5,400 manually transcribed letters. we have automatic transcripts for most of the remaining 3000+ letters, whereby our handwritten text recognition has a character error rate of around 8%. we ignore these letters in the current study. the edited part of the corpus has been published in 20 printed volumes @xcite , each of which has an index with manually curated person and place names. the index entries for a specific person or place point to the pages in the book where the name appears, but the name itself is not explicitly marked within the printed text. we used these indices to automatically mark the names in the digital versions of the letter summaries, the letter texts, and the footnotes, initially on the specified pages only. this mapping was limited by the fact that the indices contained the names in standardized form (e.g. the city name antwerpen), which made it difficult and partly impossible to detect all inflected forms and spelling variations of a given name (e.g. antwerpia, antwerpię, antorff ). all names that we marked in this step come with a unique project-internal identifier, which is linked to wikipedia or to the gnd-database. in a second step, we copied the assigned identifier of marked names to other occurrences in the same letter where they could be unambiguously assigned. we also applied the annotations to the 5,400 transcriptions. in a third step, we trained a name recognizer on these data to spread the annotation (person and place name tags without linking) to all unmarked name mentions in the 8,500 letters. in order to improve the name annotation quality, a citizen science campaign contributed by checking person and place names, with volunteers annotating the names and linking them to the corresponding entities. this enriched the corpus with several 10,000 person and place names, enabling us to conduct a comprehensive analysis of proper noun translation. currently, the 8,500 letters, the roughly 3100 manually written summaries and 75,000 footnotes are marked with 202,000 person name tags and 156,000 place name tags, out of which 188,000 person names (5924 unique ids) and 150,000 place names (2950 unique ids) are linked. the documents retain historical characters such as ę, ů, a e , o e , reflecting the orthographic conventions of the period. abbreviations commonly found in the letters have been expanded by editors and transcribers e.g. 'frid[olin] schůler') 3 . the letters constitute a rich historical resource, shedding light on politics, theological debates, regional and european news, education, and family matters. they are part of bullinger's vast correspondence, whose network extended from zurich across europe, reaching as far as denmark, england, and lithuania. the current study focuses on the bullinger letter exchange, but its results are relevant for the many other letter collections of the same period which amount to more than 100,000 letters in latin and enh-german. this applies, for instance, to the collection of the theologians' correspondence in the southwest of the empire in the early modern period (1550-1620) 4 . for an overview see @xcite . ## related work on named entities in machine translation the topic of named entities in machine translation has been addressed repeatedly. for an early paper, see @xcite , who propose "name translation" as a specific mt evaluation task. more recently @xcite report on "fast improvements" for handling named entities in machine translation by implementing named entity recognition as a separate pre-processing step. similarly, @xcite propose an "extract-and-attend" approach to improve neural mt performance between english, russian and chinese, that require at least transliteration of the names because of the different scripts. for translations between languages with the latin alphabet, @xcite report that the "categories with the highest performance (above 90%) were [...] named entities & terminology" when testing various machine translation systems against an english -german test suite that covers many linguistic phenomena. however, if the target language requires inflection of the names, then translation challenges still arise. one such case is icelandic. @xcite argue that machine translation of person and place names from english to icelandic is far from perfect. interestingly, place names proved to be more difficult than person names in their experiments. @xcite show that named entity recognition improves mt for inuktitut to english. the central issue is "lexical cohesion: the same named entity must be translated consistently across the current sentence and context sentences" @xcite . the ultimate goal is transcreation with cultural adaptation. by integrating information from a multilingual knowledge graph into neural @xcite obtained huge improvements for name translation across 10 language pairs. although there is previous work on named entity recognition for latin @xcite , we found no paper on named entities in mt for latin, nor for early new high german. we are breaking new ground in systematically evaluating names in machine translation from these historical languages to modern languages with llms. ## name translation results tables 2, 3, 4, 5 show the results of the different settings. we note the following general findings: in 5 out of 6 remaining settings, it decreases the score, averaging -0.75% across all 6. ## conclusion we conclude that gpt-4o translates more prominent names better than less-known names, and it translates place names (around 90% accuracy) better than person names (around 60%). the best setting is to adapt the prompt itself to the task by including some meta-information about the translation setting, in our case 'the following sentence is taken from a letter that is part of swiss reformer heinrich bullinger's correspondence in the 16th century.' additionally, including either a task-adapted system prompt or context to the sentence to be translated (3 sentences before and after) has been shown to improve translation quality of the proper nouns. the best setting for person names was an adapted prompt with sentence context, and for place names an adapted prompt with an adapted system prompt, but without sentence context. to synthesise, more context improves translation quality in proper nouns if it is pertinent to the task, yet too much context results in a quality decrease. a good rule of thumb is: as much pertinent content as possible with a setup as simple as possible. translation quality of a given proper noun is influenced by the following main factors: importance respectively presence on the internet and llm training data and orthographical proximity to the correct translation. finally, prompting the llm to output the data in a structured way, i.e. marking the translated proper noun for our convenience, has not proven to be a commendable approach. even if the marker is appended, post-generation of the target word accuracies were 20 to 30 percent points lower than without any marking. the task of generating structured representations of llm output is better handled separately of and after the generation. including a translation suggestion for the person or place in the prompt is the best way to deal with proper noun translations, with (near) perfect accuracies (98.8 resp. 100%), but requires named entity recognition. if that is not possible, then the use of retrieval-augmented generation (rag) is a compelling prospect for future research, as it could alleviate some of the issues associated with proper noun identification and translation. preliminary experiments with perplexity ai on 25 randomly selected sentences suggest that it performs on par with gpt-4o, highlighting its potential in this area. we focused on a subset with clean human transcriptions of the 500-year old letters. for letters that are not yet transcribed, the combination of automatic handwritten text recognition with machine translation awaits further investigation. llms are robust against a certain amount of recognition errors, but may hallucinate in translations for letters with a substantial amount of text recognition noise. georg blandrata blandratae ille item literas de negocio blandratae scriptas fideliter cognoscendas tradidit, ut nunc apud celsitudinem tuam attester ipsum legatione sua peroptime esse defunctum. 28 balthasar funk b. funken ir habt vor jaren in simili forma dem b. funken gewilfart, der doch die wahrheit jetz so schandlich verleugnet, unterdrukt unnd verfolgt; so hoff ich doch, ich hab deßgleichen und bässers dann er funk umb die warheit verdient. 13 hans peyer h[ansen] peyer gru e ß uns susannen, lysenbethli, susili und j. h[ansen] peyer und sin hußfrowen, h. hansen löwen etc 12 rudolf thumysen růdolph[o] dumysen eruditione et pietate praestantissimis viris heinrycho engelhardo, leoni iudae, heinrycho bullingero, conrado pellicano, theodoro bibliandro, batto n., r ůdolph[o] dumysen et [nicolao] zeendero, tig[urinae] ecclesiae pastoribus et doctoribus, suis in christo colendissimis praeceptoribus et fratribus charissimis. 10 georg witzel vicelii 8. quęstiones catechisticę vicelii. 10 james haddon iacobum haddonum d. iacobum haddonum anglum diligenter salutabis; indicabis curaturum me, ne quid libelli illius edatur, de quo ille ad me scripsit. 10 jean du fraisse ioh[annes] frax[ineus] tuus ioh[annes] frax[ineus], ep[iscopus] bayo[nensis]. 10 jean budé budaeus superest ut d. budaeus quod literis complecti non expedit coram vobis exponat. 10 pierre de la ramée petrum ramum petrum ramum, virum tum pietate tum eruditione praestantem, quem tam officiose salutari iubebas, vides; cuius congressum tibi iucundissimum fore non dubito, a quo utpote nostrorum hominum iam peritissimum multa audies, quae scire operae precium fuerit; itaque plura non addam. 10 rosina zollikofer zollikofferin unnd ob es sich aber begäbe, das durch ein urtheil der vilgedachten oberkeitt der vorgenempt haga der zollikofferin abgesprochen wurde, als das die oberkeitt von des dritten grads und von wegen anderer eehaften ursachenn, das versprächen irer beiden ufflößen unnd nüt wölt gelten lasßen, halten wir nitt, das sy beide inen in dem ein gwüßne machen söllind, das sy der erlütherung irer ordenlichenn oberkeit, deren sy doch sy sich ergäbenn habend, volgend unnd sich in ander weg vereelichend. 10 petrus dathenus dathenica non erat necesse, mi bullingere, ut tanta solicitudine rogares, ne commoverer dathenica intemperie; perinde mihi fuit ista intelligere convicia, ac si somnium vidissem, propterea quod eadem haec saepe audivi ab illis inculcata esse ad nauseam usque principi electori aliisque cunctis publice et privatim.