LLM s for Translation: Historical, Low-Resourced Languages and Contemporary AI Models (2025)

Abstract: Large Language Models (LLMs) have demonstrated remarkable adaptability in performing various tasks, including machine translation (MT), without explicit training. Models such as OpenAI’s GPT-4 and Google’s Gemini are frequently evaluated on translation benchmarks and utilized as translation tools due to their high performance. This paper examines Gemini’s performance in translating an 18th-century Ottoman Turkish manuscript, Prisoner of the Infidels: The Memoirs of Osman Agha of Timișoara, into English. The manuscript recounts the experiences of Osman Agha, an Ottoman subject who spent 11 years as a prisoner of war in Austria, and includes his accounts of warfare and violence. Our analysis reveals that Gemini’s safety mechanisms flagged between 14% and 23% of the manuscript as harmful, resulting in untranslated passages. These safety settings, while effective in mitigating potential harm, hinder the model’s ability to provide complete and accurate translations of historical texts. Through real historical examples, this study highlights the inherent challenges and limitations of current LLM safety implementations in the handling of sensitive and context-rich materials. These real-world instances underscore potential failures of LLMs in contemporary translation scenarios, where accurate and comprehensive translations are crucial—for example, translating the accounts of modern victims of war for legal proceedings or humanitarian documentation.

## introduction machine translation (mt) has long been a cornerstone of natural language processing (nlp), facilitating cross-linguistic communication and information accessibility. with the advent of large language models (llms) such as openai's gpt-4 and google's gemini, mt has seen significant advancements in both performance and adaptability. these models are not only evaluated on standard translation benchmarks, but are also deployed as translation tools across various domains. however, the translation of historical and low-resourced languages presents unique challenges that are often overlooked in mainstream mt research. ottoman turkish (ot), an extinct language with limited digital resources, exemplifies such a low-resourced language. translating ot manuscripts remains a laborintensive task with limited scholarly resources. to the best of our knowledge, there is no mt system specifically designed for ot-to-english (ot-en) translation. current tools for turkish-english translation are not directly adaptable for this task, despite turkish being the most closely related living language to ottoman. however, we know anecdotally that scholars in ottoman studies have been using llms for translating their sources. indeed, llms have the potential to act as first-pass translators of ot, reducing the time and effort needed to translate primary sources. accessible and reliable primary sources are indispensable for historians. however, in englishlanguage instructional settings, the scarcity of translated non-english sources limits historians' ability to teach global histories. this skews students' perception of history, reinforcing a narrow view that excludes varied cultural perspectives and further marginalizing certain groups. enhancing the availability of primary sources through effective translation is essential for diversifying history curricula and democratizing access to the past. by increasing the availability of multilingual primary sources, we can contribute to a more inclusive and comprehensive understanding of our shared history. in addition to addressing the challenges of translating low-resourced historical languages, this study explores the ethical implications of integrating artificial intelligence (ai) safety mechanisms within large language models (llms). these safety protocols are designed to mitigate the dis-semination of harmful content by flagging and restricting passages that contain violence, hate speech, or other sensitive topics. these protocols use algorithms to evaluate the contents of user prompts before these can be processed by llms, effectively content-moderating user prompts. often there is little detail or clarity as to how these algorithms are implemented and what constitutes as inappropriate prompts. in the context of translation, such mechanisms can inadvertently impede sensitive narratives from being processed by the models. translation requires accuracy and reliability, arguably even more when it comes to complex and difficult narratives of human experience. ai safety and content moderation raises ethical issues regarding the use of llms for translation. our work facilitates the examination of these ethical issues on real life data. as llms are increasingly incorporated into translation pipelines, it is crucial to understand how these safety mechanisms handle complex accounts from real sources, as opposed to synthetic texts created to test ai models. however, turning testimonies of contemporary individuals into ai test sets comes with its own set of ethical challenges, such as violations of privacy and consent. by testing llms on historical documents, we can assess the impact of these safety decisions without involving the stories of living individuals. this paper investigates the performance of google's gemini in translating an 18th-century ottoman turkish manuscript, prisoner of the infidels: the memoirs of osman agha of timis , oara, into english. by analyzing how ai safety settings influence the translation process, this study aims to uncover the limitations and potential biases introduced by these mechanisms when handling historical and context-rich materials. ## related works this research project is at the intersection of historical nlp, digital history, machine translation, and nlp research on low-resourced languages. by historical nlp, we are referring to works like those on coptic @xcite or latin @xcite that study these historical languages within the field of nlp. the use of nlp methods in history research has increased in the recent years jo (2020); de bolla (2023); @xcite . our work recognizes the value that computational approaches add to history scholarship. at the same time, we argue that digital history, much like nlp, has a bias towards english as non-english languages are extremely underrepresented in this field. thus, we see similarities between our work and those of nlp researchers studying other non-english, lowresourced languages @xcite . ## preliminary experiments while this paper deals with the performance of gemini 1.5 pro, we tested the performance of the following models on translating osman aga: gpt-3.5, gpt-4, gemini 1.0, cohere aya, before conducting the experiments discussed in this paper, and gpt-4o, gpt-o3-mini, gemini 2.0, and claude sonnet 3.7 leading up to the writing of this paper. we also tested a state-of-the-art translation model, helsinki nlp opus nmt model, for turkish to english translation @xcite and fine-tuned this model on a custom dataset that we created from turkish-english novels and handful ottoman works with english translations. we report bilingual evaluation understudy or bleu scores @xcite and character n-gram f-score or chr-f @xcite below. during our preliminary experiments, we discovered that gemini 1.5 exhibited content moderation behavior despite relatively high scores and acceptable first-pass translations. these preliminary results prompted our investigations into gemini 1.5 pro as outlined below. ## methods since our research goal is to study gemini's safety settings and its relation to translation, we searched for code examples written or approved by google. we identified this notebook from google cloud platform's github repository. the first example in this notebook was translation of french into english, which we included as figure 6 in the appendix of this paper. we modified this code to save the output of the model, safety ratings, and the other values into a csv. for our experiments, we used gemini pro 1.5 through api calls. we prompted the model to translate the manuscript sentence by sentence. we ran a first pass in which we sent requests for the entire manuscript. of the 755 sentences in ot-en dataset, 208 sentences, or 27%, were not translated. we know from previous experiments that sometimes these models can behave in an unexpected way and simply not translate. thus, we ran a second pass on these 208 sentences using the same translation prompt. in the second pass, 34 more sentences were translated and we ended up with a total of 174 untranslated sentences, which represents about 23% of the entire dataset. our quantitative analysis focused on these 174 untranslated sentences. we extracted the safety rating information for each sentence and plotted the severity and probability scores. we studied the relationship between how severe the predicted harm in a given sentence is with how confident the model is with its assessment. additionally, we realized that each sentence can be blocked for one, two, three, or all four categories. we ran further analysis to identify which of these 4 categories and their exclusive combinations are seen in these 174 sentences. we also mapped these on an histogram across the entire manuscript. the sentences in the dataset are in the order in which the manuscript was originally written. we grouped the sentences into bins of 25 sentences and colored the histogram bins based on exclusive combinations of harm categories observed in that 25 sentence chunk. we ran the same translation prompt on the german translation of the manuscript and followed the same model of doing 2 passes. as we stated above, we want to understand if these trends regarding safety are a result of the contents of the manuscript or related to the fact that ottoman turkish is a lowresourced language. the original de-en dataset consisted of 1699 sentences. in the first pass, 363 sentences, or 21% of the dataset, was not translated. in the second pass, 36 more sentences were translated, meaning that only 328 sentences, or 19% of the original dataset, were left untranslated. one sentence was not translated without triggering any safety flags or returning any response from the model in both passes. we removed that sentence from the untranslated sentences dataset and ended up with 327 sentences for analysis. we ran the same quantitative analysis on this dataset as with its ottoman counterpart. confident in its classification of harassment. figure 2: severity score -probability score plot for flagged german sentences the main difference between the two languages is in the classification of severity. for the exact same manuscript, gemini classified more of the flagged sentences with less severity for german than for ottoman turkish in 3 harm categories, shown in the severity comparison in figure 3 . note that the difference is not statistically significant. as shown in figures 4 and 5 , our findings with the distribution of the blocking across the manuscript show that the safety triggers are not random. looking at the ot histogram, we see that hate speech, dangerous content and harassment category is more prevalent towards the beginning and the end of the manuscript. those are the sections where osman agha is on the move: he is captured early in the narrative and towards the end, he flees captivity in disguise, traveling across austria. the parts marked as sexually explicit correspond to the parts of osman's story when he is developing a relationship with an austrian noblewoman, after the woman's husband passed away. the categories harassment and hate speech and harassment are distributed all across the manuscript. considering the nature of the story, it makes sense to see these two distributed across rather than clustered. these factors reaffirm our proposition that there is a relationship between the contents and the safety flags. ## analysis below we offer a close analysis of three examples of blocked sentences in the ot-en dataset. ## discussion osman agha offer a unique opportunity for investigating the relationship of translation and ai safety. this manuscript is not calling for violence or depicting gory scenes. but it contains descriptions of warfare and death, which are despite their saddening nature, part of the realities of life in the past as well as today. this manuscript does not contain any pornographic depictions, but it does mention sexual assault and at times narrates sexual affairs. osman agha does not call for hate towards any particular group, but he does use historical terms, like infidel, which are not appropriate to use in reference to people today. the diversity of themes covered in this manuscript means that there are a variety of sections that can helped us infer more about the ai safety implementations behind gemini. we recognize the importance of ai safety settings, especially when it comes to incredibly large models like gemini. however, translation and chat are not equivalent tasks. google is actively encouraging the use and deployment of gemini in translation, evident in their report @xcite and in their investment in developing the mtob benchmark @xcite . translation is a standard use case in their basic usage examples. yet, the company does not offer any details about how they see their safety settings interacting with translation. with an ever-increasing context window, it will be remarkably easy to miss a few sentences that were left untranslated. and those sentences might be exactly the ones that a victim of personal or structural violence needed to express to the rest of the world. mistakes in translation stand out. refusals to translate however can be hidden away, behind code that is designed to move onto the next sentence if it encounters an 'error'. osman agha's experiences, although sometimes not very pleasant to read, are not far from the experiences of palestinians or ukrainians, among other groups experiencing warfare in today's world. we need to ensure that ai safety implementations do not silence victims and underprivileged groups. ## implications llms are useful tools to historians, especially for those working with languages like ot that are otherwise not served by existing language technologies. historical research, whether it is testing llms on real, historical data instead of fictional test cases, or applying historical critical thinking to technologies, offers a unique perspective to computational studies. in lieu of a conclusion, we would like to offer some thoughts regarding the implications of our work. on january 28, 2025 guardian reported an interesting finding about how deepseek did not answer questions about tiananmen square in its chat interface. many reputable news agencies conducted their own analysis into this issue, including the cnn world, which used this title in its reporting: "deepseek is giving the world a window into chinese censorship and information control." on february 12, 2025, openai released an updated model spec. 1 this document contains several examples of prompts that highlight how the ope-nai models are supposed to respond in different scenarios. these scenarios include political and politicized questions that offer insights into ope-nai policies. in conjuncture with these developments, our research offers an in-depth study of one model, gemini 1.5, and through our examples, we offer a window into information control in a closed source model. users can indeed change the safety settings of gemini in api calls, much like they can run their own instance of deepseek without the layer that prevents it from responding to questions related to tiananmen square. in either case, however, studying these models as artifacts from the perspective of history and philosophy of science tells us something about the production context and use cases of these models. who is designing these technologies and for whom are these technologies designed? whose experiences do not meet the threshold of safety requirements or information policies of companies and governments? these questions are central for our understanding and evaluation of ethics of science and technology, and their impact on society today. ## ethical considerations our work enables the examination of ethical issues related to ai safety and content moderation in ai models without posing risks to contemporary individuals. it is crucial to understand how these safety mechanisms handle complex accounts that may contain harmful content. by testing llms on historical accounts, we can study the impact of these safety decisions without exposing the stories of people who are alive today to these models. osman agha's account is over 300 years old and his immediate relatives passed away long ago. this minimizes the risk associated with incorporating his experiences, however challenging they may be, into these models. additionally, we address concerns related to the use of the translation. while the original manuscript is no longer under copyright, its translations are protected. therefore, we must ensure that the data is shared in a manner that prevents the illegal recreation of the translation. we are committed to handling the translated material responsibly to avoid any unauthorized distribution