Assessing Large Language Models in Translating C optic and A ncient G reek Ostraca (2024)

Abstract: The advent of Large Language Models (LLMs) substantially raised the quality and lowered the cost of Machine Translation (MT). Can scholars working with ancient languages draw benefits from this new technology? More specifically, can current MT facilitate multilingual digital papyrology? To answer this question, we evaluate 9 LLMs in the task of MT with 4 Coptic and 4 Ancient Greek ostraca into English using 6 NLP metrics. We argue that some models have already reached a performance apt to assist human experts. As can be expected from the difference in training corpus size, all models seem to perform better with Ancient Greek than with Coptic, where hallucinations are markedly more common. In the Coptic texts, the specialised Coptic Translator (CT) competes closely with Claude 3 Opus for the rank of most promising tool, while Claude 3 Opus and GPT-4o compete for the same position in the Ancient Greek texts. We argue that MT now substantially heightens the incentive to work on multilingual corpora. This could have a positive and long-lasting effect on Classics and Egyptology and help reduce the historical bias in translation availability. In closing, we reflect upon the need to meet AI-generated translations with an adequate critical stance.

## introduction translations have been the cornerstone of scholarly activity in the fields of classics and egyptology since their inception, serving both academic and public dissemination purposes @xcite . the initial preference for latin as the target language for translations reflects its status as the scholarly lingua franca during the early phase of these disciplines @xcite . over the centuries, there has been a steady transition to vernacular languages in order to make scientific content more accessible for an attempt to partly reverse this transition; see merisalo 2015 for the example of the italian language). in contemporary practice, english has emerged as the preferred lingua franca, broadening the accessibility and scope of translated texts @xcite . this article focuses specifically on english translations in a field tangent to both classics and egyptology, namely digital documentary papyrology 1 . in this specific area of study, interdisciplinary communication between classics and egyptology have improved slightly in last decades @xcite . however, english translations of primary sources are not yet widely available on the web, as the next section shows. currently, virtually all available translations of ancient greek and coptic texts have been made by human experts. the coverage of those translations reveals major disparities. on papyri.info 2 as of 5 @xcite greek texts with transliteration are available, but only 5,678 are accompanied by translations in english, and 628 in other languages, i.e. around one tenth of the total corpus. the situation is even more critical for coptic, where out of 2,099 texts, 2,049 are untranslated, and only 50 texts are available in english, french or german, that is less than one per cent of the total corpus. if other translations exist, they are mainly printed and are not easily accessible online, making them unsuitable for research in digital papyrology. meanwhile, the landscape of ai-generated translations has evolved considerably, from simple rule-1 "digital papyrology can be defined as the whole set of electronic resources and methodologies aimed at creating, storing, accessing, processing, and publishing information pertaining to research and study in the various fields of interest of the papyrological discipline." @xcite . based systems to sophisticated machine learning models. early efforts in computer-aided translation were fundamental, but limited in terms of accuracy and scope. in recent years, the adoption of machine learning models has significantly improved the quality of translations. since its launch in 2006, google translate @xcite has long been the benchmark for machine translation tools, despite the initial lack of support for languages such as coptic or ancient greek. 3 more recently, deepl, introduced in 2017, has set new standards for the accuracy of machine translations (although it does not include the languages in question). the most advanced development in this area concerns large generative multimodal language models (llms), which are serious contenders for complex translation tasks @xcite . ## comparing results (coptic and greek) figure 4 shows the mean performance all surveyed llms achieved on one given metric at text level. it reveals that all mts of ancient greek texts obtained better scores than their coptic equivalents on those metrics. we note that the gap is even bigger in more complex, possibly more meanigful metrics like sacrebleu and meteor. we explain this perceived difficulty of current llms to translate coptic texts at the level of ancient greek texts with the likely very large gap in available training corpus for each corpus. this cannot be said in certainty for llms that are not open source, but a recent digital contribution estimated the overall size of coptic digital papyrology to 102,080 words across 1,973 texts. this represents about two percent of the greek equivalent, 4,926,263 words across 58,975 texts @xcite . otherwise, no likely secondary factors have been found that contribute to the difference in mt performance. the length or completeness of the text appears unlikely to play a role in the eight chosen texts. @xcite 5 future directions this study examines coptic/greek-english mt by llms in only eight texts and could be scaled and improved in multiple directions. ## conclusion the results presented in this paper serve as recommendations for leveraging currently available large language models for the machine translation of ancient greek and coptic texts. egyptologists and classicists seeking to focus on just one of the nine models evaluated will find the specialised coptic translator or the claude model family most beneficial. although the best-performing version, claude opus, requires a paid subscription, the lighter haiku and sonnet models produced nearly equivalent results for ancient greek. the success of the coptic translator, a smaller specialised tool fine-tuned from a larger model, stands out in the context of increasingly capable generalpurpose models. gpt-4o, the newest model included in this pilot study, did not show significant improvement for coptic but did for ancient greek. aside from gpt-3.5, which consistently underperformed, the other three models form a middle tier in performance. however, scholars are not limited to selecting only one model, especially given the rapid development and deployment of new models, alongside related ethical considerations. we advocate for the joint use of multiple models to provide the best support for human experts. clearly, more work is needed to fully understand how scholars can benefit from llm-powered translations. nevertheless, it is evident that multilingual digital papyrology will benefit from the semiautomatic generation of metadata (e.g., translations, summaries) enabled by these technologies. the potential to link and integrate previously monolingual datasets seems to outweigh the risks of hallucination, which can be formally addressed using nlp metrics like those employed in this study.