Using Large Language Models to Transliterate Endangered U ralic Languages (2024)

Abstract: This study investigates whether the Large Language Models are able to transliterate and normalize endangered Uralic languages, specifically when they have been written in early 20th century Latin script based transcription systems. We test commercially available closed source systems where there is no reason to expect that the models would be particularly adjusted to this task or these languages. The output of the transliteration in all experiments is contemporary Cyrillic orthography. We conclude that some of the newer LLMs, especially Claude 3.5 Sonnet, are able to produce high quality transliterations even in the smaller languages in our test set, both in zero-shot scenarios and with a prompt that contains an example of the desired output. We assume that the good result is connected to the large presence of materials in these languages online, which the LLM has learned to represent.

## introduction there is a long tradition of publishing transcribed texts on various uralic languages using finno-ugric transcription system, also known as uralic phonetic alphabet. recently also ipa has become more commonly used, but various transcription systems and their interpretations are widely used. thousands of pages of linguistic materials are printed or archived that use these transcription systems. when the materials are being digitized, we need to address several questions that relate to how the texts are ideally treated for contemporary use. this includes both the use within the scientific community and the language communities, which may have different needs. yet there is a shared starting point that the older transcription systems are not ideal for all modern applications. the use of orthographies in contemporary language documentation has been seen preferable to phonemic or detailed phonetic transcription @xcite , and also in this study the primary question is how to transform the transcriptions into currently used orthographies. the use of some other systematic transcription system could be likewise suggested, and it seems reasonable to expect that our approach could be extended to other comparable transformations. the target chosen here, contemporary orthography, comes with its own problems that some other choices would have avoided. there is inevitably more subjectivity in the details of the result than what i.e. phonemic representation would entail. transforming a transcription into orthography moves along the blurry boundary of transliteration and normalization. the task is not only a simple transliteration. often the exact language variety in the transcription matches only partially with the contemporary written standard. this makes the desired normalization vaguely defined, as we would like to keep some amount of dialectal features that are present in the text, but not necessarily everything. especially the dialectal morphology and lexicon should remain recognizable, if possible. we define the desired output as something comparable to what contemporary language documentation material would reasonably be expected to look like, when collected from the same dialect today using the current orthography. another expectation would be that the result would be close enough to the contemporary orthography that the modern language community members can read it, and the modern language technology tools are able to process it, possibly with minor modifications. for both of these purposes the methods to mark dialectal features that are already attested and used in the literary language would be ideal choices. ideally different versions would be stored together, possibly with alignation at word or sentence level. thereby the issue of information loss in the orthographic version is not a problem for the entire dataset. with larger uralic languages, earlier work has been done on transliteration of dialectal finnish texts. @xcite showed that detailed dialectal transcriptions can be accurately converted to modern literary finnish using neural networks. still in the finland's context, @xcite extended this work to the swedish spoken in finland. these studies connect closely to the current work as the goal has been to process scientific transcriptions. @xcite describe in detail the wider workflow into which transliterations connects to. this includes ocr or htr, transliteration and also audio processing. grapheme-to-phoneme conversion can be seen as a sister task to what is undertaken here, as the aim is to convert orthographic text into phonemic or phonetic realization. @xcite performed a complex evaluation of various related tasks, including grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. in their test no single model outperformed in all tasks. fetrat @xcite tested persian grapheme-to-phoneme conversion with llms and reported better performance than the traditional methods. ## data and experiment design our dataset contains digitized transcriptions in unicode characters and their paragraph level correspondences in the contemporary cyrillic orthographies. the languages included are komi-zyrian, udmurt, northern mansi and kildin saami. the data comes mainly from the publications of the finno-ugrian society and represent mostly early 20th century fieldwork written in finno-ugric transcription. in the case of kildin saami the selected text is a religious translation, but we believe it was still suitable for this experiment. these materials represent a very prominent and potentially underused data source in the uralic studies. the linguistic sources include: udmurt texts published in @xcite , edited by d. r. fuchs. the northern mansi example is published in @xcite , which is part of a large collection of mansi texts in several volumes. the komi-zyrian example is published in @xcite . these are published by the finno-ugrian society foot_0 . as stated, the kildin saami text differs from the other examples. it was published in genetz (1879) by a hungarian publisher, it represents a religious genre and, given its age, it is clearly in the public domain. the latter appears to be true for all of the earliest contributors to these materials. the materials used in this study have been published in github foot_1 , which allows replicating the results and makes it easy to test the outcome with different preferences and choices. the when different large language models are tested, it becomes clear that most of them are not able to process finno-ugric transcriptions. there are hundreds of models, and evaluating most of them would be entirely unnecessary in this task. the output is usually unintelligible with clearly no understanding about these languages. at the same time we see that there are individual models that do perform above the average. four different models were selected and tested further. these are claude 3 and 3.5 sonnet foot_2 , gemini 1.5 pro foot_3 and chatgpt 4o foot_4 . we conduct two different experiments, first is a zero-shot scenario where the model is asked to transliterate the text with no additional information. this is not an ideal setting, since as explained, it is not obvious what kind of representation is actually wanted. however, it gives information about the model's capabilities. it can be questioned why a zero-shot experiment is needed, as it is clear that example data should improve the result, but at the same time zero-shot scenario does give valuable information about what the models are capable to do without any further guidance. in the second experiment the same prompt is used, but there is an additional example provided, which illustrates the style and type of transformation that is desired. this example is taken from the same text as the transliterated sample, so that the dialect is certainly the same. the test text is one paragraph and the added example little bit longer. ## results when comparing the tested models, claude 3.5 sonnet performs distinctly from the others. as shown in some of the models display high word and character error rates. however, close to 5% character error rates we see in komi and udmurt are certainly useful already, and the result for mansi is in the same category. in order to get a more concrete overview, let's compare some of the transliterated sentences. the mistakes are underlined. ## 84 it is noticeable that in many cases the mistakes are matters of interpretation, especially for claude 3.5 sonnet. for udmurt the model returns dialectal forms such as чирккем, but then also normalizes forms such as вордконо into more literary вордсконо. this illustrates that part of the error connects to the line between dialectal representation and orthography. in the komi example one mistake appears in the settlement name пустыняын, which is replaced by пустӧйын. interestingly, the claude 3.5 sonnet model did return a valid but wrong word in komi. this would hint that this model is somehow more aware about the komi lexicon than the others. with the kildin saami the results are much worse than with the other languages. especially the diacritics seem to cause problems with all the models. kildin saami is the only language where claude 3.5 sonnet performs worse than the other models in the first experiment, although it outperforms others in the second experiment. compounds are another regular issue. many transcriptions seem to use hyphen in possible compounds where the current literary language does not. partially the use in transcriptions seems to be guided by etymological assumptions, i.e., parts of the word that could be analysed as etymologically distinct words are differentiated with a hyphen. reanalysis of the word boundaries has inevitable impact to the word error rate as well. this could also make the word level alignation of different versions a challenge. between the first and second experiment, it seems that especially northern mansi diacritical marks in the cyrillic orthography improved significantly when an additional example was provided. with kildin saami similar phenomena could had been expected, as the language has similarly complex macron usage, but in our kildin saami example the improvement still left the result way worse than with the other languages and the use of diacritics did not become very close to what is expected in the current orthography. how difficult this task is in general should be separately evaluated, but it clearly is far from trivial. converting these transcriptions into contemporary orthographies can be a challenge even for a specialist in these languages. especially so if we want to take the dialectal features somehow into account. we have not yet tried to estimate whether some of the transliteration tasks are objectively harder than the others. it is possible that the phonetic representation of the transcription is more complicated in case of some languages, and the dialects in these examples may differ to varying degrees from the literary languages. also, in some languages there may be conventions to show the dialectal features in the orthographical texts. one reason why the komi result is so good may be connected to extensive scanning and digitization work carried out in the komi republic, which has made komi materials widely available online. 6 similarly it could be reasonable to assume that the kildin saami results were worse than the others because the amount of text in this language available online is likely much smaller than on the others. there are no clear and up to date statistics about this, but there have been projects that have collected texts from the internet in different languages. @xcite report their results from 2017, where they have 59 sentences for kildin saami, 825 for northern mansi, 18,966 for komi-zyrian and 42,545 for udmurt. again, this is certainly not entirely reflective, as the amount of komi materials is certainly much larger, and there is a northern mansi newspaper лӯима̄сэрипос with already a decade of online presence and articles in html format. @xcite measured that between 2013 and 2019 the size of the corpus produced already by these newspaper articles is over half a million tokens. individual corpus creators may have had different well considered reasons to include and exclude various sources. however, with the modern llms the assumption must be that all materials that have been placed online may be collected and used to create these models. with the case of claude 3.5 sonnet model this data collection must have been particularly successful and wide, including numerous minority languages. also other recent studies have indicated that claude 3.5 sonnet has returned very proficient translations between russian, azerbaijani and lezgian @xcite , which matches well with our results with the uralic languages spoken in russia. similarly shandilya and palmer (2024) had the best results with claude 3.5 sonnet in glossing endangered languages with retrieval-augmented generation. there is the possibility that many minority language materials that are online contain various issues and different solutions with the character encoding. whether the authors of the large language models have systematized this type of problems, or found ways to harmonize them otherwise, may have a large impact to the final result. there are also large amounts of text online that are missing some of the officially used characters, as inputting them is not always possible, which may impact the ability of the models to output them in the correct positions when needed. ## conclusion we are starting to see large language models that are able to process endangered uralic languages at a very advanced level. the superiority of an individual model is fleeting, and in some months we expect to see new models with similar and even better capabilities. still, the high accuracy of claude 3.5 sonnet is beyond the results we see at the moment with other models that can be publicly tested to any extent. as other models develop similar capabilities, it would be important to evaluate them accurately against transliteration and other related tasks. our results show that in a language specific task such as transliteration the differences between different models can be surprisingly large, and some are capable of producing close to a correct output. as far as we have been testing these models in last years, this is the first time an llm has been able to process this proficiently smaller uralic lan-guages. this is in itself a major development, and their capabilities should be extensively tested against different tasks. these could include machine translation, interlinear glossing, disambiguation or dependency parsing, just to mention a few usually manual work phases that the researchers of uralic languages have been engaging with very regularly, and where automatization could have a major impact. transliteration and normalization in themselves are tasks that the researchers may not have performed that often before. we have only recently started to receive high quality unicode versions of the older transcriptions, and thereby the need may have not been acute yet either.