Machine Translation for L ivonian: Catering to 20 Speakers (2022)

Abstract: Livonian is one of the most endangered languages in Europe with just a tiny handful of speakers and virtually no publicly available corpora. In this paper we tackle the task of developing neural machine translation (NMT) between Livonian and English, with a two-fold aim: on one hand, preserving the language and on the other – enabling access to Livonian folklore, lifestories and other textual intangible heritage as well as making it easier to create further parallel corpora. We rely on Livonian’s linguistic similarity to Estonian and Latvian and collect parallel and monolingual data for the four languages for translation experiments. We combine different low-resource NMT techniques like zero-shot translation, cross-lingual transfer and synthetic data creation to reach the highest possible translation quality as well as to find which base languages are empirically more helpful for transfer to Livonian. The resulting NMT systems and the collected monolingual and parallel data, including a manually translated and verified translation benchmark, are publicly released via OPUS and Huggingface repositories.

## introduction many state-of-the-art natural language processing tasks have reached admirable quality on languages with abundant linguistic resources @xcite @xcite . furthermore, some neural language models and translation systems have been created for 100 and more languages (e.g. @xcite . however smaller, less or not at all spoken languages continue to struggle not only in terms of applicable computational approaches, but more critically -in terms of usable resources for training natural language processing (nlp) models or even just linguistic exploration. in this paper we set the goal of developing machine translation between english and livonian. currently there are just over 20 fluent speakers of the language @xcite . although some digital linguistic resources exist for livonian (including a dictionary with example sentences and a written monolingual corpus, @xcite , there is virtually no open parallel corpora between english and livonian, with the single exception of 35 parallel sentences in the opus tatoeba corpus @xcite . at the same time, cross-lingual transfer learning has recently helped improve the performance of several low-resource nlp tasks with the support of related languages (e.g. @xcite . this also includes zero-shot translation @xcite , the ability of multilingual nmt systems to translate between seen languages that were not represented in the parallel training data as a pair. the case of livonian is especially interesting in this regard, as there are two different sources of such support: on one hand, it is a uralic language, closely related to estonian and finnish. on the other hand, livonian has taken part in forming latvian language and livonian speakers have historically co-existed side-by-side with latvian speakers. as a result of mutual influence these two languages also share a number of grammatical, lexical and orthographic similarities. our main contributions are two-fold. first, we collected the majority of digitally available translation examples including livonian into a small parallel corpus (just over 10000 sentence pairs) of mostly livonian-latvian and livonian-estonian sentence translations with very few (1000) livonian-english examples. in order to create a clean benchmark for evaluating translation quality we selected a portion (about 10%) of this corpus and had it manually translated into latvian/estonian/english so that each sentence would have all four manually verified translations. 1 the second half of our work focuses on neural machine translation (nmt, @xcite , mainly targeting livonian↔english. we explore several options of coping with the extremely lowresource settings and use estonian and latvian for cross-lingual transfer. our experiments answer the following research questions: next we briefly describe the livonian language in section 2, then introduce the collected parallel and monolingual data in section 3. section 4 provides the details of our nmt experiments and section 5 concludes the paper. ## the livonian language livonian (iso 639-3: liv) is a finnic language indigenous to latvia and belonging to the uralic language family. during the 12th century livonian was spoken across great territories in latvia around the gulf of riga. over time, livonian areas gradually became latvian-speaking. in the 19th century, livonian still had approximately 2500 speakers, by 1 translation from livonian was a too rare and expensive service, thus we resorted to translating from one of the other three languages and instead had livonian speakers check the results for meaning correspondence afterwords. the mid-20th century around 1500 speakers. nowadays livonian is listed in unesco's atlas of the world's languages in danger as a critically endangered language @xcite . according to the 2011 census, there are 250 livonians in latvia. although there are just over 20 people who can speak the language, the livonian community is active in preserving and developing the livonian heritage @xcite and language plays a key role in this process @xcite . the livonian language developed in the contact area of baltic and finnic languages. livonian and latvian share a similar geographical location over a prolonged period of time, as a result of which they both contain traces of contact. next to other loanwords, the livonian loanword strata consists of words borrowed from latvian @xcite and vice versa. the most obvious latvian influence on livonian grammar is found in the livonian case system @xcite . livonian has the prosodic characteristics typical of a finnic language such as word-initial stress and the phonological opposition of short and long phoneme duration. it is the only finnic language that differentiates lexical tones -the plain tone and the broken tone or stød -and therefore shares similar characteristics with latvian as well as danish @xcite . the main sources of data included livonian-latvian as well as livonian-estonian translations. thus we use these two languages as base for crosslingual transfer and e.g. leave finnish out, as there was no data for it. the sources of data included: concerning sentence alignment, the dictionary examples consisted of already aligned livonian sentences. we aligned the rest of the data manually with the help of language experts -first on paragraph level, then on sentence level. the resulting amount of sentences in the resulting dataset is shown in we separated balanced portions of development (503 sentences) and evaluation (749 sentences) splits from the full dataset. the splits are balanced in terms of the original source of the texts to resemble proportions from the remaining training data. ## machine translation experiments having just over 10, 000 parallel examples constitutes extremely low-resource settings for neural machine translation. added to this, the number of monolingual livonian sentences (about 40, 000) is also too small for approaches like unsupervised machine translation @xcite . we implement the support of neighboring and related languages (estonian and latvian) via multilingual machine translation @xcite . as a first step the model is pre-trained with the larger languages (estonian, latvian, english) and then used as base for following experiments. we also perform iterative back-translation (pinnis et al., 2018) to make use of the large amounts of monolingual news data in en/et/lv, and our limited amount of monolingual data in liv. we translate the 40k liv sentences and different batches of 200k sentences from the other languages into all directions, filter the translations using simple heuristic filters @xcite , and use a mix of all back-translated data with an equal amount of random clean parallel data (including all data involving livonian) to fine-tune the base model. ## conclusion in this paper we presented a novel dataset for the highly endangered livonian language, which can be useful for machine translation, language modelling and many other natural language processing and computational linguistic research tasks. in our experiments we show how far one can get in training modern machine translation models with very scarce data, and which languages are more suitable for transfer learning when working with livonian data. while perhaps not being usable as-is in any kind of production scale, the achieved final bleu scores of 19.01 for @xmath0.03 for @xmath1. in the future we are planning to experiment with cross-lingual transfer from other languages, like the resource-rich finnish as well as resource-poor finno-ugric languages like võru and sami @xcite . given the limited amount of existing monolingual livonian data, generating synthetic livonian data with other means besides backtranslation might be helpful: for example, forwardtranslation or using gpt-like language models. finally, work on the already collected livonian monolingual and parallel data is ongoing at the institute of the livonian language. adding english translations to the lexical items and example sentences is an ongoing effort and will evaluate in practice, if the mt systems created as part of the current work can facilitate that. one of the key focuses is also manually verifying the data and making sure the existing corpus contains correct livonian texts and their translations