Neural Machine Translation for the Indigenous Languages of the A mericas: An Introduction (2023)

Abstract: Neural models have drastically advanced state of the art for machine translation (MT) between high-resource languages. Traditionally, these models rely on large amounts of training data, but many language pairs lack these resources. However, an important part of the languages in the world do not have this amount of data. Most languages from the Americas are among them, having a limited amount of parallel and monolingual data, if any. Here, we present an introduction to the interested reader to the basic challenges, concepts, and techniques that involve the creation of MT systems for these languages. Finally, we discuss the recent advances and findings and open questions, product of an increased interest of the NLP community in these languages.

## introduction more than 7 billion people on earth communicate in nearly 7000 different languages @xcite . of these, approximately 900 languages are native of the american continent @xcite . most of these indigenous languages of the americas (ila) are endangered at some degree @xcite . this huge variety in languages is simultaneously a rich treasure for humanity and also a barrier to communication among people from different backgrounds. human translators have been important in overcoming language barriers. however, trained translators are not accessible to everyone on earth and even scarcer for endangered and minority languages. the need for translations is even written in the constitutions of several countries like mexico, peru, paraguay, venezuela, and bolivia @xcite to allow native speakers to have equal language rights regarding law. this is why developing mt is crucial: it helps humanity overcome language barriers while simultaneously allowing people to continue using their native tongue. however, the challenges to achieving these problems are not trivial. it is not only the amount of available data (a common thesis among the nlp community) but also a set of challenging issues (dialectical and orthographic variations, noisy texts, complex morphology, etc.) that must be addressed. mt has always been an important task within the larger area of natural language processing (nlp). in 1954, the georgetown-ibm experiment @xcite was the first that showed at least some effectiveness of mt. further research resulted in rule-based systems and statistical models. in 2023, neural models define state of the art for mt if training data is plentiful -i.e., for so-called high-resource languages (hrls) -and have also achieved impressive results for low-resource languages (lrls). mt is also the most studied nlp task for the ila @xcite . the common issue among these languages is the extreme low-resource conditions they are confronted with. the research interest for these languages has increased in the last years, including the recent americasnlp 2021 shared task @xcite on 10 indigenous languages to spanish, and the wmt (conference on machine translation) shared task for inuktitut-english @xcite . in this work we aim to provide a comprehensive introduction to the challenges that involve creating mt systems for ila, and the current status of the existing work. we organize this work as follows: we start by introducing state-of-the-art nmt models ( §2). then, we discuss the current challenges for these languages ( §3); and we introduce the key concepts related to low-resource nmt and the implications for endangered languages of the americas( §3). this is followed by a discussion of available data ( §4). afterwards, we introduce the important concepts for lrl and endangered languages ( §5); then we introduce the main strategies aimed at improving nmt with limited training data ( §6); and finally we give an overview of the work done for ila on mt ( §7). in doing so, we provide insights into the following questions: which systems define the state of the art on low-resource nmt applied to the ila? what is the route that ahead to improve the translations of the ila? ## background and definitions formally, the task of mt consists of converting text x in a source language l x into text y in a target language l y that conveys the same meaning. 1 translating text @xmath0: x and y can be of variable length, such as phrases, sentences, or even documents. if other languages are used during the translation process, e.g., as pivots, we denote them as l 1 , . . . , l n . we refer to a corpus of monolingual sentences in language l i as m l @xmath1, ..., s n . probabilistic modeling and data when using probabilistic mt models, the goal is to find @xmath2, given @xmath3. under the supervised machine learning paradigm, a parallel corpus c @xmath4, y 1 ), ..., (x n , y n ) is used to learn a set of parameters θ, which define a probability distribution over possible translations. given c parallel , the training objective of an nmt model is generally to maximize the log-likelihood l with respect to θ: within this overall framework, there are a number of design decisions one has to make, such as which model architecture to use, how to generate translations, and how to evaluate. decoding decoding refers to the generation of output ŷ , given the parameters θ and an input x. often, decoding is done by approximately solving the following maximization problem: most nmt systems factorize the probability of ŷ = ŷ1 , ..., ŷt in a left-to-right fashion: thus, the probability of token ŷt at time step t is computed using the previously generated tokens ŷ<t , the source sentence x and the model parameters θ. common algorithms for finding a highprobability translation are greedy decoding, i.e., picking the token with the highest probability at each time step, and beam search (lowerre, 1976). ## challenges and open questions in an overview of the datasets and recent studies of mt for the ila, we found the following main issues to be handled. extreme low-resource parallel datasets even with the recent advances, the resources available to train mt systems are extremely scarce, having training set between 4k and 20k sentences (see §4), with notable exceptions for inuktitut, guarani and quechua @xcite . lack of monolingual data most of these languages are mostly used in spoken form. in recent years, with the advancement and democratization of mobile technologies, indigenous languages have seen a slight increase in massaging systems and private spheres @xcite . however, the usage of these languages on the internet is rather limited. even wikipedia has a limited amount of these languages @xcite . low domain diversity . as most parallel datasets are scarce, they are restricted to a small number of domains, making it challenging to adapt it, or try to aim for general translation models. this has been recognized as a major problem during the americasnlp st @xcite . rich morphology an important number of these languages are morphological highly rich. in many cases, we find polysynthetic, with or highly agglutinative languages @xcite or even fusional phenomenon @xcite . ## available mt datasets for ila the parallel datasets available for mt have been increasing during the last years. at this moment, we can show in two folds the development of these resources: as shown in language-specific corpus collection work has been done for many languages, where parallel corpus has been the main component. in recent time we have seen cherokee-english (opus) @xcite , wixarika-spanish @xcite , shipio-konibo @xcite , and others (see for wide-spoken languages like guarani, it is even possible to collect a web crawled dataset, including news articles and social media parallel aligned data @xcite this dataset also includes monolingual data. this is possible as guaraní is one of the most spoken indigenous languages of the continent. in contrast to the language-specific datasets, we find broader approaches (see recently community-driven research groups have started the creation of own parallel datasets, such as masakhane @xcite for african languages, and americasnlp for indigenous languages of the americas @xcite . the americasnli dataset is an important effort to have a common evaluation benchmark for the 10 indigenous languages of the americas for the mt and nli tasks. given the constitutional rights of indigenous languages in many countries of the americas, it is possible to access this data. @xcite made available this resource during their shared task system development. finally, it is important to mention that many of the languages spoken in the americas have wikipedia's set of articles available 4 . collection of new data a common way to create parallel data with the help of bilingual speakers is via elicitation (translating the foreign text into another language). it has the disadvantage of biasing the created text to forms and topics, culture, and even grammatical forms towards the source language @xcite . a method that avoids this problem is language documentation, which consists of storing and annotating commonly used speech or text @xcite . however, it is costly and requires specialists. in this process, involving the community members that are bilingual speakers is important @xcite . ## low-resource mt for the purpose of this paper we define lrls as languages for which standard techniques are unable to create well performing systems, which makes it necessary to resort to other techniques (cf. figure 1 ) such as transfer learning. for mt, the amount of available resources differs widely across language pairs: some have less than 10k parallel sentences, while other have more than 500k, with some exceptions in the orders of several million. emulating a low-resource scenario by downsampling available data for high-resource languages is common and helps understanding a model's performance across different settings. however, further evaluating methods on a diverse set of low-resource languages is crucial, since many languages exhibit particular linguistic phenomena @xcite , that perturb the final results, especially since most large datasets are from the indo-european language family, to which only 6.16% of the world's languages belong @xcite . importantly, there is no strong correlation between the number of resources available per language and the number of speakers: javanese with 95 million speakers and kannada with 44 million are considered lrls, while french, with only 64 million native speakers, is among the most widely studied languages. improving models to handle lrls will extend access to information online as well as human language technology to all monolingual speakers of those languages. in the case of ila, most languages are endangered at some degree, but most of them have the same issue: they are low resourced for parallel and monolingual data. endangered languages krauss (1992) estimates that 50% of all languages are doomed or dying, and that in this century we will see either the death or the doom of 90% of all human languages. the current proportion of languages that are already extinct or moribund ranges from 31% down to 8% depending on the region, with the most severe cases in the americas and australia (simons and @xcite . to determine how endangered a language is, @xcite proposes a classification scale called egids with 13 levels. the higher the number on this scale, the greater the level of disrup-tion of the language's inter-generational transmission. 5 mt for endangered lrls has the potential to help with documentation, promotion and revitalization efforts @xcite . however, as these languages are commonly spoken by small communities, or indigenous people, researchers should aim for a direct involvement of those communities @xcite . what is polysynthesis? a polysynthetic language is defined by the following linguistic features: the verb in a polysynthetic language must have an agreement with the subject, objects and indirect objects @xcite ; nouns can be incorporated into the complex verb morphology @xcite ; and, therefore, polysynthetic languages have agreement morphemes, pronominal affixes and incorporated roots in the verb @xcite , and also encode their relations and characterizations into that verb. the most common word orders present in these languages are sov, vso, svo and free order. it is important to notice that a polysynthtic language can have a aggutinative foot_6 or can have also fusional characteristics, like totonaco or tepehua @xcite . ## low-resource mt paradigms most languages of the americas do not have high amount of data for mt. therefore, we introduce the most important paradigms to improve lowresourced machine translation. figure 1 shows a general overview of the methods and options to improve lrl mt. for a more detailed understanding of this techniques we refer the reader to specialized low-resource mt surveys @xcite @xcite . ## advances in mt for the indigenous languages of the americas in recent years the interest in mt for indigenous languages of the americas has increased. the task is not easy. the first usage of nmt systems has not been successful @xcite . however, with the use of lrl mt methods, we have witnessed great improvements. the cherokee-english @xcite ) language pair has been explored using a pre-trained bert @xcite for the english side. a system demonstration of this approach is also accessible @xcite . the back translation strategy for bribri-spanish nmt transformers has also been explored @xcite and by @xcite (for four peruvian languages to spanish) with good results. the scarce indigenous language monolingual text can be replaced to some extent with spanish text or extracted from pdfs, and other sources (bustamante et al., 2020). one of the main challenges for the complex morphological languages in the area has been the prepossessing step. @xcite show that even if morphological segmentation has less perplexity a the language modeling time, it is still under-performing or equivalent against bpes for mt (for inuktitut--english, yupik-english data, guaraní-spanish data). a more comprehensive (on the segmentation modeling side) was done by @xcite exploring a wide array of segmentation models.the latter study showed that supervised morphological segmentation underperform unsupervised. however, unsupervised morphological segmentation like lmvr @xcite and flatcat @xcite perform better than bpes. @xcite studied how better to perform word segmentation for the inuktitut-english pair. they found that for this language pair, a morphological segmentation, or a combination of bpes and morphological segmentation, works better than just applying vanilla bpes. also, training word embeddings for guarani-spanish translation is an excellent opportunity to increase the mt performance of these languages @xcite . the usage of transfer learning from multilingual systems has been tried, with limited results @xcite (training an own t5 model for indigenous languages) and @xcite . however, pertaining a spanish-english model together with ila, and then fine-tuning it (together with a careful prepossessing and filtering step) has been the most successful strategy @xcite . the quality of mt systems of ila has been a constant debate. however, @xcite shows that the quality of mt for these languages is enough to improve other tasks like natural language inference (nli). inuktitut-enlgish st the wmt 2020 news translation task included inuktitut-english translation @xcite . the participating systems explored the difficulties of working with a polysynthetic language in a medium resource scenario. participating teams in this competition were: @xcite @xcite @xcite @xcite . americasnlp 2021 and 2023 st in 2021, the americasnlp community organized a workshop on machine translation for 10 indigenous languages of the americas in 2021 @xcite and 2023 @xcite with an additional indigenous language (chatino). the americasnlp shared task winner was @xcite in 2021, and a more mixed result in 2023 foot_7 . other participants in this shared task are @xcite @xcite @xcite . it is important to point at the importance of clean datata. for quechua, @xcite got the best results generating an additional amount of clean data. americasnlp 2022 competition is a competition on speech-to-text translation is organized and is targeting the following language pairs: bribri-spanish, guaraní-spanish, kotiria-portuguese, wa'ikhana-portuguese, and quechua-spanish @xcite 8 . ## ethical aspects when working with ilas are also interacting with communities and nations that speak these languages. in most cases, these speakers have been exposed to a colonial past, or to a local oppression, by the majority language and culture. it is important to point to best practices and recommendations when performing our research. bird (2020) and @xcite advocate to include community members as co-authors @xcite as well as considering data and technology sovereignty. this is also aligned with the community building aimed at by @xcite . @xcite summarizes the main aspects that should be considered as follows: i) consultation, negotiation and mutual understanding. it is important to inform the community about the planned research, negotiating a possible outcome, and reaching a mutual agreement on the directions and details of the project should happen in all cases. ii) respect of the local culture and involvement. as each community has its own culture and view of the world, researchers should be familiar with the history and traditions of the community. also, it should be recommended that local researchers, speakers, or internal governments should be involved in the project. iii) sharing and distribution of data and research. the product of the research should be available for use by the community, so they can take advantage of the generated materials, like papers, books, or data. ## conclusion machine translation for ila has gained interest in the nlp community over the last few years. here, we provide an exhaustive overview of the basic mt concepts and the particular challenges for mt for ila (in the context of low-resource scenarios and its relation to endangered languages). we additionally survey the current advances of mt for these languages.