Curated Datasets and Neural Models for Machine Translation of Informal Registers between M ayan and S panish Vernaculars (2024)

Abstract: The Mayan languages comprise a language family with an ancient history, millions of speakers, and immense cultural value, that, nevertheless, remains severely underrepresented in terms of resources and global exposure. In this paper we develop, curate, and publicly release a set of corpora in several Mayan languages spoken in Guatemala and Southern Mexico, which we call MayanV. The datasets are parallel with Spanish, the dominant language of the region, and are taken from official native sources focused on representing informal, day-to-day, and non-domain-specific language. As such, and according to our dialectometric analysis, they differ in register from most other available resources. Additionally, we present neural machine translation models, trained on as many resources and Mayan languages as possible, and evaluated exclusively on our datasets. We observe lexical divergences between the dialects of Spanish in our resources and the more widespread written standard of Spanish, and that resources other than the ones we present do not seem to improve translation performance, indicating that many such resources may not accurately capture common, real-life language usage. The MayanV dataset is available at https://github.com/transducens/mayanv.

## introduction the mayan language family is spoken in an area covering the modern states of guatemala, belize, and southern mexico @xcite . it consists of around 30 languages grouped into five or six major sub-groups, depending on the source @xcite . most subgroups and most speakers are found today in guatemala, where between 40-60% of the population are native speakers @xcite . mayan languages are relatively healthy, but their presence online and on the global scene in general is almost non-existent. in effect, mayan languages, despite the total number of speakers, are considered to be somewhat in decline: according to @xcite , only around half the population of ethnic mayas are mayan speakers, and the languages are associated in many social contexts to backwardness, ignorance and poverty @xcite . to begin addressing the problem of lack of representation in the digital realm, and increase access to modern technology and information sources for indigenous communities, we seek to develop neural machine translation (nmt) @xcite systems. to train and evaluate them, it is first necessary to produce and curate corpora of all the languages involved. in this paper, we present mayanv, a series of curated parallel corpora between various mayan languages and spanish. the language register in these datasets is informal, familial, and non-domain-specific, which best reflects the most common use of the languages involved. in general, online resources for building working nmt models are very scarce: the two greatest sources for parallel texts are the bible, whose overly formal and potentially archaic language does not reflect most modern use cases, and the parallel texts of the jehova's witnesses website (jw.org), whose language, though much closer to modern usage, is still somewhat divorced from the common, day-to-day activities carried out by most mayan speakers, as our dialectometric analysis suggests. outside these two sources, bilingual and monolingual texts longer than a few thousand sentences are rare and not suited for processing, existing mostly as humanreadable pdf files. because of such scarcity of parallel resources for any mayan language, especially those with just a few thousand, or even a few hundred, speakers, we use the parallel corpora we have built to train and evaluate a number of bilingual and multilingual nmt systems; in particular, multilingual systems have proven effective when dealing with low-resource and underrepresented languages @xcite . while other notable efforts in nmt of lowresource and endangered languages have been carried out recently, such as the no language left behind (nllb) @xcite and mad-lad projects @xcite , these include little to no focus on mayan languages. our paper, in contrast, focuses on a more formal introduction of the mayan languages to the larger natural language processing (nlp) community and on the presentation, curating, and release of parallel datasets that may be used for benchmarking future translation endeavours. the corpora are available at https://github.com/ transducens/mayanv/ . the rest of the paper is organised as follows. the next section offers a historical and linguistic overview of the mayan languages motivated by their relative obscurity amongst the nlp community. section 3 then presents the related work in the field. section 4 presents in detail the extraction and curating of resources for dataset creation and model training, including a dialectometric analysis by which we characterise the dialectal and register divergence between the spanish found in mayanv and the more standard variety of jw.org. afterwards, section 5 describes and evaluates the nmt systems we have built. the paper ends with final remarks and a description of potential future research directions. ## overview of the mayan languages the oldest attested mayan language, referred to as classic maya, dates back to ca 300 bc. written in the the mayan script, it belongs to a tradition corresponding to the few instances in human history in which writing was independently invented, along with the systems developed in ancient egypt, sumer, and ancient china @xcite . figure 1 shows an example of the script. the mayan script is logosyllabic: glyphs may act either as logograms, representing a complete semantic unit, or syllabograms, representing a syllable @xcite . modern mayan languages are written using the latin script, with additional diacritics used to denote features such as vowel length and glottalisation, amongst others. much variation exists in terms of breadth and number of speakers. the three most spoken languages -k'iche', yucatec mayan, and q'eqchiall have between a million and half a million native speakers, and are spoken across different geographical areas in guatemala and mexico @xcite . in contrast, there are some languages with very few speakers and in imminent danger of language death, as is the case of itza', which is only spoken by elderly adults and has fewer than 1 000 native speakers (instituto nacional @xcite . however, despite their size in terms of speakers and geographical extent, as shown in figure 2 , mayan languages are usually not given an official status in their respective countries; their use is widespread in daily activities and familial environments, but they are nearly non-existent in matters of governance, education, mass media, and healthcare @xcite . for example, in guatemala, during covid-19 pandemic, the official online portal to register for the government-sponsored vaccination program was only accessible in spanish @xcite ; to this day it continues to lack any mayan language version. even when official support is said to exist, the actual implementation of educational initiatives remain deficient, as is the case with incipient government-sponsored programs to teach yucatec mayan in schools @xcite . bilingualism with spanish is very common, though not uniform across geographical, population and gender lines: isolated communities often present a high number of monolingual speakers, and men usually exhibit higher proficiency in bilingualism @xcite @xcite . language shift, whereby a mayan language is displaced by either spanish or, less commonly, another mayan language @xcite , occurs more quickly in urban environments, where a lingua franca is expected to be used. literacy is overall low as a result of decades of policy where using native languages in schools was discouraged or even punished @xcite . as a result, many mayan speakers regard the orthography of their own languages as more difficult and inaccessible than that of spanish. change was slowly brought about with the introduction of bilingual educational programs in the early 1990s, which finally led to a continued effort to revitalise the role of written mayan languages. the guatemalan academy of mayan languages (almg), established in 1990, plays an important role in the standardisation of both the orthography of the different linguistic communities and their corresponding spoken languages, while also engaging in literacy and publication efforts. foot_0 a similar role is played by the proyecto lingüístico francisco marroquín foundation (plfm), foot_1 also in guatemala, and national institute of indigenous languages (inali), established in 2003, in mex-ico, foot_2 both of which have published several grammars. mayan languages exhibit a high degree of dialectal variation @xcite . while the most important reason for this is natural language change and historical innovation, the sprachbund of mayan languages in guatemala and southern mexico has resulted in much linguistic exchange in the forms of loanwords and calques, usually manifesting as the influence of a language with more speakers and political leverage over another with fewer speakers and less influence. dialectal divergence within the same language is also attested, as is the case with the dichotomies of the western and eastern, and standard and lowland dialects of q'eqchi' @xcite . additionally, the politics of identity play a key role in delimiting the difference between a language and a dialect in the mind of their speakers, as seen in the cases of achi, akatek, and chalchitek, which are sometimes considered dialects of kiche, qanjobal, and awakatek, respectively @xcite . in general, mayan languages exhibit limited mutual intelligibility, and code-switching with spanish and other mayan languages in areas of high contact is common @xcite . despite their relative obscurity in the field of nlp, mayan languages are well studied and documented in matters of historical linguistics, morphosyntax, phonology, and semantics, as seen in extensive works such as those by @xcite , @xcite , coon (2016), @xcite , @xcite and many others. ## related work african languages exist in a similar, albeit superlative, situation to that of indigenous languages in the americas. thousands of african languages exist across the continent, boasting millions of speakers and serving as an important symbol of culture and cultural exchange; nevertheless, they are poorly represented in nlp applications. the masakhane project @xcite seeks to address this situation by fostering a community of researchers and non-researchers alike dedicated to advancing the development of nlp for african languages. @xcite describes a number of challenges the african nlp commu-nity faces: little official support for african indigenous languages; lack of resources for any kind of nlp task, and when those resources exist, they are hard to find; lack of benchmarks; and low reproducibility. mayan languages, and indigenous languages in general, face the same issues. nevertheless, the work on nlp of indigenous languages of the americas continues @xcite . of note is the work by tyers and @xcite and tyers and howell (2021), who focus specifically on k'iche' and develop an annotated corpus for morphosyntactic structure and perform a survey of part-of-speech tagging methods respectively, and also @xcite who work on a finite-state transducer for performing morphological analysis on yucatec maya. similarly to our work, oncevay (2021) presents a multilingual nmt system, including both spanish-tomany and many-to-spanish models for aymara, ashaninka, quechua, and shipibo-konibo, all indigenous languages spoken in peru; in general, efforts to bring modern mt into the realm of endangered indigenous languages are gaining traction, with work focused on nahuatl, otomi, guarani, quechua, and many other prominent indigenous languages from countries where there exist a sizeable population of indigenous peoples @xcite @xcite @xcite . crucially, however, mayan languages are nearly non-existent in these endeavours. the introduction of multilingual and crosslingual nmt @xcite , along with their application in lowresource scenarios @xcite @xcite ) was of vital importance in the effort to improve nmt in languages with otherwise small or almost nonexistent written bodies of work. recent work on low-resource languages has been put forward by meta's nllb project @xcite , a contribution that includes several new benchmarks, including the flores+ dataset foot_3 (based on flores-200, an update of flores-101 @xcite ), and a state-of-the-art nmt translation model, called nllb-200, focusing on underrepresented and low-resource languages, though, unfortunately, not a single mayan language was included in their efforts. in contrast, google's madlad-400 dataset, along with its accompanying translation model @xcite , does include a number of mayan languages in the form of monolingual corpora originating from jw.org and bible sources. ## development and curation of mayanv to develop the mayanv corpora, we manually crawled, extracted, and cleaned a number of online resources, mostly published by the almg, except for the tzeltal dictionary @xcite ; these extracted resources, which we collectively call the mayan vocabularies following the naming convention laid out by the almg, include the following languages: achi (teletor @xcite , awakatek (academia de lenguas mayas de guatemala, 2006), chuj @xcite , itza' (academia de lenguas mayas de guatemala, 2020), ixil (laynez ayay et al., 2018), q'eqchi' @xcite , q'anjob'al @xcite ## evaluating nmt systems with mayanv we develop bilingual and multilingual bidirectional nmt systems for mayan languages and spanish. our aim is to assess the impact of the mayanv dataset on the translation quality of informal, familial domain texts. thus, we consider baseline models that have not been exposed to this data during training and compare their performance to models trained with mayanv data. the validation and test sets consist exclusively of sentences from mayanv. therefore, we will only evaluate language pairs for which mayanv data is available. other languages still contribute to the training of multilingual models but are not explicitly evaluated. all corpora involved are parallel between spanish and at least one mayan language. ## conclusions very little work has been done with indigenous american languages regarding their informal, dayto-day use and interaction with the dominant languages of their surroundings, and our results reflect this unfortunate fact. we have developed and publicly released a curated set of parallel datasets between several mayan languages and spanish, which we call mayanv, focusing on the fact that the dialect and the register of the corpora is informal and non-domain-specific, which reflects a more common use case for the majority of native speakers. we train baseline bilingual and multilingual nmt mayan-spanish models from scratch, and fine-tune the nllb-200 model for the bilingual case, and compare these with models whose train set is identical plus the addition of mayanv; we evaluate these models on a separate subset of mayanv and observe considerable improvements with respect to the baseline. this, along with a dialectometric analysis of the corpora involved, suggests that the vast majority of the available resources do not reflect the day-to-day usage of mayan languages by their native speakers, nor do they facilitate the training and development of mt systems that might be useful for the most common use cases of the language. however, we do observe several instances of improvement in performance when comparing multilingual models with their bilingual counterparts, suggesting that this remains a valid pathway for developing and training ever-improving nmt models for mayan languages. future work in the area of nmt of mayan languages should focus on the mining and production of datasets that reflect a closer use case to that which is useful for rural and often times marginalised indigenous communities, who usually do not speak using overly formal or archaic language, nor have consistent access to the internet and its associated zeitgeist. finally, interesting work is to be done by performing multilingual fine-tuning of larger pre-trained translation models, such as nllb-200 itself, or madlad-400, whose purpose is to work with low-resource and endangered languages and which, sadly, minimise, or even outright exclude, mayan languages in their current form.