Translation systems for low-resource Colombian Indigenous languages, a first step towards cultural preservation (2024)

Abstract: The use of machine learning and Natural Language Processing (NLP) technologies can assist in the preservation and revitalization of indigenous languages, particularly those classified as “low-resource.” Given the increasing digitization of information, the development of translation tools for these languages is of significant importance. These tools not only facilitate better access to digital resources for indigenous communities but also stimulate language preservation efforts and potentially foster more inclusive, equitable societies, as demonstrated by the AmericasNLP workshop since 2021. The focus of this paper is Colombia, a country home to 65 distinct indigenous languages, presenting a vast spectrum of linguistic characteristics. This cultural and linguistic diversity is an inherent pillar of the nation’s identity, and safeguarding it has been increasingly challenging given the dwindling number of native speakers and the communities’ inclination towards oral traditions. Considering this context, scattered initiatives exist to develop translation systems for these languages. However, these endeavors suffer from a lack of consolidated, comparable data. This paper consolidates a dataset of parallel data in four Colombian indigenous languages - Wayuunaiki, Arhuaco, Inga, and Nasa - gathered from existing digital resources. It also presents the creation of baseline models for future translation and comparison, ultimately serving as a catalyst for incorporating more digital resources progressively.

## introduction in the field of natural language processing (nlp), low-resource languages are characterized by limited written or spoken digital material. consequently, applying translation models rooted in advanced neural architectures to these languages is challenging due to the models' high dependence on substantial data volumes @xcite . however, recent years have seen a growing trend towards working with low-resource languages for machine translation based on transformers. notably, efforts to preserve indigenous languages have significantly contributed to advancements in this area @xcite @xcite . the americas, in particular, host numerous endangered indigenous languages spoken by small populations. in response, several researchers have devoted their work towards developing translation models for some of these languages @xcite . highlighting this effort, americasnlp was convened in 2021 as the first global workshop dedicated to the application of nlp to american indigenous languages @xcite . the development of translation tools catering specifically to indigenous languages has the potential to confer numerous advantages, including enriched access to digital resources and the promotion of language preservation efforts. as the availability of digital resources proliferates, it becomes increasingly imperative for indigenous communities to have access to information in their respective languages, in order to safeguard their unique cultures and traditional ways of living. regrettably, the limited resources available in indigenous languages restrict their accessibility to crucial digital materials. the creation of translation tools could serve as a resolution to this issue, enabling communities to translate digital resources into their languages. beyond simply providing improved access to information, these tools could stimulate language preservation by offering a medium for language revitalization. furthermore, translation tools could provide a platform for indigenous communities to engage with the global community, contributing to their economic and social development. by fostering cross-cultural understanding and mutual respect, translation tools could play a critical role in constructing more inclusive and equitable societies. as acknowledged by the national indigenous organization of colombia (onic) (de gobierno indígena -onic, 2015), colombia's linguistic diversity is marked by the existence of 65 indigenous languages, alongside spanish and two creole languages. among the 32 different departments in the territory, the regions of amazonas and vaupés, located in the southern sector of colombia, stand out for their significant diversity of indigenous languages. this assortment of indigenous languages in colombia is highly distinctive due to their different characteristics. for instance, colombia accommodates tonal languages similar to those found in southeast asia and central africa, along with languages representing all four classic morphological types. these include inflectional (as exemplified by kogui and arhuaco), agglutinative (like achagua, andoque, and páez), isolating akin to the malayo-polynesian languages (such as embera and creole), and polysynthetic (such as kamsá) (de estudios de lenguas aborígenes , c.c.e.l.a). according to onic, the task of preserving these indigenous voices is increasingly formidable, primarily due to their endangerment amid the dwindling number of native speakers. over half of these languages, alarmingly, have fewer than a thousand active speakers, thus exacerbating their preservation and conservation challenges (de gobierno indígena -onic, 2015). further complexities arise from the proclivity of certain communities to uphold their oral traditions over written modes. take, for example, the inga community, wherein cultural identity preservation is embedded in the prioritization of oral tradition as the primary vehicle for knowledge transmission. such communities contend that the detachment from oral traditions can incite a loss of numerous practices that necessitate face-to-face interaction and stimulate dialogue @xcite . despite the existence of scattered efforts aimed at creating translation systems for some indigenous colombian languages, we found no consolidated data that allows for reuse and/or comparison @xcite @xcite . only recently have initiatives emerged that have promoted data replicability and openness in colombian indigenous languages @xcite . the primary objective of this work is to consolidate a dataset of parallel data in four colombian indigenous languages: wayuunaiki, arhuaco, inga, and nasa. this dataset is a product of the compilation, processing, and align-ment of already existing digital resources in these languages. it aims to serve as a starting point to encourage the incorporation of new digital resources progressively. a second objective is to create a set of baseline models for translation that can be used for comparison in future research. ## related work low-resource languages, which lack the digital or written material needed to build a corpus or a linguistic collection, include indigenous or endangered languages, region-specific dialects, or languages without substantial digital resources despite the existence of millions of speakers. the shortage of available data, which often results from limited access to technology, opens up opportunities to apply various techniques like data augmentation, back-translation, and transfer learning to mitigate this scarcity. studies on diverse languages have explored neural machine translation, transfer learning, and advanced model architectures. for instance, an investigation of the tigrinya ethiopian language yielded positive results using transfer learning @xcite . in another study comparing three different models, the transformer network performed the best with a parallel corpus of yoruba and english @xcite . also, a modification of the transformer architecture led to improved results for south african languages @xcite . numerous studies demonstrate the effectiveness of transfer learning and advanced modeling techniques across global languages. for instance, finnish and czech were used as parent languages to assist low-resource estonian and slovak languages via transfer learning, leading to improvements over the baseline for almost all pre-trained models @xcite . the combination of attention layers and byte-pair encoding in transfer learning also notably enhanced translation capabilities for turkic languages @xcite . notably, the pairing of the transformer architecture with the back-translation technique resulted in improved translation quality for several language pairs @xcite . american indigenous languages, historically deficient in written records, pose unique challenges for language preservation. the americasnlp workshop provided a platform to unite global research groups to address these challenges, with a focus on machine translation across various indigenous languages. numerous techniques and strategies were employed across the participating teams in the different versions, yielding promising results. while some teams achieved success with multilingual neural networks @xcite @xcite , others found statistical machine translation more effective @xcite . additionally, using unique sources such as wikipedia and biblical texts to build the corpus yielded significant results, enhancing progress beyond baseline starts @xcite . in the recent 2023 edition of the workshop, the winning team's shared-task strategy comprised extending and finetuning several variants of the @xcite . this cutting-edge machine translation model is specifically tailored for environments with scarce resources. their submission surpassed the baseline by an average chrf of 11% across all languages, yielding especially considerable enhancements for aymara, guarani, and quechua (gow- @xcite . as for the indigenous colombian languages' translations, @xcite and @xcite studies are the only works we discovered that present a translation system from wayuunaiki to spanish and ika (arhuaco) to spanish. graichen et al. (2023) applied various unsupervised and semisupervised subword segmentation methods to enrich the data used to train a transformerbased model with linguistic information. according to the results, the incorporation of linguistic knowledge helps the system to generate improved translation. nonetheless, these methodologies introduced substantial noise into the process. ## baseline construction following the recent results obtained by the work of @xcite and (gow-smith and sánchez villegas, 2023) we use nllb-200, a stateof-the-art machine translation model specifically designed for low-resource settings. we experiment with different distilled versions of nllb-200 with 600m and 1.3b parameters. each dataset was randomly divided into training, validation, and testing sets, comprising 80%, 10%, and 10% of the total number of sentences, respectively. the resulting partition, along with the models and code, can be accessed at https://github.com/juanks235/ mt-colombian-indigenous-languages . ## conclusion and future work the preservation of indigenous languages, encompassing their stories, wisdom, and traditions is instrumental in fostering cross-cultural understanding. however, working with low-resource languages such as these often presents unique challenges, particularly in regions like colombia, which are teeming with linguistic diversity. we constructed a dataset of parallel data in four indigenous colombian languages, and the resulting dataset is freely accessible and usable for future research projects. additionally, we developed baseline translation models for each language pair. our findings demonstrated that the nllb 1.3b model excelled overall in comparison to the 600m model as expected. also, a contrast emerged in the range of the bleu score: from as low as 1.71 (inga) to as high as 15.37 (wayuu). such a significant difference is attributable to the disparities in data volume, with inga being the most challenging language. we also tested the influence of the inclusion in the training data of some of the sources in particular those that were challenging in the extraction phase. although this project did not involve direct engagement with community members, future work will prioritize establishing connections with these communities to expand the dataset and evaluate translation systems more thoroughly. our focus will be on incorporating additional indigenous colombian languages and exploring alternative models or architectures to potentially enhance translation outcomes.