Neural Machine Translation Models with Back-Translation for the Extremely Low-Resource Indigenous Language B ribri (2020)

Abstract: This paper presents a neural machine translation model and dataset for the Chibchan language Bribri, with an average performance of BLEU 16.9±1.7. This was trained on an extremely small dataset (5923 Bribri-Spanish pairs), providing evidence for the applicability of NMT in extremely low-resource environments. We discuss the challenges entailed in managing training input from languages without standard orthographies, we provide evidence of successful learning of Bribri grammar, and also examine the translations of structures that are infrequent in major Indo-European languages, such as positional verbs, ergative markers, numerical classifiers and complex demonstrative systems. In addition to this, we perform an experiment of augmenting the dataset through iterative back-translation (Sennrich et al., 2016a; Hoang et al., 2018) by using Spanish sentences to create synthetic Bribri sentences. This improves the score by an average of 1.0 BLEU, but only when the new Spanish sentences belong to the same domain as the other Spanish examples. This contributes to the small but growing body of research on Chibchan NLP.

## introduction state-of-the-art systems in neural machine translation require large amounts of rich and varied parallel data between the source and target language @xcite . while this is less of a problem for highresource language pairs like english-german or english-spanish, there are much fewer resources when one of the languages is an indigenous language. while there are some researchers creating systems to optimize the training of nmt frameworks with lower resources, most work primarily from a "warmstart" where models are pre-trained on a small, but still significant amount of parallel data @xcite . for example even in papers geared towards developing techniques for low resource languages, training datasets can exceed millions of sentences @xcite . our motivation in adapting lowresource techniques to extremely low-resource situations is that they can help improve understanding of their underlying mechanisms and lower the barrier of entry for machine translation to be used for language revitalization purposes. however, in the effective body of research, many papers use simulated low-resource scenarios where high resource parallel datasets are artificially truncated to test the efficacy of their techniques. while this does provide a level of standardization across different techniques, it is still artificial. this paper seeks to explore nmt in an authentic scenario, using an actual extremely low-resource language. we leverage a variety of sources to produce a novel translation dataset in the indigenous bribri language for demonstrating methods of neural machine translation (nmt) in extremely low-resource situations. to demonstrate usage of the dataset, we apply the technique of iterative back-translation with validation @xcite . finally, we present a translation analysis to show the unique challenges that this bribri dataset presents. ## challenges for nmt in extremely low-resource languages in extremely low resource situations, the drawbacks of nmt systems begin to show. nmt is a datahungry approach to machine translation and much less efficient with respect to training data compared to other approaches @xcite , but with enough data, it has shown excellent results @xcite . still, these approaches assume that written data is continuously produced in the languages. however, there are 7000 languages in the world @xcite ; many of them are spoken in indigenous communities, have small populations, and are scarcely used in social media or in other forms amenable to automatic scraping @xcite @xcite . in many of these communities, languages like english and spanish have displaced the indigenous languages in domains such as technology and chatting, and so the available data is curtailed. in addition to this, many indigenous communities have chronic digital inequalities, which makes it difficult to generate crowd-sourcing campaigns for those languages. finally, in many cases the data that is most valuable to speakers of the language is that collected from elders and knowledge keepers, but those elders might be the people who have the least access to technological means of communication. nmt also struggles with out of domain translation @xcite which is only exacerbated by a smaller dataset. our primary solution to this is employing a popular dataset augmentation technique called iterative back-translation @xcite . additionally, nmt models are fairly opaque @xcite in that their learning process is not deterministic and is difficult to interpret. here, we try to provide a closer, linguistic analysis of the translations produced by the models. ## nmt training from bribri bitext using the resources cited above, we created a dataset with 5923 bribri-spanish sentence pairs. we performed diacritic and writing system standardization with a rule-based, deterministic system, with a result that smooths out some author-specific variation and that can be converted to either type of contemporary orthography (the @xcite and the jara murillo (2018a) conventions). for example, the string ùx represents a nasal high-tone 'u', as the 'x' is not used as a letter in the language. (for simplicity, we will use the convention of indicating nasalization with a line underneath when showing bribri text on this paper). we normalized forms to the amubri orthography for phonological variations (e.g. ñalà∼ñol ò 'road') when there is a systematic way to convert them back to coroma orthography, but preserved lexemes that are exclusive to each variant (coroma: alàralar 'children'). finally, we attempted to standardize common function words (e.g. copula rör to dör). after standardizing all sentence pairs, we encoded each sentence with separate byte pair encoding models for each language @xcite where each encoding had a vocabulary size of about 2200 tokens. for each trial the dataset was split between 80% training, 10% validation, and 10% for testing. additionally, the data was randomly shuffled ten times, resplit and retrained for further validation. from these sets we trained transformer encoder/decoder models which in previous research have shown good performance for translation tasks. we used near identical hyperparameters to the base model described in @xcite with the exceptions being the number of training steps: each model was trained 4000 steps with a batch size of 4096 tokens on a single gpu. the models were only trained 4000 steps to prevent overfitting on the small dataset. each translation model was trained using the pytorch port of the opennmt package @xcite on google colaboratory instances with a range of different gpus. the training took approximately 50 minutes per model. each model was then evaluated against the testing data with the bilingual evaluation understudy score (bleu) technique @xcite where the specific implementation was the multi-bleu script from moses @xcite . the code for sentence standardization, training, and a sample of bri-spa sentence pairs can be found here: http://github.com/rolandocoto/bribri-coling2020 . ## iterative back-translation back-translation is the technique for leveraging large monolingual corpora and weak translation models to augment parallel datasets @xcite . a translation model is built using real parallel data from the target language into the source language. then, monolingual sentences in the target language are translated using that model to create synthetic bitext between the two languages. finally, this synthetic bitext is concatenated with the real bitext and a final model is trained. this simple technique has been shown to increase scores in some models by almost 2 bleu @xcite . iterative backtranslation builds upon this technique by using the first synthetic bitext and real bitext to train a second target-to-source translation model to create new synthetic bitext that is, in theory, more accurate @xcite .