NRC - CNRC Machine Translation Systems for the 2021 A mericas NLP Shared Task (2021)

Abstract: We describe the NRC-CNRC systems submitted to the AmericasNLP shared task on machine translation. We submitted systems translating from Spanish into Wixárika, Nahuatl, Rarámuri, and Guaraní. Our best neural machine translation systems used multilingual pretraining, ensembling, finetuning, training on parts of the development data, and subword regularization. We also submitted translation memory systems as a strong baseline.

## introduction this paper describes experiments on translation from spanish into wixárika, nahuatl, rarámuri, and guaraní, as part of the first workshop on natural language processing (nlp) for indigenous languages of the americas (americasnlp) 2021 shared task on open-ended machine translation. our approach to this task was to explore the application of simple, known methods of performing neural machine translation (nmt) for low-resource languages to a subset of the task languages. our initial experiments were primarily focused on the following questions: (1) how well does multilingual nmt work in these very low resource settings? (2) is it better to build multilingual nmt systems using only closely-related languages or does it help to add data from additional languages? (3) is applying subword regularization helpful? as we progressed through the task, it raised questions regarding domain and about use cases for low-resource machine translation. the approaches that we used for this task are not entirely languageagnostic; they might be more appropriately characterized as "language naïve" in that we applied some simple language-specific pre-and post-processing, but did not incorporate any tools that required indepth knowledge of the language. we submitted four systems, including ensembles, single systems, and a translation memory baseline. our best system (s.0) consisted of an ## data and preprocessing the shared task provided data for 10 language pairs, all with the goal of translating from spanish. we chose to start with wixárika (hch; mager et al., 2018), nahuatl (nah; gutierrez-vasques et al., 2016), and rarámuri (tar; brambila, 1976) as our main three languages of interest, all of which are languages in the uto-aztecan family indigenous to mexico. we added guaraní (gn; chiruzzo et al., 2020) as an unrelated language (as spoken in paraguay), to explore building multilingual nmt systems within and across language families. ebrahimi et al. (2021) describes work on collecting development and test sets for the languages in the shared task. the datasets vary in size, dialect and orthographic variation/consistency, and level of domain match to the development and test data. due to space considerations, we direct readers to the task page and the dataset information page for more information on the languages and on the datasets provided for the task. @xcite given the size of the data ( ## models and experiments we report chrf (popović, 2015) scores computed with sacrebleu (post, 2018). ## results our results consistently placed our submissions as the second-ranking team (behind helsinki's top 2-3 submissions) in the with-development-set group, and second or third ranking team (2nd, 3rd, or 4th submission) within the no-development-set cluster as measured by chrf. for wixárika and rarámuri particularly, our tm submission proved to be a surprisingly strong baseline. we note that chrf and bleu are not strictly correlated, and for all languages, scores are low. this raises questions about goals, metrics, and use cases for very low resource machine translation. we provide a short discussion of this in appendix a. it will require future work and human evaluation to determine whether such systems are useful or harmful in downstream tasks.