The H elsinki submission to the A mericas NLP shared task (2021)

Abstract: The University of Helsinki participated in the AmericasNLP shared task for all ten language pairs. Our multilingual NMT models reached the first rank on all language pairs in track 1, and first rank on nine out of ten language pairs in track 2. We focused our efforts on three aspects: (1) the collection of additional data from various sources such as Bibles and political constitutions, (2) the cleaning and filtering of training data with the OpusFilter toolkit, and (3) different multilingual training techniques enabled by the latest version of the OpenNMT-py toolkit to make the most efficient use of the scarce data. This paper describes our efforts in detail.

## introduction the university of helsinki participated in the americasnlp 2021 shared task on open machine translation for all ten language pairs. the shared task is aimed at developing machine translation (mt) systems for indigenous languages of the americas, all of them paired with spanish @xcite . needless to say, these language pairs pose big challenges since none of them benefits from large quantities of parallel data and there is limited monolingual data. for our participation, we focused our efforts mainly on three aspects: (1) gathering additional parallel and monolingual data for each language, taking advantage in particular of the opus corpus collection @xcite , the jhu bible corpus @xcite and translations of political constitutions of various latin american countries, (2) cleaning and filtering the corpora to maximize their quality with the opusfilter toolbox @xcite , and (3) contrasting different training techniques that could take advantage of the scarce data available. we pre-trained nmt systems to produce backtranslations for the monolingual portions of the data. we also trained multilingual systems that make use of language labels on the source sentence to specify the target language @xcite . this has been shown to leverage the information available data across different language pairs and boosts performance on the low-resource scenarios. we submitted five runs for each language pair, three in track 1 (development set included in training) and two in track 2 (development set not included in training). the best-performing model is a multilingual transformer pre-trained on spanish-english data and fine-tuned to the ten indigenous languages. the (partial or complete) inclusion of the development set during training consistently led to substantial improvements. the collected data sets and data processing code are available from our fork of the organizers' git repository. foot_0 ## data preparation a main part of our effort was directed to finding relevant corpora that could help with the translation tasks, as well as to make the best out of the data provided by the organizers. in order to have an efficient procedure to maintain and process the data sets for all the ten languages, we utilized the opus-filter toolbox foot_1 @xcite . it provides both ready-made and extensible methods for combining, cleaning, and filtering parallel and monolingual corpora. opusfilter uses a configuration file that lists all the steps for processing the data; in order to make quick changes and extensions programmatically, we generated the configuration file with a python script. figure 1 shows a part of the applied opusfilter workflow for a single language pair, spanish-raramuri, and restricted to the primary training data. the provided training set and (concatenated) ## models we experimented with two major model setups, which we refer to by a and b below. both are multilingual nmt models based on the transformer architecture @xcite and are implemented with opennmt-py 2.0 @xcite . all models were trained on a single gpu. the training data is segmented using sentence-piece @xcite subword models with 32k units, trained jointly on all languages. following our earlier experience @xcite , subword regularization @xcite is applied during training. further details of the configurations are listed in appendix b. ## results we submitted three systems to track 1 (development set allowed for training), namely a-50dev, b-50dev and b-100dev, and two systems to track 2 (development set not allowed for training), namely a-0dev and b-0dev. the results are in in track 1, our model b-100dev reached first rank and b-50dev reached second rank for all ten languages. model a-50dev was ranked third to sixth, depending on the language. this shows that model b consistently outperformed model a, presumably thanks to its spanish-english pre-training. including the full development set in training (b-100dev) further improves the performance, although this implies that savepoint selection becomes guesswork. for track 2, the tendency is similar. model b-0dev was ranked first for nine out of ten languages, taking 2nd rank for spanish-quechua. a-0dev was ranked second to fourth on all except quechua. 8 ## conclusions in this paper, we describe our submissions to the americasnlp shared task, where we submitted translations for all ten language pairs in both tracks. our strongest system is the result of gathering additional relevant data, carefully filtering the data for each language pair and pre-training a transformer-based multilingual nmt system with large spanish-english parallel data. except for spanish-quechua in track 2, all our submissions ranked top for both tracks. aymara aym train globalvoices (tiedemann, 2012; prokopidis et al., 2016) extra boconst: https://www.kas.de/c/document_library/get_ file?@xmath0, ayr-x-bible-1997-v1 mono wikipedia crawls (tiedemann, 2020) bribri bzd train (feldman and coto-solano, 2020) bibles bzd-x-bible-bzd-v1 norm https://github.com/americasnlp/ americasnlp2021/blob/main/data/bribri-spanish/ orthographic-conversion.csv ashaninka cni train https://github.com/hinantin/ashaninkamt (ortega et al., 2020; cushimariano romano and sebastián q., 2008; mihas, 2011) bibles cni-x-bible-cni-v1 mono shashiyayi (bustamante et al., 2020): https://github.com/ iapucp/multilingual-data-peru guarani gn train (chiruzzo et al., 2020) extra* pyconst: http://ej.org.py/principal/ constitucion-nacional-en-guarani/ bibles gug-x-bible-gug-v1 mono wikipedia crawls (tiedemann, 2020) wixarika hch train https://github.com/pywirrarika/wixarikacorpora (mager et al., 2018) extra mxconst: https://constitucionenlenguas.inali.gob.mx/ bibles hch-x-bible-hch-v1 mono https://github.com/pywirrarika/wixarikacorpora (mager et al., 2018) norm https://github.com/pywirrarika/wixnlp/blob/master/ normwix.py (mager hois et al., 2016) nahuatl nah train axolotl (gutierrez-vasques et al., 2016) extra mxconst: https://constitucionenlenguas.inali.gob.mx/ bibles nch-x-bible-nch-v1, ngu-x-bible-ngu-v1, nhe-x-bible-nhe-v1, nhw-x-bible-nhw-v1 mono wikipedia crawls (tiedemann, 2020) spanish bibles spa-x-bible-americas, spa-x-bible-hablahoi-latina, spa-x-bible-lapalabra, spax-bible-newworld, spa-x-bible-nuevadehoi, spa-x-bible-nuevaviviente, spa-xbible-nuevointernacional, spa-x-bible-reinavaleracontemporanea