M oses and the Character-Based Random Babbling Baseline: C o AS ta L at A mericas NLP 2021 Shared Task (2021)

Abstract: We evaluated a range of neural machine translation techniques developed specifically for low-resource scenarios. Unsuccessfully. In the end, we submitted two runs: (i) a standard phrase-based model, and (ii) a random babbling baseline using character trigrams. We found that it was surprisingly hard to beat (i), in spite of this model being, in theory, a bad fit for polysynthetic languages; and more interestingly, that (ii) was better than several of the submitted systems, highlighting how difficult low-resource machine translation for polysynthetic languages is.

## introduction shared tasks on machine translation are often conducted on large parallel training corpora: for example, the majority of datasets used in the wmt20 shared tasks have sentence pairs in the hundred thousands, often even millions @xcite . in contrast, the americasnlp 2021 shared task @xcite provided us with as little as 3,883 sentence pairs (for ashaninka), and with the exception of quechua (125k pairs), all languages had fewer than 30k sentence pairs. additionally, many of these languages are polysynthetic, which is known to provide additional challenges for machine translation @xcite . we initially focused our efforts on two areas: (i) obtaining more data, both parallel and monolingual (sec. 2); and (ii) exploring a range of different neural machine translation techniques, particular those specifically developed for low-resource scenarios, to find a promising system to build on and tweak further. unfortunately, we were wholly unsuccessful in the latter (sec. 5). all neural models that we tried performed extremely poorly when compared to a standard statistical phrase-based model (sec. 3.1). the overall low performance of all our models further prompted us to implement language source(s) aym aymara prokopidis et al. (2016) bzd bribri feldman and coto-solano (2020) cni asháninka ortega et al. (2020), cushimariano romano and sebastián q. (2008), mihas (2011) gn guaraní chiruzzo et al. (2020) hch wixarika mager et al. (2018a) nah nahuatl gutierrez-vasques et al. (2016) oto hñähñu comunidad elotl (2021) quy quechua agić and vulić (2019) shp shipibo-konibo galarreta et al. (2017) tar rarámuri @xcite ## data we train models for all languages provided by the shared task, using their official training datasets (cf. ## results results of our models are shown in phrase-based mt our phrase-based model (sec. 3.1) was ranked in track 1 of the shared task evaluation as it makes use of the development sets for tuning. compared to the other systems evaluated in this track, we observe a solid average performance of our model-it usually ranks in the middle of the field, with the best placement being 3rd on bribri, hñähñu, and shipibo-konibo, and the worst ranking being 8th out of 11 on guarani. in terms of chrf score, the model ranges between 0.159 (on raramuri) and 0.297 (on shipibo-konibo), but we note that there is a noticeable gap to the bestperforming system, helsinki-2, which outperforms ours by about +0.09 chrf on average. ## things that did not work here we briefly describe other ideas that we pursued, but were unfortunately not successful with, so we did not submit any systems based on these techniques for evaluation. pre-trained transformers following rothe et al. (2020), we use an auto-encoding transformer as the encoder and an auto-regressive transformer as the decoder of a sequence-to-sequence model. out of the several configurations we experimented with, the best performance was observed when the encoder is pre-trained on the spanish oscar corpus @xcite and the decoders are pre-trained on language-specific monolingual corpora collected from the web (cf. sec. 2) along with the target files of the training data. however, the results were not on-par with the simpler models; averaging over all languages, we observed a chrf score of 0.12 on the dev sets, compared to 0.23 with the phrase-based model (cf. sec. 3.1). we postulate that the training data was just not enough to train the cross-attention weights between the encoder and decoders. note that these weights need to be trained from scratch, as opposed to the other weights which are initialized from language modelling checkpoints. ## conclusion the (relative) success of our random babbling baseline shows that many mt systems fail to reproduce even superficial characteristics of word formation and character distribution in the target languages; a result that was confirmed by our own failed attempts at training a competitive neural mt model. out of the neural models we tried, purely character-level mt was among the more promising ones. we speculate that in the spanish-to-target setting, a model that combines a strong pre-trained spanish encoder with a purely character-level decoder might be a promising direction for further experiments. we also note that there are several languagespecific resources, such as morphological segmentation tools, 9 that might be worth using. we focused our efforts here on finding a broadly applicable architecture without any language-specific components, but would be curious to see if including such components can yield significant improvements on individual languages.