Open Machine Translation for Low Resource S outh A merican Languages ( A mericas NLP 2021 Shared Task Contribution) (2021)

Abstract: This paper describes the team (“Tamalli”)’s submission to AmericasNLP2021 shared task on Open Machine Translation for low resource South American languages. Our goal was to evaluate different Machine Translation (MT) techniques, statistical and neural-based, under several configuration settings. We obtained the second-best results for the language pairs “Spanish-Bribri”, “Spanish-Asháninka”, and “Spanish-Rarámuri” in the category “Development set not used for training”. Our performed experiments will serve as a point of reference for researchers working on MT with low-resource languages.

## introduction the main challenges in automatic machine translation (mt) are the acquisition and curation of parallel data and the allocation of hardware resources for training and inference purposes. this situation has become more evident for neural machine translation (nmt) techniques, where their translation quality depends strongly on the amount of available training data when offering translation for a language pair. however, there is only a handful of languages that have available large-scale parallel corpora, or collections of sentences in both the source language and corresponding translations. thus, applying recent nmt approaches to low-resource languages represent a challenging scenario. in this paper, we describe the participation of our team (aka, tamalli) in the shared task on open machine translation held in the first workshop on nlp for indigenous languages of the americas (americasnlp) @xcite . 1 the main goal of the shared task was to encourage the development of machine translation systems for indigenous languages of the americas, categorized as low-resources languages. this year 8 different teams participated with 214 submissions. accordingly, our main goal was to evaluate the performance of traditional statistical mt techniques, as well as some recent nmt techniques under different configuration settings. overall, our results outperformed the baseline proposed by the shared task organizers, and reach promising results for many of the considered pair languages. the paper is organized as follows: section 2 briefly describes some related work; section 3 depicts the methodology we followed for performing our experiments. section 4 provides the dataset descriptions. section 5 provides the details from our different settings, and finally section 6 depict our main conclusions and future work directions. ## related work machine translation (garg and agarwal, 2018) is a field in nlp that aims to translate natural lan-guages. particularly, the development of (mt) systems for indigenous languages in both south and north america, faces different challenges such as a high morphological richness, agglutination, polysynthesis, and orthographic variation @xcite . in general, mt systems for these languages in the state-of-theart have been addressed by the sub-fields of machine translation: rule-based @xcite , statistical @xcite and neuralbased approaches @xcite . recently, nmt approaches @xcite have gained prominence; they commonly are based on sequence-to-sequence models using encoder-decoder architectures and attention mechanisms @xcite . from this perspective, different morphological segmentation techniques have been explored @xcite for indigenous american languages. it is known that the nmt approaches are based on big amounts of parallel corpora as source knowledge. to date, important efforts toward creating parallel corpora have been carried out for specific indigenous languages of america. for example, for spanish-nahuatl @xcite , wixarika-spanish @xcite and quechua-spanish @xcite which includes morphological information. also, the jhu bible corpus, a parallel text, has been extended by adding translations in more than 20 indigenous north american languages @xcite . the usability of the corpus was demonstrated by using multilingual nmt systems. ## methodology since the data sizes are small in most language pairs as shown in ## dataset for training and evaluating our different configurations, we used the official datasets provided by the organizers of the shared task. it is worth mentioning that we did not use additional datasets or resources for our experiments. a brief description of the dataset composition is shown in ## experimental results we used 5 settings for all the 10 pair translations. the output of each set is named as version [1-5] and submitted for evaluation (shown under column submission# in version 1: version 1 uses the statistical mt. the source and target language text were first tokenized using moses tokenizer setting the language to spanish. then we trained the ibm translation model 2 @xcite implemented in nltk.translate api. after obtaining the translation target tokens, the detokenization was carried out using the moses spanish detokenizer. version 2: this version uses the one-to-one nmt model. first, we learned sentence piece bpe tokenization @xcite by combining the source and target language text. we set the maximum vocabulary size to {8k, 16k, 32k} in different runs and we considered the run that produced the best bleu score on the dev set. the transformer model @xcite was implemented using pytorch @xcite . the number of encoder and decoder layers was set to 3 each and the number of heads in those layers was set to 8. the hidden dimension of the self-attention layer was set to 128 and the position-wise feedforward layer's dimension was set to 256. we used a dropout of 0.1 in both the encoder and the decoder. the encoder and decoder embedding layers were not tied. we trained the model using early stopping with a patience of 5 epochs, that is, we stop training if the validation loss does not improve for 5 consecutive epochs. we used greedy decoding for generating the translations during inference. the training and translation were done using one gpu. version 3: this version uses the one-to-many nmt model. for tokenization, we learned sentence piece bpe tokenization @xcite ) by combining the source and target language text from all the languages (11 languages in total). we set the maximum shared vocabulary size to {8k, 16k, 32k} in different runs and we considered the run that produced the best bleu score on the dev set. the transformer model's hyperparameters were the same as in version 2. the language embedding dimension in the decoder was set to 128. the encoder and decoder embedding layers were not tied. we first trained the one-to-many model till convergence using early stopping with the patience of 5 epochs, considering the concatenation of the dev data from all the language pairs. then we fine-tuned the best checkpoint using each language pair's data separately. the fine-tuning process was also done using early stopping with patience of 5 epochs. finally, we used greedy decoding for generating the translations during inference. the training and translation were done using one gpu. version 4: this version is based on one-to-one nmt. we have used the transformer model as implemented in opennmt-py (pytorch version) @xcite . 4 . to train the model, we used a single gpu and followed the standard "noam" learning rate decay, foot_4 see @xcite for more details. our starting learning rate was 0.2 and we used 8000 warmup steps. the model es-nah trained up to 100k iterations and the model checkpoint at 35k was selected based on the evaluation score (bleu) on the development set. version 5: this version is based on one-to-one nmt. we have used the transformer model as implemented in opennmt-tf (tensorflow version) @xcite . to train the model, we used a single gpu and followed the standard "noam" learning rate decay, foot_5 see @xcite for more details. we used 8k shared vocab size for the models and the model checkpoints were saved at an interval of 2500 steps. the starting learning rate was 0.2 and 8000 warmup steps were used for model training. the earlystopping criterion was 'less than 0.01 improvement in bleu score' for 5 consecutive saved model checkpoints. the model es-gn was trained up to 37.5k iterations and the model checkpoint at 35k was selected based on evaluation scores on the development set. the model es-quy was trained up to 40k iterations and the model checkpoint at 32.5k was selected based on evaluation scores on the development set. we report the official automatic evaluation results in ## conclusions our participation aimed at analyzing the performance of recent nmt techniques on translating indigenous languages of the americas, low-resource languages. our future work directions include: i) investigating corpus filtering and iterative augmentation for performance improvement @xcite , ii) review already existing extensive analyses of these low-resource languages from a linguistic point of view and adapt our methods for each language accordingly, iii) exploring transfer learning approach by training the model on a high resource language and later transfer it to a low resource language @xcite .