A fro MT : Pretraining Strategies and Reproducible Benchmarks for Translation of 8 A frican Languages (2021)

Abstract: Reproducible benchmarks are crucial in driving progress of machine translation research. However, existing machine translation benchmarks have been mostly limited to high-resource or well-represented languages. Despite an increasing interest in low-resource machine translation, there are no standardized reproducible benchmarks for many African languages, many of which are used by millions of speakers but have less digitized textual data. To tackle these challenges, we propose AfroMT, a standardized, clean, and reproducible machine translation benchmark for eight widely spoken African languages. We also develop a suite of analysis tools for system diagnosis taking into account the unique properties of these languages. Furthermore, we explore the newly considered case of low-resource focused pretraining and develop two novel data augmentation-based strategies, leveraging word-level alignment information and pseudo-monolingual data for pretraining multilingual sequence-to-sequence models. We demonstrate significant improvements when pretraining on 11 languages, with gains of up to 2 BLEU points over strong baselines. We also show gains of up to 12 BLEU points over cross-lingual transfer baselines in data-constrained scenarios. All code and pretrained models will be released as further steps towards larger reproducible benchmarks for African languages.

## introduction accuracy of machine translation systems in many languages has improved greatly over the past several years due to the introduction of neural machine translation (nmt) techniques @xcite @xcite , as well as scaling to larger models @xcite . however, many of these advances have been demonstrated in settings where very large parallel datasets are available @xcite , and nmt systems often underperform in low-resource settings when given small amounts of parallel corpora @xcite . one solution to this has been leveraging multilingual pretraining on large sets of monolingual data @xcite @xcite , leading to improvements even with smaller parallel corpora. however, this thread of work has focused on scenarios with the following two properties: (1) pretraining on a plurality of european languages and (2) cases in which the monolingual pretraining data greatly exceeds the parallel data used for finetuning (often by over 100 times) @xcite . however, in the case of many languages in the world, the above two properties are often not satisfied. in particular, taking the example of african languages (the focus of our work), existing (small) parallel corpora for english-to-african language pairs often comprise the majority of available monolingual data in the corresponding african languages. in addition, african languages are often morphologically rich and from completely different language families, being quite distant from european languages. moreover, despite the importance of reproducible benchmarks to measuring progress on various tasks in an empirical setting, there exists no standardized machine translation benchmark for the majority of african languages. in this work, we introduce (1) a new machine translation benchmark for african languages, and (2) pretraining techniques to deal with the previously unexplored case where the size of monolingual data resources for pretraining is similar or equal to the size of parallel data resources for finetuning, and (3) evaluation tools designed for measuring qualities regarding the unique grammar of these languages in machine translation systems for better system evaluation. our proposed benchmark, afromt, consists of translation tasks between english and 8 african languages -afrikaans, xhosa, zulu, rundi, sesotho, swahili, bemba, and lingala -four of which are not included in commercial translation systems such as google translate (as of @xcite . in §2, we describe the detailed design of our benchmark, including the language selection criterion and the methodology to collect, clean and normalize the data for training and evaluation purposes. in §3, we provide a set of strong baselines for our benchmark, including denoising sequence-to-sequence pretraining @xcite , transfer learning with similar languages @xcite , and our proposed data augmentation methods for pretraining on low-resource languages. our first method leverages bilingual dictionaries to augment data in high-resource languages (hrl), and our second method iteratively creates pseudo-monolingual data in low-resource languages (lrl) for pretraining. extensive experiments in §4 show that our proposed methods outperform our baselines by up to ∼2 bleu points over all language pairs and up to ∼15 bleu points in data-constrained scenarios. ## afromt benchmark in this section, we detail the construction of our new benchmark, afromt. we first introduce our criteria for selecting the languages ( §2.1), and then describe the steps to prepare the dataset ( §2.2, 2.3). ## afrobart given that we aim to provide strong baselines for our benchmark, we resort to multilingual sequenceto-sequence training. however, existing pretraining techniques have often been focused on the situation where monolingual data can be found in a larger quantity than parallel data. in this section we describe our proposed multilingual sequenceto-sequence pretraining techniques developed for the novel scenario where even monolingual data is scarce. ## experimental setup in this section, we describe our experimental setup for both pretraining and finetuning strong baselines for our benchmark. furthermore, we look to evaluate the efficacy of our proposed pretraining ## conclusion in this work we proposed a standardized, clean, and reproducible benchmark for 8 african languages, afromt, as well as novel pretraining strategies in the previously unexplored low-resource focused setting. our benchmark and evaluation suite are a step towards larger, reproducible benchmarks in these languages, helping to provide insights on how current mt techniques work for these underexplored languages. we will release this benchmark, our pretrained afrobart models, dictionaries, and pseudo monolingual data to the community to facilitate further work in this area. in future work we look to use similar methodology to advance in both of these directions. we look to increase the number of language pairs in afromt to be more representative of the african continent. additionally, we look to scale up our pretraining approaches for increased performance. 20-core intel xeon e5-2698 v4 2.2 ghz. for pretraining we make use of a compute cluster using 8 nodes with 4 nvidia v100 16gb gpus per node.