ANVITA - A frican: A Multilingual Neural Machine Translation System for A frican Languages (2022)

Abstract: This paper describes ANVITA African NMT system submitted by team ANVITA for WMT 2022 shared task on Large-Scale Machine Translation Evaluation for African Languages under the constrained translation track. The team participated in 24 African languages to English MT directions. For better handling of relatively low resource language pairs and effective transfer learning, models are trained in multilingual setting. Heuristic based corpus filtering is applied and it improved performance by 0.04-2.06 BLEU across 22 out of 24 African to English directions and also improved training time by 5x. Use of deep transformer with 24 layers of encoder and 6 layers of decoder significantly improved performance by 1.1-7.7 BLEU across all the 24 African to English directions compared to base transformer. For effective selection of source vocabulary in multilingual setting, joint and language wise vocabulary selection strategies are explored at the source side. Use of language wise vocabulary selection however did not consistently improve performance of low resource languages in comparison to joint vocabulary selection. Empirical results indicate that training using deep transformer with filtered corpora seems to be a better choice than using base transformer on the whole corpora both in terms of accuracy and training time.

## introduction africa is very rich in languages, and around 1200 to 2100 languages are spoken in african countries 1 , 24 african languages and 100 language pairs were selected for the wmt22 large-scale machine translation evaluation for african languages shared task @xcite . selected 24 african languages include afrikaans(afr), amharic(amh), chichewa(nya), hausa(hau), igbo(ibo), kamba(kam), kinyarawanda(kin), 1 https://en.wikipedia.org/wiki/languages_of_africa lingala(lin), luganda(lug), luo(luo), nigerian fulfulde(fuv), northern sotho(nso), oromo(orm), shona(sna), somali(som), swahili(swh), swati(ssw), setswana(tsn), umbundu(umb), wolof(wol), xhosa(xho), xitsonga(tso), yoruba(yor) and zulu(zul) and language pairs include african-english, selective african-french, and african-african pairs, where many of the pairs fall under the low resource category. in this task, organizers permitted two submissions, best scoring submission is considered as primary model and other one being the contrastive model. this paper describes our submission to wmt 2022 large-scale machine translation evaluation for african languages shared task where we participated for translation of 24 african languages to english. we are not officially given a rank as we didn't participate in all african mt directions. ## related work developing quality machine translation system for low resource languages still remains a major challenge and many of the world languages fall under this category. some of the recent developments do show that multilingual nmt is a promising direction. in massively multilingual neural machine translation, the authors have shown to train a single model for translating 102 languages to and from english and the results outperformed the strong bilingual baseline mt system especially for low resource languages @xcite . however, it cannot be generalized to all high and medium resource languages. @xcite built a multilingual neural machine translation system capable of translating from 500 source languages to english which includes medium, low and extremely low resource languages. @xcite improved zero-shot translation in multilingual neural machine translation by random back translation. @xcite have shown that represen-tations of high resource and/or linguistically similar languages are more robust when fine-tuning on an arbitrary language pair, which is critical to determining how much cross-lingual transfer can be expected in a zero or few-shot setting. @xcite shown deep architectures for neural machine translation and post ensemble have shown improved results on machine translation tasks. @xcite presented language independent heuristics for filtering noisy pairs from parallel corpus. @xcite proposed progressive training, in which the mt system is trained from shallow to deep architectures -increasing number of encoder and decoder layers. however, major improvement is observed while increasing encoder layers. adelani et al. (2022a) ## datasets we used all the parallel corpora provided by wmt 2022 organizer. corpus contain existing opus repository @xcite , wmt 2022 novel corpus 2 and comprises of sources such as wikimedia, ccmatrix, ccaligned, bible-uedin, gnome, xlent, qed,kde4, mozilla-i10n, spc, ted2020, tatoeba, elrc_2922, opensubtitles, ubuntu, lava corpus2, mafand- @xcite , kentrans @xcite , webcrawl-african foot_1 and huggingface (provided by organisers) etc. @xcite . combining all a total 140 million parallel sentences for the 24 african-english language pairs are extracted. language wise statistics of corpus used in our system is listed in ## experimental evaluation and result analysis experiments carried out for 6 distinct configurations to assess effect of filtering, deep transformer and strategies used for vocabulary selection. ## conclusion this paper describes our submission to wmt 2022 shared task on large-scale machine translation evaluation for african languages under the constrained translation track. we focused on 24 african languages to english mt directions. multilingual model with deep transformer showed significant improvement in bleu and chrf2++ scores across all 24 african to english mt directions. vocabulary size of 4k to 6k per language for estimating size of joint source vocabulary seems to be a good choice in a multilingual setup. heuristic based filtering did improve the bleu scores. however the biggest gain of filtering observed is in terms of training time speed up by 5x. empirical results indicate that training using deep transformer