University of Cape Town’s WMT 22 System: Multilingual Machine Translation for S outhern A frican Languages (2022)

Abstract: The paper describes the University of Cape Town’s submission to the constrained track of the WMT22 Shared Task: Large-Scale Machine Translation Evaluation for African Languages. Our system is a single multilingual translation model that translates between English and 8 South / South East African Languages, as well as between specific pairs of the African languages. We used several techniques suited for low-resource machine translation (MT), including overlap BPE, back-translation, synthetic training data generation, and adding more translation directions during training. Our results show the value of these techniques, especially for directions where very little or no bilingual training data is available.

## introduction southern african languages are underrepresented in nlp research, in part because most of them are low-resource languages: it is not always possible to find high-quality datasets that are large enough to train effective deep learning models @xcite . the wmt22 shared task on large-scale machine translation evaluation for african languages @xcite presented an opportunity to apply one of the most promising recent developments in nlp -multilingual neural machine translation -to southern african languages. for many languages, the parallel corpora released for the shared task are the largest publicly available datasets yet. for some translation directions (e.g. between southern african languages), no parallel corpora were previously available. in this paper we present our submission to the shared task. our system is a transformer-based encoder-decoder @xcite that translates between english and 8 south / south east african languages (afrikaans, northern sotho, 1 our model is available at https://github.com/khalidnabigh/uct-s-wmt22-shared-task . shona, swati, tswana, xhosa, xitsonga, zulu) and in 8 additional directions (xhosa to zulu, zulu to shona, shona to afrikaans, afrikaans to swati, swati to tswana, tswana to xitsonga, xitsonga to northern sotho, northern sotho to xhosa). we trained a single model with shared encoder and decoder parameters and a shared subword vocabulary. we applied several methods aimed at improving translation performance in a low-resource setting. we experimented with bpe @xcite and overlap bpe @xcite , the latter of which increases the representation of low-resource language tokens in the shared subword vocabulary. we used initial multilingual and bilingual models to generate back-translated sentences @xcite for subsequent training. first, we trained a model to translate between english and the 8 southern african languages. then we added the 8 additional translation directions and continued training. for some of these additional directions no parallel corpora were available, so we generated synthetic training data with our existing model. by downsampling some of the parallel corpora to ensure a balanced dataset, we were able to train our model effectively in the new directions, while retaining performance in the old directions. we describe the development of our model and report translation performance at each training stage. our final results compare favourably to existing works with overlapping translation directions. while there is considerable disparity in performance across languages, our model nonetheless achieves results that indicate some degree of effective mt across all directions (most bleu scores are above 10 and most chrf++ scores are above 40). we also discuss our findings regarding techniques for low-resource mt. we found overlap bpe and back-translation to improve performance for most translation directions. furthermore, our results confirm the value of multilingual models, which proves critical for the lowest-resource languages. ## datasets the wmt22 dataset is released along with the shared task. it contains bitext for 248 pairs of african languages, referred to as wmt22_african. 2 we use wmt22_african for both training and validation; the first 3 000 sentences from each language pair is reserved for validation and the rest for training. ## methodology in this work, we only focus on south and south east african languages, their translation to/from english, and eight translation directions between these languages. we divided the training of the translation system into two stages. in the first stage, we trained a multilingual model for translating from all lrls to english and vice versa. to incorporate the translation directions between lrls into the system, we did further training on the translation model from stage 1. we divided the training process into stages instead of training the model in one session due to computational resource constraints. both stages are explained in more detail below. all models were trained with the fairseq toolkit @xcite . we used the transformer-base architecture @xcite for training all bilingual models. we base the multilingual models on the bart architecture @xcite , using tang et al.'s (2021) implementation and hyperparameters, including adding a token to indicate the source language before the input sentence and a token for the target language before the output sentence. ## results we primarily used bleu score for evaluating all models on the flores dev set. the final test set evaluation by the shared task organizers additionally used sentence piece bleu (spbleu) and chrf2. ## conclusion we have presented our multilingual neural mt model for 8 southern african languages. until recently, it would not have been possible to train a multilingual model for these languages because of data scarcity. during model development we found the benefits of multilingual modelling to be especially great for the lowest-resourced languages. our results show that overlap bpe, backtranslation, and synthetic training data generation are all valuable techniques for low-resource mt. more generally, we find multilingual modelling to be a fruitful approach to southern african mt. for future work we would like to investigate further approaches for training large multilingual models for low-resource languages with a limited compute budget.